Everyone knows open source. Software. Right? It’s not about being for free (that’s usually an extra), it’s about giving the user access to the sources of the software to change it, customize it, extend it, integrate it, repair it, understand it, learn from it, …

The success of open source software is based on the principle that if you are going to consume something that you have the right to understand how it was produced, and that you are allowed to go for something different or pick it up. And it is the only way that allows thousands of people to work together on projects that are much larger than what each one alone could accomplish. Like building a web-browser that is faster, safer, and more reliable than the one that used to be installed on 99% of all running systems.

But who said that open source is a software thing? Think of making music or movies. Most of us have seen the one or other movie end titles (you know, all the names that are shown after the hero has kissed the girl). They put these names there because the film could not be made without them. Including all the weird positions after the last wookie actor and before the “very special thanks goes to”. Because you need that many people, it’s usually a big money subject to make good evening filling movies. But making good software used to be a big money subject as well before open source.

This is where Valkaama and other open source movie projects pick up. Allow anyone who likes to participate in making the movie. Script, act, film, speak, sing, play, cut, edit, improve, … as with open source software, you need access to the source “code” of the movie. The script, all raw filmed material, original sound, music (with original tracks), film editing plan, dubbed scenes, etc. Only then you are able to pick up a specific aspect of the film and change it for improvement. The animated short-movie Elephants Dream was the first that was released under an open-source license.

Valkaama is one of the first full feature-length film projects in HD1080p standard that is close to going public. Two trailers including all trailer sources have been released under the Creative CommonsAttribution – Share Alike 3.0 license on the project website.

The movie tells the story of two unlike young men, each seeking for his personal future, that are thrown together by fate to travel to ‘Valkaama’ to find what each is looking for. As their paths cross, they do not realize how much of their journey has already been determined by their pasts.

I have seen an early cut of the movie. I like it a lot. But as with large scale, high-quality software, large scale, high-quality movies need many helping hands. If you are into movies, this might be your thing. Have a look on the project’s website: http://www.valkaama.com/

disclaimer: the author is personally acquainted with the project leader of Valkaama

This post is the first in a series of posts where I write down what I have found out about the Eclipse Graphical Modeling Framework (GMF) by hard work, deep digging in the web and trial-and-error. These posts are meant to complement the existing documentation of GMF. I hope that this knowledge helps others in solving their problems faster than me.

This post (or say tutorial if you like) treats the definition and positioning of external labels of nodes and arcs (or connections) in a GMF diagram. For the rest of this post we assume that you are familiar with the basic GMF workflow of deriving an editor from an .ecore file. If not, you should start with the official GMF Tutorial first and come back when you are done.

1. First you need a node class and an arc class in your Ecore meta model, each having a property which you would like to display as a label of this node.

We define two classes Node and Arc. We are not concerned with the references between these classes here. They could be completely unrelated for the purpose of defining a label. We will use attribute name of class Node and attribute weight of class Arc as properties we want to display as labels. A sample attribute specification could be the following.

2. Next you need an appropriate graphics definition. Open your .gmfgraph file. I assume you defined a Figure Descriptor for your node or Arc. For an external node label, create a new Figure Descriptor, e.g. NodeNameFigure that only contains a Label, e.g. NodeNameLabel, and a Child Access to this label. We have to create this separate Figure Descriptor to make sure that our label is not internal to the graphics of our node. To label an arc (a connection), just add a Label and a Child Access to the corresponding Figure Descriptor of the Arc, e.g. ArcFigure.

Now we need to declare the labels themselves. Create a Diagram Label for the node, e.g. nodeName and set its Figure to the Figure Descriptor for the label that you’ve just created, e.g. NodeNameFigure. Done. This automatically makes this label an external label.

For the arc’s label also create a Diagram Label, e.g. arcWeight in our case; set its Figure to the Figure Descriptor of the arc’s figure, e.g. ArcFigure and set the Accessor to the corresponding Child Accessor, e.g. getFigureWeight. To control the distance of the label of the arc’s label to the arc, add a Label Offset Facet. Set its attributes X and Y to the distance you would like the label have from its arc. Done.

3. Now comes the mapping of graphics to the meta model. Open your .gmfmap file of your project. The relation between the graphics and meta model is defined by Feature Label Mappings. Again I assume that you’ve already defined Node Mappings and Link Mappings for your classes, we now add their labels.

For the node, find the corresponding Node Mapping. In our example this a node mapping for a Transition class which specializes the earlier Node class. Hence the Transition class also has the name attribute. At the Node Mapping, add a Feature Label Mapping and set the following properties: Diagram Label is the one you’ve just defined for your node class, e.g. nodeName. In Features choose the class attributes you want to display in the label, one attibute is suffcient; in our case this is name. Done.

For the arc, find the corresping Link Mapping, again in our example we specialized class Arc to class ArcToPlace; hence it also has the weight attribute. To add the label, add a Feature Label Mapping and set the corresponding properties, this time use the label defined for the arc.

4. Create the .gmfgen file. You can now refine the positioning of the arc label. Locate it in the tree. It will be in Gen Diagram <YourDiagram> / Gen Link <YourLink>EditPart / Gen Link Label <YourLink><Attribute>EditPart. Its sub-features control the distance of the label to the link (that you’ve just set above). The property Diagram Label / Alignment of the Gen Link Label allows you to specify where the label shall be positioned along the link (beginning, middle, end).

Unfortunately, such a refined definition is not available for external labels of nodes. You have to do this in the source code (see below).

5. Generate the source code. Your arc label is now fine, but you can fine touch your node label, specifically the distiance of the external node label to its node. Open the Edit Part source file of your node in <your-package>.diagram.edit.part, e.g. TransitionEditPart.java . Now locate the method addBorderItem() in this class. It is responsible for registering the external label within a certain distance at your node’s graphics. The statement
locator.setBorderItemOffset(new Dimension(-20, -20));
borderItemContainer.add(borderItemEditPart.getFigure(), locator);

specifies the distance of your label. Replace (-20,-20) by the values you like. I prefer (-5,-5) for a close distance.

That’s it. Run your editor and enjoy your labels.

This video shows a physical, self-organizing system.

These days a lot of people talk about such kinds of systems for information technology, called self-X systems. Self-X like self-healing, self-organizing, self-stabilizing. The self-X paradigm envisions software systems that come with some inherent dynamics to automatically keep a complex system in a “good” state while it is constantly under “bad” influences. A “bad” influence are request to a web server, the “good” state is that the server responds. State-of-the-art technology allows that a web server stays “good” as longs as there are not too many requests. In case of a (distributed) denial-of-service attack the server will no longer respond to each request; “bad” influence. A self-X web server would be able to deal with this by

• distinguishing “bad” from “good” requests,
• automatically providing additional capacity (in memory, CPU and network) for the “good” requests, and
• efficiently denying all “bad” requests with minimal effort.

This sound too good, to be true. And its obviously not easy to achieve – otherwise we would already have such kinds of systems everywhere. The automatic software updates your operating system and web browser performs these days is one of the first industrial applications of this idea. The skype login mechanism is another example. I guess you want more of this. That’s why a lot of people are researching self-X systems, mostly by thinking of clever algorithms, architectures, network technologies etc.

But the development of these systems requires some profound understanding of self-organization. The good news is that many disciplines have some concept of self-organizing system. The bad news is that these concepts are different although they have something in common. Of course, chemistry, physics and informatics cannot have the same definition of self-organizing system – each talks about different things. But this does not help when one tries to build such a system.

As a starting point, though, I think the five self-organizing metronomes are very illustrative. Let’s have a look on them again:

Each metronome on its own is a single entity that can swing if it is pushed. The movement of one metronome’s pendulum is independent of the other metronomes. That’s why the five metronomes swing non-synchronized in the beginning of the clip. Think of yourself trying to synchronize all of them. By stopping and pushing each metronome one after the other. You will most likely not succeed.

What then happens in the clip is that the metronomes are coupled. This coupling is key in this self-organizing system. The coupling happens by putting all metronomes on a common board and placing this board on two rolls such that the board can move in the direction of the pendulum movement. The metronome was not changed, or more precisely, the mechanism of the pendulum was not changed. But by the laws of physics, the movement of the pendulum applies a force to the ground of the metronome, which is the board. So now, the metronomes are coupled – each applies “its” force to the board – the sum of these forces pushes the board, which moves, which pushes the metronomes. As a result each pendulum is “pushed” by the others. Instead of five independent metronomes we got five coupled metronomes.

Now by another law of physics, the entire system seeks an energetically optimal point. This optimal point lies in a synchronization of the moving parts such that the entire system looses as few energy as possible. If one pendulum is out-of-sync with the others, its movement will dampen the movement of the others and vice versa, and dampening means loss of energy. Actually, because of dampening, the system gets into a synchronized state at all. In the worst case, all metronomes have to stop to get the system synchronized. But luckily, each metronome can give away energy on its own (each metronome has its own dampening mechanism). Therefore, the system can reach a state where each pendulum has been dampened such that it is no longer dampened by the movement of the other metronomes.

I am no physicist, so I am not aware of all details of this process, nor can I explain them appropriately. But what I can say for sure is that the synchronization was achieved by two things:

1. The coupling mechanism allows communication (i.e exchange of energy) between the parts that shall be synchronized.
2. Each part is designed in a way that the part can be influenced via the coupling mechanism, most importantly by push and repulsion.

The difficulty in computer science is that we have to find such coupling and interaction mechanisms in a world where there are no laws like the laws of physics – well, not yet.

I just want to wrap up how I managed to recover the data from a laptop’s damaged hard disk drive. The 40GB disk was formatted with FAT32, had a Windows XP and some serious I/O error problems preventing it from booting.

I cleaned up my external 80 GB HDD, plugged it via USB into the laptop and booted the system with the latest Knoppix-CD. I reformatted the entire disk (/dev/sda1) with the ext3 file system, mounted it with (# mkdir /media/sda1 ; mount -t auto /dev/sda1 /media/sda1) and gave full access to the entire disk (# cd /media/sda1 ; chmod 777 *).

Since Knoppix is missing the dd_rhelp script, I downloaded the latest package and put it on that external disk (into /media/sda1/dd_rhelp). The damaged disk in the laptop was /dev/hda1. I set disk-access parameters for that disk back to basic but safe, disabling DMA and higher level PIO mode access:
# hdparm -m 0 -d 0 -p 0 -A 0 -X 08 /dev/hda
This makes things horribly slow in copying the large chunks of good data, but it improved the overall process since it can handle bad blocks in a better fashion. Next, I edited the dd_rhelp script and set the values “max_bs=1024″ and “min_bs=64″ in /media/sda1/dd_rhelp/dd_rhelp. The standard values of dd_rhelp are fine, but slow down the copying of data if the disk is damaged to a larger degree.

Ready to copy the data. Change to root with “su” and then run
# /media/sda1/dd_rhelp/dd_rhelp /dev/sda1 /media/sda1/sda1_backup.img
This process makes a bit-per-bit copy of the entire disk and writes it to the image sda1_backup.img. Any bit that cannot be read by the program will be zero. This takes a while. Quite a while. For my 40GB about a week. Which had nothing to do with the hdparm above but with the overall state of the disk (I ran the process twice). You may kill the dd_rhelp and the dd_rescue processes at any time and continue later on. dd_rhelp writes a log file that keeps track of the overall progress that has been made.

When you are done make a copy of the rescued image /media/sda1/sda1_backup.img and get yourself another hard drive that is at least as big as the image you just rescued or make some space on another hard drive. You will have to create a new FAT32 partition that is at least as big as the image. If the file system of the disk from where you just rescued the data has file system X, create a new X partition of that size. In my case, that new partition mounted as /media/sdb6. Now, copy the data from the image to that partition with
# dd if=/media/sda1/sda1_backup.img of=/dev/sdb6
Mounted /dev/sdb6 and saw the data. You can now run any file system/data/rescue operations on that disk (or the image file) or copy what’s left of it.

Update: compiled some data in a How To: Recover Windows Partitions.

So this was one of the weeks where I learned Linux configuration the hard way. I simply thought of getting my virtual server up to the most recent version. It was/is running a Debian 3.1 (Sarge), I guess. You’ll probably realize by now that I know a lot about my server…

Anyway, I simply ran
apt-get upgrade
which brought about 20+ packages to the most recent version. Including exim, my mail agent, and clamav, a virus-scanner that runs with mail agents. I guess because of a version change in either exim, or clamav, or both I was asked to do so configuration stuff which I followed to my best knowledge and guess. However, I ended up with bouncing mails to my server. After a while, I found out that clamav got a version change, but the configuration on the system didn’t go with it. The exim4 logfile /var/log/exim4/paniclog kept telling me malware acl condition: clamd: unable to connect to UNIX socket /var/run/clamav/clamd.ctl (Connection refused) While exim4 was already properly calling clamav, clamav refused to answer.

A little bit of search with that phrase revealed the following page http://koivi.com/exim4-config/ which was quite instructive to me to some degree. There was a problem with the access privileges to the clamav runtime files from exim4. So gave some privileges:
adduser clamav Debian-exim
which was already done before. I then checked that /etc/clamav/clamd.conf contains a line that reads:
AllowSupplementaryGroups
Done. Then I gave permissions for the /var/run/clamav directory to allow for the correct user to use it:
chown Debian-exim.Debian-exim /var/run/clamav
Since /var/run/clamav/ contained a file freshclam.pid I also ran
chown Debian-exim.Debian-exim /var/run/clamav/freshclam.pid
Now a restart with
/etc/init.d/clamav-daemon restart
gave me an error that AllowSupplementaryGroups in /etc/clamav/clamd.conf requires a BOOL value. Luckily Ping! from the ClarkConnect Forums pointed out that

In my /etc/clamd.conf file, there was an error in every line that required a BOOL value. I had to add “yes” or “1″ to the followin options to get the clamd to succeed

I did that to each line of /etc/clamav/clamd.conf that had no argument next to it and the
/etc/init.d/clamav-daemon restart
succeeded. However, I still got bouncing mails and /var/run/clamav/clamd.ctl was still missing. It appears that clamav and exim4 just need the existence of that file, no specific content, plus proper access rights. A
touch /var/run/clamav/clamd.ctl
chown Debian-exim.Debian-exim /var/run/clamav/clamd.ctl
/etc/init.d/clamav-daemon restart
did the trick. Thanks to aioshin.

update Well… that didn’t help… therefore I did it the hard way:
apt-get --purge remove clamav clamav-base clamav-daemon clamav-freshclam libclamav1
apt-get -t sarge install clamav clamav-daemon
adduser clamav Debian-exim
reboot
Thanks to D.J.Fan. After the apt install, I told the setup to take the package maintainer’s version of the configuration file. A choice I made wrong earlier… grrr.

The guys at Skype must have had a tough night. My client logs in again and I just saw about 2.8 million nodes can do the same. Still I just popped out again, so it’s not been solved yet entirely.

even though it is too early to call out anything definite yet we are now seeing signs of improvement in our sign-on performance. We continue to monitor the situation through Europe waking up this morning and have more for you in approximately 4 hours.

My client is now occasionally bringing in stacks of unread multichats. Hope yours too.

from http://heartbeat.skype.com/2007/08/looking_slightly_better_at_070.html

I’m still up with my initial guess that it has nothing to do with the client code, since we did not have to update the clients (unless Skype has some silent module update built-in). Must have been something with the supernodes. It’ll be interesting to see how the network recovers slowly during the day.

Anyway, it was fun to participate in blogging about something ongoing. Though I hardly get the point as it’s quite some sort of procrastination… I better get back to my work. Speaking of which: I have to write a journal article.

I knew for quite some time that I wouldn’t get my PhD without a profound caffeine addiction. Now, I’m quite a step closer to my degree: I am 81% Addicted to Coffee, found at 85m²

This is a follow-up to my earlier post on “Skype Network Problems“. I’ve just remembered an analysis of the Skype P2P Protocol from the Columbia University that might give some insight on what’s wrong. The subsequent quotes are taken from: “An Analysis of the Skype Peer-to-Peer Internet Telephony Protocol“, Salman A.Baset, Henning Schulzrinne, 2004, Columbia University

Login is perhaps the most critical function to the Skype operation. It is during this process a [skype client] authenticates its user name and password with the login server, advertises its presence to other peers and its buddies, determines the type of NAT and firewall it is behind, and discovers online Skype nodes with public IP addresses. We observed that these newly discovered nodes were used to maintain connection with the Skype network should the [super node] to which [the skype client] was connected became unavailable.

Each skype client can become a super node if it has sufficient CPU cycles, memory and bandwidth to spare. The super nodes form a fully meshed network (each super node knows the IP of each other super node that is logged on).

We observed that a SC must establish a TCP connection with a [super node] in order to connect to the Skype network. If it cannot connect to a super node, it will report a login failure.

So, if there is a problem with reaching super nodes in general, the whole network goes down. Now since a skype client can connect to the network once in a while, there is no general failure of the super nodes, but appearantly, we get disconnected from all neighboring super nodes after a short period of time.

After a [skype client] is connected to a [super node], the [skype client] must authenticate the user name and password with the Skype login server. The login server is the only central component in the Skype network.

As we can log on to the network for a couple of minutes, the problem is unlikely to be with the login server alone, although they constitute the single point of failure in the network. (otherwise, we couldn’t log in at all like when entering an unknown username/password combination). Finally, the entire super node mesh network us bootstrapped by seven hard-coded super node IPs.

After logging in for the first time after installation, [the host cache] was initialized with seven IP address and port pairs. We observed that upon first login, [the host cache] was always initialized with these seven IP address and port pairs except for a rare random occurrence. In the case where [the host cache] was initialized with more than seven IP addresses
and port pairs, it always contained those seven IP address and port pairs. It was with one of these IP address and port entries a [skype client] established a TCP connection when a user used that [skype client] to log onto the Skype network for the first time after installation. We call these IP address and port pairs bootstrap super nodes.

Consequently, there seems to be a problem that certainly involves the bootstrap super nodes as a client always tries to reach these to connect to the network. I believe that the network suffers a combination of unstable state of the overlay network and high loads on the (bootstrap) super nodes.

Since this morning, I’m experiencing some severe Skype network problems. The client disconnects regularly, needs about 5 minutes to reconnect and drops the connection quite soon again (about a minute later). Likewise, the number of client that are logged in has dropped to about 2.5 million. Which also means that the p2p network is just about half the size that it’s used to be. If a good portion of the Skype users experiences the same problems, I guess the network is not present at all as any joining node leaves the network quite fast again.

One might draw the conclusion that although there is a 2.5 million node network, it is more of a virtual kind, as it might be fast changing due to failing nodes and missing links. This behavior is actually quite close to ad-hoc networks. I’m interested to see what the Skype guys find out about the cause for this behavior: http://heartbeat.skype.com/2007/08/problems_with_skype_login.htm (thanks to AvionarU for the link)

[edit 14:06] Ok, here’s an educated guess about what might be going on: It appears to me that the network gets fragmented. Nodes (Clients) get disconnected or disconnect themselves for some reason. A node getting disconnected also disconnects other nodes from the network, especially if they are one of the heavy duty nodes that constitute the hubs of the Skype network. Then millions of nodes try to reconnect to other hubs, causing quite some load to them and maybe causing them to go down again. It could be some sort of vicious circle.

[edit 14:31] That’s a new record: my client get’s to know about 509.071 connected clients in the Skype network. Looks like there are quite some people that have problems connection to the network. Quoting Skype heartbeat:

UPDATED: Some of you may be having problems logging in to Skype. Our engineering team has determined that it’s a software issue and hopes to have this resolved quickly. Meanwhile, you can simply leave your Skype client running and as soon as the issue is resolved, you will be logged in. We apologize for the inconvenience.

[edit 15:22] Skype Heartbeat says: “We expect this to be resolved within 12 to 24 hours.”

Maybe this is ridiculous, but I never remember the program calls for using GNU autotools. And the Google search always gives me lots of information I never wanted to know. It’s just the order of calls, so here we go:

autoheader
aclocal
autoconf
automake ----add-missing
./configure
make

Fine. Now I have a place where I can look things up.

edit: occasionally (read: in case checking out sources from a CVS under cygwin) the file format is wrong. Put a
dos2unix *before the first line above.