algernon @ BalaBit

A little while I ago, I had a vision. Before my eyes, I saw swirling log messages, rushing through space from a host of sources towards many destinations, and I was caught in the middle, observing this traffic, that’s normally hidden from the human eye, as it happens on the wire. Ever since, I was planning to figure out a way to make this vision reality. Today, while I’m not quite there yet, I do have something interesting to show!

I got side-tracked the other day, and figured that while writing the tool that’d implement the original dream, I can come up with something that’s still interesting, and much much easier to do: feeding my logs to Gource, a version control visualisation tool! So I took my logs, as stored in MongoDB (via syslog-ng), wrote a little tool, that extracts the data from the database, and turns it into a format that Gource understands, and behold: Twelve Days of Log was born. It’s not all that spectacular yet, there’s a lot of knobs to turn and tune still, but it’s a start. I will make another, much longer video once all my logs are exported (after about 16 hours, it’s around 35% done!), perhaps even two, and just maybe, I’ll mix them together for fun.

And for extra goodness, if one does not want to export old logs, and is fine with only pushing new ones to Gource, that can be done right here, right now, without any external tool:

destination d_gource {
 file("/var/cache/gource/logs.src"
  template("$UNIXTIME|$HOST@$SOURCE|M|$HOST/$PROGRAM/messagesn"));
};

And once it has a fair amount of logs (a day or two is a good start), this can be turned into a neat animation with a commandline like the following:

$ gource --log-format custom --highlight-all-users 
   --file-idle-time 5 --user-friction 0.25 
   --bloom-intensity 0.25 --bloom-multiplier 0.25 
   --hide filenames,dirnames 
   -e 2 --user-scale 0.5 
   /var/cache/gource/logs.src

The video above was made using a similar command too – only slightly different. Now imagine what happens when there are more logs, more sources and more hosts! What a swirling, jupming, twisting and turning fun it must be!

The code to turn logs stored in MongoDB into gource-ready format is available from the syslog-ng module collection git repository, for anyone who may want to play with this. Though, it easier to just spend a day or two exporting new logs into a gource-compatible format.

Update: Three Months of Log finished rendering, and is now available. There is no music for this one, however, and you’re all advised to not watch it in one sitting.