algernon @ BalaBit

A few months ago, I wrote a piece about syslog-ng and disk buffers, wherein I mentioned RabbitMQ, and sounded a gentle call for help. I do not know whether it was that, or something else, but help arrived.

A week or so ago, a new destination driver was submitted for review: one that allows syslog-ng to publish messages to AMQP, in a very straighforward and flexible way. While this is very different to what I imagined and outlined in my earlier post, it achieves almost the same thing.

I will not explain what and how the driver does in detail here, that is for another post. Instead, I’ll take a look at the differences between the disk buffering the AMQP driver provides, and the disk buffering available in the commercial edition of syslog-ng.

Let’s have a quick overview of some of the most important properties first!

As one can clearly see, while the AMQP driver does provide some kind of disk buffering, that is not its strongest point – nor should it be. For the purpose of disk buffering, syslog-ng PE wins in pretty much every possible way: it’s easier to configure, one has more control over it, it performs much better.

In summary, the AMQP driver provides a much needed way to publish structured messages, and allow very flexible consumer-side configurations, where consumers can come and go, subscribe to parts of the published stream, without having to change a single bit on the publishing side. That is the big advantage of AMQP. The fact that it can persist its queues to disk is a useful thing, but does not in any way compete against syslog-ng PE’s disk buffering solution, which was built to provide stable, reliable performance, and cater to the specific needs of disk buffering.

The commercial version of syslog-ng, called the Premium Edition offered support for disk buffers for a long long time now. This is something that the Open Source Edition is lacking even now, while there is a clear demand for it, and competitors support something very similar aswell.

While I won’t go into the whys, let me explain a few ideas people came up with to implement disk buffers, their shortcomings, and plans currently in motion that will ease the task in the not too distant future.

The Goal

The goal we want to accomplish, is to have a reliable way of storing messages that are still in flight, being processed by syslog-ng, on disk. There are two reasons we’d like to do this:

1. To ensure that messages do not get lost, not even when syslog-ng unexpectedly stops
2. To have a larger buffer for peaks, when storing all pending messages in memory would be unfeasible.

The problem

The main issue at hand is not only that syslog-ng OSE does not have a disk buffer, but that it’s very far from trivial to add this feature to it. And that is because the message queue system in OSE is not as extensible as it is in PE: you can’t just hook in a disk buffering system and live happily ever after.

Would that be the case, OSE would already have a disk buffer solution, as coding up a simple one is about a day’s worth of work.

The workarounds

Use a buffer file!

One of the first things people think of is: “What if I write to a file, and use that as the source for another destination?“. This could be as simple as:

source s_network { tcp(flags(no-parse)); udp(flags(no-parse)); };
destination d_buffer { file("/var/cache/syslog-ng/buffer.log"
                       template("$MESSAGE\n")); };
log { source(s_network); destination(d_buffer); };

source s_buffer { file("/var/cache/syslog-ng/buffer.log"
                       follow-freq(0.1)); };
destination d_net { tcp("1.2.3.4"); };
log { source(s_buffer); destination(d_net); };

This has a few drawbacks, however.

• The buffer will grow indefinitely.
• The buffer will need to be frequently polled to keep latency low.
• The configuration is far longer than one would want it to be.

Use a buffer file, with rotation!

To solve the issue of the file growing forever and ever, we could, of course, rotate it away from time to time. But what would we gain from that? Two new problems!

We still can’t expire old files, not easily, not reliably anyway, as we do not know which files have been fully consumed by the s_buffer source.

And, we added extra complexity to the config, because we now have a large number of buffer files, and we need to figure out when to start polling new ones. If we use an external rotation mechanism, we could SIGHUP syslog-ng after tweaking the config to use the new file, but then we have a window where messages in the old buffer haven’t been processed yet.

Or we could use wildcard sources (once they’re ported from PE to OSE), which makes the following of buffer files easy:

destination d_buffer { 
  file("/var/cache/syslog-ng/buffer-${HOUR}.log"
       template("$MESSAGE\n")); 
};
source s_buffer { file("/var/cache/syslog-ng/buffer-*.log"
                       follow-freq(0.1)); };

However, we still can’t expire old files reliably, as we have no way of knowing when a buffer was wholly consumed.

Lets parse syslog-ng.persist!

Thankfully for us, syslog-ng stores what position it is standing at for all files it is using as a source. We just need a way to get that out of the persist file. Sadly, no tool exists yet that would allow us to do so.

Technically, if we could figure out the file position from the persist file, and we’d have wildcard sources, implementing disk-buffers would only take a couple of lines, and a cron job to clean up old stuff: find all files that are read fully and are older than a set limit, like, if it’s twice as old as the rotation interval, then it is a candidate for deletion.

Can we do better?

While the above solution will work nicely, that too, has a few drawbacks: it’s hacky, it needs longer configuration than one would ideally want, and it needs an external tool to clean up old buffer files. We can surely do better than that!

To do better, we need a way to extend – or replace – the queue mechanism used within syslog-ng. Doing this properly is a hard task, but most of it has been done already, it just needs to migrate from PE to OSE – a question of time and energy, really. I won’t go into more details about this right now, as it will happen, and as far as the disk queues are concerned, it’s not important to know how the queue extending/replacing system works under the hood.

So, for a moment, lets imagine that we already have a pluggable queue system, ok? We can completely replace the queues syslog-ng uses internally.

Now, take a step back, and think about what we use the queue system for: to pass messages around. Does that sound familiar? Of course it does. Let’s use a message queue then! Preferably one that already supports disk buffering, like RabbitMQ.

If we’d replace syslog-ng’s queue mechanism with such a queue system, we wouldn’t only gain disk buffers, but a host of other interesting things too, but that is for a different post. For now, lets see how a rabbitmq-based config would look like:

options {
  queue(rabbitmq);
  rabbitmq(durable(yes)
           delivery(persistent));
};

source s_net { tcp(); udp(); };
destination d_net { tcp(); };
log { source(s_net); destination(d_net); };

And that’s about it. “Where’s the disk queue size?” – you may ask, and rightfully so! Well, we replaced syslog-ng’s internal queue mechanism with rabbitmq, so log_fifo_size() is now the number of messages a particular queue will hold. There’s no separate disk buffer size setting in this case, as that’s entirely handled by RabbitMQ.

When will we have all this?

Wildcard source is already written, although it needs to be ported (and potentially rewritten in the process) from PE to OSE, it’s entirely possible to get this done for syslog-ng 3.4.

A tool to inspect persist files has been requested multiple times, but currently, there is no active effort on writing such a tool – help is, of course, welcome in this area. Nevertheless, implementing as much of this as we’d need for buffer file rotation is likely a very easy task, and as such, can be done in syslog-ng 3.4 too.

Extending the queue mechanism is available in PE, but I’d want something more: to replace the whole queuing system. That’s a huge task, and it won’t be coming soon. It is very unlikely that I’ll start working on this before 3.5 opens up for development.

To put it otherwise, the slightly hackish, not so nice solution is entirely possible to do within the scope of syslog-ng 3.4. The most powerful solution, that also buys us a whole host of neat things along with disk buffering, is something that will have to wait.

Can we help?

Of course! If you have some experience writing C code, then a persist file inspection tool likely won’t be a huge challenge for you – and I am always happy to help and give hints.

But even if you’re not a coder-type, or simply lack the time, expressing your interest, and showing support for our efforts of making syslog-ng better and better is also greatly appreciated!