Algernon@BalaBit

Structured logging is awesome, it’s something I’ve been doing and pushing for ever since I wrote the MongoDB destination driver for syslog-ng back in late 2010. It’s what our $(format-json) template function helps one do, also available in syslog-ng for quite a while now.

This is one of the reasons I became interested in CEE, because it’s striving to standardise the JSON payload within a syslog message, while still maintaining backwards compatibility (among other things, but this is the part that interests me).

Since I’m very interested in this kind of thing, and I heavily build on syslog-ng’s ability to transform whatever text I send it to into well structured stuff, I also wrote a JSON parser for it, which was merged into 3.4 in October, 2011.

With these, combined with other features available to syslog-ng, one has been able to parse and emit (this latter part was already possible with syslog-ng 3.3, parsing is new in 3.4) CEE messages for months now, using a reasonably simple config:

source s_cee { tcp(); };
parser p_cee { json-parser(prefix(".cee.")); };
filter f_cee { match("@cee:" type(glob)); };
rewrite r_cee { subst("^@cee: *", "", value("MESSAGE")); };
destination d_json {
 file("/var/log/cee.json"
      template("$(format-json --key .cee.*)n"));
};
log {
 source(s_cee);
 filter(f_cee);
 rewrite(r_cee);
 parser(p_cee);
 destination(d_json);
};

This config accepts connections on tcp, using the legacy syslog protocol, and filters on messages whose message part starts with the “@cee:” cookie, then proceeds to remove that prefix, and parse the rest as JSON, and finally emit all the stuff it found to a file, in JSON format. We could also emit CEE payload too, using a destination like this:

destination d_cee {
 tcp(ip("192.168.1.1")
     template("@cee: $(format-json --key .cee.*)n"));
};

Of course, this kind of setup isn’t particularly useful, but it’s simple enough to show the flexibility syslog-ng provides. It’s generic enough that it can be used not only for CEE, but any kind of JSON parsing, in any part of the incoming data, no cookies are hard-coded.

It also has the benefit of not needing to hardcode any name-value pairs we want to transmit: we can just use a pattern. The parser is told to prefix every key with a string, and the emitter transfers only those same keys. It’s also possible to rewrite the key names before sending over, but that’s something I’m saving for another post in the very near future.

I’ve been using something similar in my home network: I collect and prepare logs locally, and send to a central location with a JSON payload (though, I just send raw JSON, without a cookie or any other protocol framing added), which parses it and puts it into MongoDB. An example of what I end up with can be seen here. Mind you, this example was made before I became aware of CEE, so the naming and structuring does not follow that, but as one can see, the logs are structured, and most of the information was extracted from the message part using patterndb (on the client, the server received the processed JSON only).

This setup has been keeping me happy for the past few months, since it reached the level of usability. Of course, there are still a few issues left, like being able to reconstruct nested structured, but that’s something I didn’t immediately need (as MongoDB does the neccessary transformation for me). Nevertheless, it’s something often requested, and as such, is under development, and will be ready as soon as I have a little time to work on it. Perhaps a week or two, perhaps less, it’s about a day worth of coding, another afternoon for writing tests and suitable documentation.

I’m also quite happy to see rsyslog following in our footstep, and realising that structured logging, and CEE is a good thing! Kudos to them, and I’m looking forward to test our interoperability!