Comments on: Distributed Logging: Syslog-ng & Splunk

By: Steve

Steve — Tue, 25 Nov 2008 21:59:56 +0000

Just a fix to Kent’s post - you had the address wrong (or it has changed):

http://developers.facebook.com/scribe/

By: edward

edward — Thu, 30 Oct 2008 19:32:01 +0000

is facebook’s open-souce dist. log solution better than spluks? Have anybody tried ?

By: Kent

Kent — Sat, 25 Oct 2008 00:06:29 +0000

Splunk is great but their licensing model is fundamentally flawed for encouraging broad adoption because you get financial PUNISHED heavily for using it for what it’s best at; aggregating and analyzing lots of log data. It’s always made me sad. I spoke to someone from Splunk back at the first CloudCamp SF and they said they were working on that. But, alas, nothing I’ve seen yet. I’ll have to follow up.

I just saw that Facebook released their dist. logging solution open source as well. But, I haven’t messed with it at all. Would be good to hear any feedback.
http://developer.facebook.com/scribe/

By: Ilya Grigorik

Ilya Grigorik — Fri, 24 Oct 2008 02:10:24 +0000

Dmitriy, there is an entire collection of generic *nix network loggers: syslogd, socklog, rsyslog (as Mark mentioned). Also, every language has a collection of logger implementations (Java, for one, seems to have a few too many). In Ruby land, a good one is: analogger (same guys that did swiftiply).

Having said that, in terms of actual log analysis, I’m not aware of any good alternative for log analysis. phpLogCon is one, but it’s not nearly as exciting.

By: Vysnu » Magnolia Post

Vysnu » Magnolia Post — Thu, 23 Oct 2008 07:04:06 +0000

[...] Developer News, Interviews and LinksRating: ★ ★ ★ ★ ★ Distributed Logging: Syslog-ng & Splunk - igvita.comRating: ★ ★ ★ [...]

By: Dmitriy

Dmitriy — Thu, 23 Oct 2008 02:06:39 +0000

Ilya,
Mind posting what you have in mind for distributed log aggregators other than Splunk?

By: Ilya Grigorik

Ilya Grigorik — Wed, 22 Oct 2008 17:52:52 +0000

Tobi, perhaps lower the logging level? Or, I’m not sure how Splunk licensing is structured, but theoretically, you could run multiple instances. Albeit, of course, you loose the ability to query all of the data if you go down that route.

Last but not least, it’s a hefty price tag, but it’s a perpetual license.

By: tobi

tobi — Wed, 22 Oct 2008 17:13:31 +0000

At our log volume we would pay more than 20k for splunk :-(

By: Ilya Grigorik

Ilya Grigorik — Wed, 22 Oct 2008 16:44:24 +0000

Ah, that’s awesome. Syslog-ng can also do TLS, but like Splunk, it requires a premium license to make that work. Thanks for the tip Mark.

By: Mark

Mark — Wed, 22 Oct 2008 16:04:50 +0000

If you look @ rsyslog it can also do ssl tunneled syslog, which is ideal if you have an ec2 cloud and an offsite syslog & splunk setup.