Comments on: Cluster Monitoring with Ganglia & Ruby

By: Аниме

Аниме — Mon, 08 Mar 2010 00:39:26 +0000

Конечно, как все говорят, любопытное рядом! :)

By: Jonah

Jonah — Fri, 19 Feb 2010 01:26:29 +0000

We monitor over 1000 servers using ganglia - 30+ metrics per server. This means that our gmetad has to do 30,000+ updates/minute to the RRD files. We moved to SSD and things are working great. SSDs are getting cheap enough now to be a good alternative to running gmetad on ramdisk and possibly loosing your data.

By: Dmitriy

Dmitriy — Tue, 02 Feb 2010 15:35:50 +0000

Ilya, no, in my case I didn’t use topic exchange. Graphite native protocol is simple tuples (metric.name, value, timestamp), so I put these into AMQP message payload and used direct exchange. But implementing the same functionality using topic exchange to fit your publishing logic better and then do translation into graphite’s super-simple internal format should be trivially easy.

Install and setup were very easy and straightforward. Can’t comment on operations aspect, especially with tons of metrics and a lot of activity like in your case, because my use case was small.

By: Ilya Grigorik

Ilya Grigorik — Tue, 02 Feb 2010 06:03:52 +0000

Dmitriy, that’s really slick. So just to confirm, the metric hierarchy is inferred based on the routing key of the topic exchange you are publishing to? How did you find the actual setup and maintenance of Graphite?

Jan, indeed you’re right. That metric should be a uint16 to be in range. Great catch, updated the example.

By: links for 2010-01-31 « Stand on the shoulders of giants

links for 2010-01-31 « Stand on the shoulders of giants — Mon, 01 Feb 2010 07:02:04 +0000

[...] Cluster Monitoring with Ganglia & Ruby – igvita.com (tags: sysadmin monitoring ganglia cluster cloud igvita) [...]

By: Jan

Jan — Sat, 30 Jan 2010 22:32:08 +0000

Nice post. Your example gives an “unsigned 8 bit” a value of 7000… that would be out of bounds. Or am I missing what the type is for?

By: Dmitriy

Dmitriy — Sat, 30 Jan 2010 17:01:02 +0000

Nice post.

Another alternative to Ganglia may be Graphite. It was developed inside Orbitz.com and then open sourced.

Like Ganglia, it requires zero setup for new metrics. I wrote an addon to publish data into Graphite using RabbitMQ - http://github.com/somic/graphite-rabbitmq, and I am actually hearing they might be considering adding support for AMQP into their main tree. Publishing data to Graphite from Ruby via AMQP is obviously a piece of cake.

Unlike Ganglia, Graphite doesn’t use RRD and instead uses its own implementation called whisper. Whisper is designed to handle irregular updates better than RRD - check out this FAQ. It helps when graphing not only things that are always happening like disk usage, but also things that happen from time to time such as errors, abnormal conditions etc.

By: Ilya Grigorik

Ilya Grigorik — Sat, 30 Jan 2010 03:11:28 +0000

Carson, thanks for the tip. I’ve actually never seen OpenNMS before - it does look like a cross between Nagios and Ganglia. Having said that, after playing with the demo for last five minutes, I don’t think it would replace Ganglia - the flexibility of the graphing and metrics coverage in ganglia is hard to beat.

By: Carson McDonald

Carson McDonald — Fri, 29 Jan 2010 14:05:50 +0000

This is a nice addition to Ganglia. One of the monitoring systems you didn’t list is OpenNMS (www.opennms.org) and I’m wondering if it might be a better fit for doing alerts and graphing resources in one place instead of relying on Nagios and Ganglia separately.

By: Dylan

Dylan — Thu, 28 Jan 2010 17:42:20 +0000

Fantastic post Ilya… (as always!)