Comments on: Splunk Your Distributed Logs in EC2 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/ A goal is a dream with a deadline. Fri, 29 Aug 2008 03:28:59 +0000 http://wordpress.org/?v=2.5.1 By: links for 2008-07-24 « Brent Sordyl’s Blog http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-107022 links for 2008-07-24 « Brent Sordyl’s Blog Thu, 24 Jul 2008 14:31:45 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-107022 [...] Splunk Your Distributed Logs in EC2 Managing logs is hard as it is, and now imagine you have several (dozen) servers in EC2, the process becomes a chore, the debugging is hard, and frustration abounds. (tags: ec2 logging deployment) [...] [...] Splunk Your Distributed Logs in EC2 Managing logs is hard as it is, and now imagine you have several (dozen) servers in EC2, the process becomes a chore, the debugging is hard, and frustration abounds. (tags: ec2 logging deployment) [...]

]]> By: People Over Process » links for 2008-07-05 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-105128 People Over Process » links for 2008-07-05 Sat, 05 Jul 2008 07:37:49 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-105128 [...] Splunk Your Distributed Logs in EC2 (tags: splunk ec2 logmanagement cloud itmanagement redmonkclients) [...] [...] Splunk Your Distributed Logs in EC2 (tags: splunk ec2 logmanagement cloud itmanagement redmonkclients) [...]

]]> By: Ilya Grigorik http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104548 Ilya Grigorik Tue, 01 Jul 2008 00:38:55 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104548 You had me at <a href="http://www.splunkbase.com/apps/All/Technologies/app:Splunk+Replay" rel="nofollow">Splunk Replay</a>, and <a href="http://www.splunkbase.com/apps/All/Business_Intelligence/app:Splunk+Globe" rel="nofollow">Splunk Globe</a> took the cake. Wasn't aware of all the additional apps available for Splunk! You had me at Splunk Replay, and Splunk Globe took the cake. Wasn’t aware of all the additional apps available for Splunk!

]]> By: Michael Wilde http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104219 Michael Wilde Wed, 25 Jun 2008 04:17:50 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104219 Ilya and readers... I just finished up a post which includes a longer form video called "Splunk Ninja - Inside the Cloud" I cover everything from CLI, Elasticfox, Rightscale, CloudStatus.com and a few new "Applications" on top of Splunk. Check it out and keep the conversation going -- I'm game! http://blogs.splunk.com/thewilde/2008/06/24/splunk-ninja-inside-the-cloud/ Ilya and readers… I just finished up a post which includes a longer form video called “Splunk Ninja - Inside the Cloud” I cover everything from CLI, Elasticfox, Rightscale, CloudStatus.com and a few new “Applications” on top of Splunk. Check it out and keep the conversation going — I’m game!

http://blogs.splunk.com/thewilde/2008/06/24/splunk-ninja-inside-the-cloud/

]]> By: splunk on ec2 — award tour http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104196 splunk on ec2 — award tour Mon, 23 Jun 2008 23:33:07 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104196 [...] Splunk Your Distributed Logs in EC2 - igvita.com. splunk always struck me as a cool app. [...] [...] Splunk Your Distributed Logs in EC2 - igvita.com. splunk always struck me as a cool app. [...]

]]> By: links for 2008-06-21 « Mike Does Tech http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104155 links for 2008-06-21 « Mike Does Tech Sat, 21 Jun 2008 00:31:32 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104155 [...] Splunk Your Distributed Logs in EC2 - igvita.com (tags: ec2 logging rubyonrails splunk) [...] [...] Splunk Your Distributed Logs in EC2 - igvita.com (tags: ec2 logging rubyonrails splunk) [...]

]]> By: Carl Mercier http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104141 Carl Mercier Fri, 20 Jun 2008 13:09:43 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104141 For the record, we ran plenty of disk benchmarks for a talk Morgan Tocker and I gave in April. Here's the beef: http://tinyurl.com/5gg55t For the record, we ran plenty of disk benchmarks for a talk Morgan Tocker and I gave in April. Here’s the beef: http://tinyurl.com/5gg55t

]]> By: Michael Wilde http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104140 Michael Wilde Fri, 20 Jun 2008 12:48:44 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104140 @Carl Mercier - I use XL's for pure performance. Having all of those cores and memory makes for a pretty darn fast search, but let me break performance down for you. Architecture matters -- big time (well depending on how much data you have). When Splunk stores data we organize it by time. on 32-bit systems, the size of time bucket we can go to is about 200MB per bucket vs. on a 64-bit system, the buckets can be up to 10GB. 64-bit can handle so much more when reading. If you have a huge amount of data over some determinate amount of time on a 32-bit system, we're gonna have to open a boatload of files to find your data vs on a 64-bit which we wont. Indexing = CPU intensive operation. Splunk will only open 1 thread by default for indexing, but can be configured to consume more threads and memory depending on your data rate. First non-founding employee Brian Murphy has a great video on perf which i refer to alot. http://www.splunk.com/article/2183 Search - Architecture matters here as I mentioned above, but so does Disk I/O. I have found that AWS is actually better than VMWare when it comes to Disk I/O (but not by much). Wanna make a Splunk win on Expert in Guitar Hero, give it RAIDed 15k RPM SAS drives, as search is a disk intensive activity. I have a Penguin Relion dual quad-core's running at 3ghz, 16GB RAM and those fast disks I just mentioned -- i should be able to get about 300GB per day indexing rate. Personally, I haven't done any scientific level benchmarking on AWS yet--our dev group is starting a QA project on EC2 (which i love!). To benchmark disk I/O use the Bonnie++ tool (or just do some heavy duty GZIPing to see max thruput on the most intensive disk operations). To benchmark indexing, use the techniques in Brian Murphy's video. @Carl & @Ilya - Persistent storage is a problem so far on EC2, what i have done in some instances is use PersistentFS which is a FUSE based filesystem that lets you mount S3 and through a syncing process, files are copied to and from S3. I have used it for backup, and could use it for an archive in the "coldToFrozen" roll (see docs). You don't want to have Splunk index/search data on that type of volume--not sure if it'll even work because we do have FS-specific locking code we use -- but nonetheless speed would by way slow. @Carl Mercier - I use XL’s for pure performance. Having all of those cores and memory makes for a pretty darn fast search, but let me break performance down for you.

Architecture matters — big time (well depending on how much data you have). When Splunk stores data we organize it by time. on 32-bit systems, the size of time bucket we can go to is about 200MB per bucket vs. on a 64-bit system, the buckets can be up to 10GB. 64-bit can handle so much more when reading. If you have a huge amount of data over some determinate amount of time on a 32-bit system, we’re gonna have to open a boatload of files to find your data vs on a 64-bit which we wont.

Indexing = CPU intensive operation. Splunk will only open 1 thread by default for indexing, but can be configured to consume more threads and memory depending on your data rate. First non-founding employee Brian Murphy has a great video on perf which i refer to alot. http://www.splunk.com/article/2183

Search - Architecture matters here as I mentioned above, but so does Disk I/O. I have found that AWS is actually better than VMWare when it comes to Disk I/O (but not by much). Wanna make a Splunk win on Expert in Guitar Hero, give it RAIDed 15k RPM SAS drives, as search is a disk intensive activity. I have a Penguin Relion dual quad-core’s running at 3ghz, 16GB RAM and those fast disks I just mentioned — i should be able to get about 300GB per day indexing rate.

Personally, I haven’t done any scientific level benchmarking on AWS yet–our dev group is starting a QA project on EC2 (which i love!). To benchmark disk I/O use the Bonnie++ tool (or just do some heavy duty GZIPing to see max thruput on the most intensive disk operations). To benchmark indexing, use the techniques in Brian Murphy’s video.

@Carl & @Ilya - Persistent storage is a problem so far on EC2, what i have done in some instances is use PersistentFS which is a FUSE based filesystem that lets you mount S3 and through a syncing process, files are copied to and from S3. I have used it for backup, and could use it for an archive in the “coldToFrozen” roll (see docs). You don’t want to have Splunk index/search data on that type of volume–not sure if it’ll even work because we do have FS-specific locking code we use — but nonetheless speed would by way slow.

]]> By: Ilya Grigorik http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104139 Ilya Grigorik Fri, 20 Jun 2008 11:15:26 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104139 Bryan, you can certainly redirect all your logs into one file, but that's not what has me excited. For me, it's the ability to aggregate logs from multiple hosts, and the ability to search them intelligently from the comfort of my web browser. Kord, thanks for addressing the questions! Michael, thanks for the video reply. Didn't know about transactions and have to agree, that is incredibly powerful. I've added your reply at the bottom of the post. Carl, those are great questions, I'll defer to Michael / Kord to give us the answers (pinged them). Having said that, for the S3 mirror, Splunk does not do this natively, but it wouldn't be a hard thing to setup. Bryan, you can certainly redirect all your logs into one file, but that’s not what has me excited. For me, it’s the ability to aggregate logs from multiple hosts, and the ability to search them intelligently from the comfort of my web browser.

Kord, thanks for addressing the questions!

Michael, thanks for the video reply. Didn’t know about transactions and have to agree, that is incredibly powerful. I’ve added your reply at the bottom of the post.

Carl, those are great questions, I’ll defer to Michael / Kord to give us the answers (pinged them). Having said that, for the S3 mirror, Splunk does not do this natively, but it wouldn’t be a hard thing to setup.

]]> By: Carl Mercier http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104130 Carl Mercier Fri, 20 Jun 2008 03:30:20 +0000 http://www.igvita.com/2008/06/19/splunk-your-distributed-logs-in-ec2/#comment-104130 Forgot... 1. What kind of EC2 machine is recommended for Splunk? An XL seems to be overkill for something like this. How far will a small get me? 2. Are the logs mirrored on S3 automatically, or they're simply kept on the local disk (ie: /mnt) 3. How many logging entries per second can I expect? Has it been benchmarked? Thanks! Great article! Forgot…

1. What kind of EC2 machine is recommended for Splunk? An XL seems to be overkill for something like this. How far will a small get me?

2. Are the logs mirrored on S3 automatically, or they’re simply kept on the local disk (ie: /mnt)

3. How many logging entries per second can I expect? Has it been benchmarked?

Thanks! Great article!

]]>