Comments on: Measuring & Optimizing I/O Performance

By: Future of RDBMS is RAM Clouds & SSD - igvita.com

Future of RDBMS is RAM Clouds & SSD - igvita.com — Mon, 07 Dec 2009 16:52:09 +0000

[...] The evolution of disks has been extremely uneven over the last 25 years: disk capacity has increased 1000x, data transfer speeds increased 50x, while seek and rotational delays have only gone up by a factor of 2. Hence, if we only needed to transfer several hundred kilobytes of data in the mid 80’s to achieve good disk utilization, then today we need to read at least 10MB of data to amortize the costs of seeking the data - refresh your memory on seek, rotational, and transfer times of our rusty hard drives. [...]

By: Mark Seger

Mark Seger — Tue, 30 Jun 2009 12:01:28 +0000

re overzealous monitoring: don’t forget underzealous monitoring ;-)

I’ve seen too many people run tests and simply measure the start/end time with no clue as to what’s going on the middle. I’ll bet that’s why most people never even knew there was a bug in the way disk I/O stats were calculated. It wasn’t until I did some collectl testing at 1 second samples and found the bug, reported it and got it fixed.

Whenever we do any benchmarking we always have collectl running in another window doing 1 second samples. It never impacts the measurements. I also highly doubt most other tools would either. If they do, they were poorly written.

When I wrote collectl I paid very close attention to overhead and if you’re really worried about overhead and feel its current load of <0.1% as a daemon doing 10 second samples is too high, you can always skip monitoring processes/slabs and a few other things and get it down to <0.01% or lower.

I’ve even used collectl at 0.1 second sampling intervals when wanting to take a look at cache effects during disk I/O, but don’t do that during benchmarking.

-mark

By: Ilya Grigorik

Ilya Grigorik — Mon, 29 Jun 2009 12:37:18 +0000

Domas, that’s a really good point. Currently playing with libaio - will report on any findings!

By: Domas Mituzas

Domas Mituzas — Mon, 29 Jun 2009 05:14:43 +0000

Great summary, though you missed a major suggestion once you see delayed I/O. Not buffering, not caching, but simply - parallel I/O. Async I/O, threads, etc - can all be used a lot to overcome underlying storage layer limitations.

I/O can be parallel at quite a few layers - block layer can merge requests, RAID write-behind caching can merge requests, file system can merge requests, and even disk can merge and reorder requests. Parallel is good. ;-)

By: andy.edmonds.be › links for 2009-06-26

andy.edmonds.be › links for 2009-06-26 — Sat, 27 Jun 2009 00:37:35 +0000

[...] Measuring & Optimizing I/O Performance - igvita.com (tags: performance iostat optimization io) This was written by andy. Posted on Saturday, June 27, 2009, at 1:37 am. Filed under Uncategorized. Bookmark the permalink. Follow comments here with the RSS feed. Post a comment or leave a trackback. [...]

By: Mark Carey

Mark Carey — Fri, 26 Jun 2009 16:39:00 +0000

Be careful with regular monitoring of disk io. iostat and tools like it are very expensive and are usually best left for initial profiling or tracking down problems related to io. I’ve seen more than once “bug” caused by overzealous monitoring.

By: Ilya Grigorik

Ilya Grigorik — Thu, 25 Jun 2009 03:45:25 +0000

Julien, I agree, I’d love to see a set of consistent benchmarks across all the cloud providers. Hmm!

Bill, great catch, sloppy wording on my part.

Alexander, dstat looks pretty nice. I’ve also been using collectl with great success, it looks to be very similar to dstat.

By: Some performance monitoring stuff that’s been on my radar lately - Daemian Mack

Wed, 24 Jun 2009 13:11:37 +0000

[...] Measuring & Optimizing I/O Performance using iostat… If IO performance is suspect, iostat is your best friend. Having said that, the man pages are cryptic so don’t be surprised if you find yourself reading the source. To get started, identify the device in question and start a monitoring process… [...]

By: Measuring & Optimizing I/O Performance - igvita.com « Netcrema - creme de la social news via digg + delicious + stumpleupon + reddit

Tue, 23 Jun 2009 21:15:31 +0000

[...] Measuring & Optimizing I/O Performance - igvita.comigvita.com [...]

By: Alexander

Alexander — Tue, 23 Jun 2009 20:17:23 +0000

You might want to checkout dstat. Here is an excerpt from the man page:

Dstat is a versatile replacement for vmstat, iostat and ifstat. Dstat
overcomes some of the limitations and adds some extra features.

Dstat allows you to view all of your system resources instantly, you
can eg. compare disk usage in combination with interrupts from your IDE
controller, or compare the network bandwidth numbers directly with the
disk throughput (in the same interval).