Comments on: Subqueries and SQL Syntax Sugar Diet

By: Ilya Grigorik

Ilya Grigorik — Tue, 26 Aug 2008 12:39:18 +0000

Lynton, I think this may have been a transitional bug in one of the point releases. I’ve since tested this same scenario on newer versions (didn’t bother reverting back to old as control), and the problem seems to have gone away.

By: Lynton Morgan

Lynton Morgan — Mon, 25 Aug 2008 23:33:38 +0000

Did anyone confirm if these wild differences where down to storage engines or something else? No one who got the opposite result confirmed they were using Innodb

By: Ilya Grigorik

Ilya Grigorik — Tue, 23 Oct 2007 04:06:43 +0000

Ah, that could explain it! Thanks aleco, very interesting read.

By: aleco

aleco — Mon, 22 Oct 2007 14:09:31 +0000

It seems as if this is an example of mysql’s improper handling of optimization (seems as if 5.2 will fix it). Basically what happens is that MySQL will execute the outer statement for each item in the IN() list, therefor looping through your table 500 times instead of aggregating the 500 posts first. Plus, I think “IN” can’t always make use of the index, therefor adding dramatic slowdown.

There are some details regarding the optimization of IN/ANY queries here: http://dev.mysql.com/doc/refman/5.0/en/in-subquery-optimization.html

By: Ilya Grigorik

Ilya Grigorik — Sat, 22 Sep 2007 21:05:28 +0000

Fred, interesting results, thanks for sharing. It appears that the query optimizer is all over the map on different platforms / versions of MySQL.

By: Fred

Fred — Fri, 21 Sep 2007 12:39:39 +0000

forgot to format the code
here is the code I used

def time_query(&block) start = Time.now block.call puts "#{Time.now - start} seconds \n" sleep 1 end


ids = []

5000.times do |t|

  t=t+1

  ids < < t

end
# Syntax sugar galore :

time_query {
	result = User.find_by_sql("SELECT * FROM users WHERE id in (#{ids.join(',')})")

}
# SQL on a diet: vanilla UNION way

time_query {

	sql = ids.collect { |w| "(SELECT * FROM users WHERE id = #{w})" }

	sql = sql.join(" UNION ")

	result = User.find_by_sql(sql)

}

Cheers :)

By: Fred

Fred — Fri, 21 Sep 2007 12:34:58 +0000

I run these tests on my macbook, and the results were completely different..

The union query was actually 5 to 10 times *slower*

I made it to do with 5000 records:

def time_query(&block)
start = Time.now
block.call
puts “#{Time.now - start} seconds \n”
sleep 1
end

ids = []
5000.times do |t|
t=t+1
ids
RESULTS in Macbook:
mysql Ver 14.12 Distrib 5.0.41, for apple-darwin8.5.1 (i686) using readline 5.0

Using IN:
0.877374 seconds

Using UNION:
8.407131 seconds

RESULTS in Gentoo Linux Server:
mysql Ver 14.12 Distrib 5.0.44, for pc-linux-gnu (i686) using readline 5.2

Using IN:
1.158719 seconds

USING UNION
7.831463 seconds
it’s very weird to have these results.

I Also tunned my.conf to have higher tunned values.
Maybe that could be the reason.

By: Morning Brew #73

Morning Brew #73 — Mon, 10 Sep 2007 11:01:22 +0000

[...] Subqueries and SQL Syntax Sugar Diet - unexpected SQL optimization. [...]

By: Ilya Grigorik

Ilya Grigorik — Fri, 07 Sep 2007 22:02:27 +0000

Brian, I’ve had the same experience with MySQL as well. Syntax sugar and optimizer don’t seem to play all that well at the moment.

Matt, I just tested both queries independently (without ActiveRecord) on cold-start mysqld instances: same results as above. I wonder if I stumbled across a platform/version dependent bug. I’ll try to dig deeper into it over the weekend and see what I can uncover.

The joys of SQL. :)

By: Matt

Matt — Fri, 07 Sep 2007 19:06:14 +0000

Re: Chris’s comment above. Yeah, that was my thought when I first saw this too. Ilya, any chance you could try switching the order of the finds around and re-testing to put our minds at ease?

My hunch is that there’s something else going on here, and the fact that you’re arriving at your results through ActiveRecord is only serving to cloud the matter. I tried a quick test directly in the MySQL client with a set of 1000 ids, but my queries were completing in under a second so the differences weren’t significant enough to call.