Comments on: Efficient Updates & Data Import in Rails

By: Edgar S

Edgar S — Wed, 16 Dec 2009 19:47:40 +0000

Hey I’m trying to use this method instead of modifying the “set multiple statements on” of MySQL client to true, but I’m having some trouble it seems the options (:on_duplicate_key_update) is not being picked up by the ar_ext, so I always get the duplicate key error from mySQL, any Ideas? what I need is a massive update of records, could be several thousands. Thanks in advanced.

By: Ilya Grigorik

Ilya Grigorik — Sun, 10 May 2009 14:59:48 +0000

Matthew, unfortunately I’m not aware of any function in MySQL which will give you that. One workaround I’ve seen people do when they needed this functionality is to keep an external counter for ID’s, increment it by required amount prior to insert and then insert the records with the ID values preset in the data - that way, you know what range your records occupy. It’s a little messy, but it works.

By: Matthew

Matthew — Fri, 08 May 2009 23:12:10 +0000

Thanks for the great article. A few people had posted about multi-row inserts that need id’s to join. When doing a Widget.import widgets, is there a way to get a list of the insert’ed id’s the mysql equivalent of last_insert_id for multi row insets?

By: … So We Built One :: ActiveRecord::Base.connection.execute(… multiple statements against a mysql db …)

Wed, 26 Nov 2008 06:23:48 +0000

[...] almost all cases it would be better to use ar-extensions with the duplicate key error update trick. Some colleagues of mine, who invented a plugin to invalidate find_by_sql unless you provide an [...]

By: Eric Methot

Eric Methot — Sun, 20 Jan 2008 11:23:03 +0000

Hi Ilya, I would really be interested in the results you get on real prod code whenever you get around to it. Thanks in advance, Eric

By: Ilya Grigorik

Ilya Grigorik — Sat, 12 Jan 2008 19:09:12 +0000

Michael, believe it or not, that should be it. The steps are: establish connection to database, load your items, import your items. After you execute Item.import, check your database — all of your records should be residing in there!

Eric, very interesting extension to ActiveRecord. I did a quick pass over your code and it’s a pretty neat, Ruby idiom specific, implementation. I’ll make sure to test-drive it with some production code in the future.

By: Eric Methot

Eric Methot — Sat, 12 Jan 2008 09:40:10 +0000

I put together a small plugin (multi-statement-query) that allows AR to take advantage of MySQL’s ability to handle such queries (with some limitations of course). It could be used as a complement to Zack’s extension allowing you to also do selects, updates and other insert statements in one go. I mention it here because this is one of the sites that I have stumbled upon when researching the plugin.

Cheers

By: Michael Staton

Michael Staton — Thu, 10 Jan 2008 00:25:29 +0000

Ilya, this all makes sense except for the steps left out for the noobs…. Where do I put this code? How do I execute it?

I got a huge file that goes, for example,
items = [Item.new(data hash), etc. ]
Item.import items

Now what?

By: Ilya Grigorik

Ilya Grigorik — Tue, 13 Nov 2007 07:43:01 +0000

Marc, unfortunately your example will require some manual ‘muck’. There is no direct way (that I’m aware of), to do a multi-value habtm import aside from running two separate multi-value imports.

As long as you know the correct mappings between ID’s, I would first load the parent table, and then import the children as a separate query.

By: Marc Busch

Marc Busch — Mon, 12 Nov 2007 10:57:19 +0000

Ilya, nice article. But I have a problem to understand how this can be used with habtm tables, i.e. two tables that are linked with a join table.

Example:
Parent and Child have a has_and_belongs_to_many relation. How can I utilize ar-extension’s import functionality for something like this:

1000.times {
parent.children.create(:par0 => “test0″, :par1 => “test1″)
}

For every single ‘create’ there are two(!) INSERTs in the database, one to ‘child’ and the other to the join table ‘chidren_parents’. If I use ‘import’ as described in your article, the join table will not get updated, right?

Can this be solved somehow?