Comments on: Loading Netflix Dataset into SQL http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/ A goal is a dream with a deadline. Thu, 20 Nov 2008 20:07:49 +0000 http://wordpress.org/?v=2.5.1 By: Ilya Grigorik http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-103825 Ilya Grigorik Fri, 06 Jun 2008 12:07:24 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-103825 Awesome, thanks for sharing the tips Larry! Awesome, thanks for sharing the tips Larry!

]]> By: Larry Freeman http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-103603 Larry Freeman Sun, 01 Jun 2008 00:15:31 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-103603 Hi, I just discovered your web site. I really like it! :-) I just started a blog myself which will focus on analyzing algorithms. I've started with the algorithm that won the Netflix Progress Prize. At this point, my approach is pure SQL on MySQL with very minor help from awk and java. I've hacked a script for loading up all the Netflix data with indexing in 3 hours. Here are the <a href="http://setupandconfig.blogspot.com/2008/04/loading-up-netflix-prize-data-into.html" rel="nofollow">details</a> I've also used this script to reproduce the 11 global effects reported in the BellKor paper. It's <a href="http://algorithmsanalyzed.blogspot.com/2008/05/bellkor-algorithm-global-effects.html" rel="nofollow">here</a> Cheers, -Larry Hi,

I just discovered your web site. I really like it! :-)

I just started a blog myself which will focus on analyzing algorithms. I’ve started with the algorithm that won the Netflix Progress Prize. At this point, my approach is pure SQL on MySQL with very minor help from awk and java.

I’ve hacked a script for loading up all the Netflix data with indexing in 3 hours. Here are the details

I’ve also used this script to reproduce the 11 global effects reported in the BellKor paper. It’s here

Cheers,

-Larry

]]> By: Ilya Grigorik http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-103429 Ilya Grigorik Thu, 22 May 2008 10:55:16 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-103429 Rasha, I've heard that Netflix changed their data files slightly (I believe they released more data since this was posted). Hence, it's not surprising that my transform script is not working as well as it should. More likely than not, it's the regular expression that's outdated - take a close look at your data files and how the data is being extracted. Rasha, I’ve heard that Netflix changed their data files slightly (I believe they released more data since this was posted). Hence, it’s not surprising that my transform script is not working as well as it should. More likely than not, it’s the regular expression that’s outdated - take a close look at your data files and how the data is being extracted.

]]> By: Rasha http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-103391 Rasha Tue, 20 May 2008 10:49:01 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-103391 I have tried this over 10 times, the transform script creates all 17000 files in data-load folder but they are all empty!!! :( I have tried this over 10 times, the transform script creates all 17000 files in data-load folder but they are all empty!!! :(

]]> By: Ilya Grigorik http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-58392 Ilya Grigorik Tue, 07 Aug 2007 12:51:20 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-58392 Francisco, looks great. Good luck! Francisco, looks great. Good luck!

]]> By: Francisco Marco-Serrano http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-58390 Francisco Marco-Serrano Tue, 07 Aug 2007 12:43:59 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-58390 Netflix Prize for Dummies. ==================== Hi, I'm quite limited in programming and databases so I'm using the Prize as my Quest for learning. I'm documenting here: http://blogs.kproductivity.com/fmwaves/category/culture/movies/netflix/ PS The dummy is ME! Netflix Prize for Dummies.
====================

Hi, I’m quite limited in programming and databases so I’m using the Prize as my Quest for learning. I’m documenting here: http://blogs.kproductivity.com/fmwaves/category/culture/movies/netflix/

PS The dummy is ME!

]]> By: Ilya Grigorik http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-35349 Ilya Grigorik Mon, 30 Apr 2007 14:27:42 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-35349 Peter, thanks for the link. That looks like a nice way to go about it! Peter, thanks for the link. That looks like a nice way to go about it!

]]> By: Peter http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-35026 Peter Sun, 29 Apr 2007 17:31:35 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-35026 Hi, i just found an even better way which avoids the unnecessary creation of the temporary files. http://www.juretta.com/log/2007/04/24/loading_the_netflix_dataset_into_mysql/ Peter Hi,

i just found an even better way which avoids the unnecessary creation of the temporary files.

http://www.juretta.com/log/2007/04/24/loading_the_netflix_dataset_into_mysql/

Peter

]]> By: Ilya Grigorik http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-25912 Ilya Grigorik Fri, 16 Mar 2007 15:27:47 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-25912 Ben, my import went really fast, so it must be something with the DB configuration. A couple of thoughts: 1) Set your 'max packet size' to something much higher than the alloted ~1mb. 2) Flat out remove all innodb references in your config file. 3) Remove some of the keys on the table and see if that helps. Building the index is usually by far the most expensive operation. 4) Run the import, and then do 'show full processlist' to get a feel for where the bottleneck is. I'm not running any special hardware on my end: AthlonXP 2800, 2GB RAM, 10K RPM hard drive. Chances are, that could be worse than what you've got. Ben, my import went really fast, so it must be something with the DB configuration. A couple of thoughts:

1) Set your ‘max packet size’ to something much higher than the alloted ~1mb.
2) Flat out remove all innodb references in your config file.
3) Remove some of the keys on the table and see if that helps. Building the index is usually by far the most expensive operation.
4) Run the import, and then do ’show full processlist’ to get a feel for where the bottleneck is.

I’m not running any special hardware on my end: AthlonXP 2800, 2GB RAM, 10K RPM hard drive. Chances are, that could be worse than what you’ve got.

]]> By: Ben http://www.igvita.com/2006/12/01/loading-netflix-dataset-into-sql/#comment-25571 Ben Thu, 15 Mar 2007 16:53:27 +0000 http://www.igvita.com/blog/2006/12/01/loading-netflix-dataset-into-sql/#comment-25571 I have followed the directions above but importing the data is taking hours... you say it only took you 20 minutes!? I am on OSx, MySQL5, with pretty good hardware. I am using the my-huge.cnf as suggested but the process in still taking forever. To turn off InnoDB/transaction support all you need to do is set the table type to MyISAM like listed above, correct? Or is there a setting I have to use to turn it off? Thanks. I have followed the directions above but importing the data is taking hours… you say it only took you 20 minutes!? I am on OSx, MySQL5, with pretty good hardware. I am using the my-huge.cnf as suggested but the process in still taking forever. To turn off InnoDB/transaction support all you need to do is set the table type to MyISAM like listed above, correct? Or is there a setting I have to use to turn it off? Thanks.

]]>