Comments on: Correlating Netflix and IMDB Datasets http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/ A goal is a dream with a deadline. Thu, 18 Mar 2010 00:25:10 +0000 http://wordpress.org/?v=2.7.1 hourly 1 By: Pavel http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-169497 Pavel Sat, 14 Feb 2009 15:28:11 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-169497 yeah I tried doing something similar myself: doing an exact string match against all entries in results page and then comparing the year. But this only worked on 48% of the netflix titles. Adding some heuristic rules (like moving 'the' to the front, removing non alphanumeric characters, etc) got me another 20%. yeah I tried doing something similar myself: doing an exact string match against all entries in results page and then comparing the year. But this only worked on 48% of the netflix titles. Adding some heuristic rules (like moving ‘the’ to the front, removing non alphanumeric characters, etc) got me another 20%.

]]> By: Ilya Grigorik http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-169489 Ilya Grigorik Sat, 14 Feb 2009 15:12:32 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-169489 Pavel, you're right, I used a naive matching technique. It's been a while since I've done this, but I thought I went one step further by comparing the year the movie was released. Once again, this won't give you 100% match, but it will eliminate some false positives. Atul, I would recommend that you take a look at pyimdb or some other libraries which have already solved those mysteries. ;) Pavel, you’re right, I used a naive matching technique. It’s been a while since I’ve done this, but I thought I went one step further by comparing the year the movie was released. Once again, this won’t give you 100% match, but it will eliminate some false positives.

Atul, I would recommend that you take a look at pyimdb or some other libraries which have already solved those mysteries. ;)

]]> By: Atul http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-168938 Atul Fri, 13 Feb 2009 06:58:05 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-168938 I have been trying to read a few IMDB datasets but they are set up really absurd. Do you have any documentation that helped you understand what is the structure of various .list files? I have been trying to read a few IMDB datasets but they are set up really absurd. Do you have any documentation that helped you understand what is the structure of various .list files?

]]> By: Pavel http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-164946 Pavel Sat, 31 Jan 2009 14:12:43 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-164946 Looking at your imdbMatrix script it seems that you correlated the netflix movies with the top search result in the imdb list. However, that seems like a very poor method. Just by looking at the first few films in the netflix dataset and doing a search for their name in the imdb site I get quite a few entries that infact relate to a different film. ex: netflix id 17 is "7 seconds" but searching for it on imdb the first result is "Gone in 60 seconds". This seems to be also true for quite a few other films. Looking at your imdbMatrix script it seems that you correlated the netflix movies with the top search result in the imdb list. However, that seems like a very poor method. Just by looking at the first few films in the netflix dataset and doing a search for their name in the imdb site I get quite a few entries that infact relate to a different film. ex: netflix id 17 is “7 seconds” but searching for it on imdb the first result is “Gone in 60 seconds”. This seems to be also true for quite a few other films.

]]> By: DNAHelix.org: Netflix Prize http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-157991 DNAHelix.org: Netflix Prize Thu, 08 Jan 2009 00:44:29 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-157991 [...] Pickled Python data http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/ [...] [...] Pickled Python data http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/ [...]

]]> By: Netflix Prize Progress « Social Mode http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-107404 Netflix Prize Progress « Social Mode Sun, 27 Jul 2008 18:59:36 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-107404 [...] I used the python pickles from Ilya G. He did the hard work of creating feature vectors (coded descriptions) from IMBD data for things like release date, genre, cast and so forth.  This makes the KNN algorithms more interesting than just going off of the movie title, rating and year. [...] [...] I used the python pickles from Ilya G. He did the hard work of creating feature vectors (coded descriptions) from IMBD data for things like release date, genre, cast and so forth.  This makes the KNN algorithms more interesting than just going off of the movie title, rating and year. [...]

]]> By: Diarmuid http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-101782 Diarmuid Wed, 05 Mar 2008 22:40:04 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-101782 Actually, now that I look at the data more carefully, I see that the numbers map 1 to 1. Sorry about that. thanks D Actually, now that I look at the data more carefully, I see that the numbers map 1 to 1. Sorry about that.

thanks

D

]]> By: Diarmuid http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-101780 Diarmuid Wed, 05 Mar 2008 22:21:24 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-101780 Hi, Great job on the extract. I have loaded up the dictionary and see that I can get the features for a movie by passing in the ID. However, the id is from 1 to 17770, so how does this relate to the movie id in the neflix data and more specifically the data generated by pyflix? Also, have you any info on what real world feature each numeric feature corresponds to. I have assumed 1030 to 1035 is the rating as they are common to all. I have inverted your data to produce a dictionary of features and corresponding movies. cheers Diarmuid Hi,
Great job on the extract. I have loaded up the dictionary and see that I can get the features for a movie by passing in the ID. However, the id is from 1 to 17770, so how does this relate to the movie id in the neflix data and more specifically the data generated by pyflix?
Also, have you any info on what real world feature each numeric feature corresponds to. I have assumed 1030 to 1035 is the rating as they are common to all.
I have inverted your data to produce a dictionary of features and corresponding movies.
cheers

Diarmuid

]]> By: Ilya Grigorik http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-48550 Ilya Grigorik Fri, 29 Jun 2007 13:30:00 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-48550 Pickle is a default python library, so you can deserialize the contents fairly easy. But to answer your question, the file itself is a dictionary in the following format: movie_id => [feature_id, feature_id,...]. What the feature_id is, the file does not say, but you don't need to know that either, all you care about is feature_id overlap with over movies. Pickle is a default python library, so you can deserialize the contents fairly easy. But to answer your question, the file itself is a dictionary in the following format: movie_id => [feature_id, feature_id,...]. What the feature_id is, the file does not say, but you don’t need to know that either, all you care about is feature_id overlap with over movies.

]]> By: guest http://www.igvita.com/2007/01/27/correlating-netflix-and-imdb-datasets/comment-page-1/#comment-48519 guest Fri, 29 Jun 2007 07:12:22 +0000 http://www.igvita.com/blog/2007/01/27/correlating-netflix-and-imdb-datasets/#comment-48519 great thanks for your process on imdb, but i don't quite understand the two features-map.pickle files. so would mind explain a little bit on the file, what is the meaning for each line for a segment: (dp0 I1 (lp1 I737 aI885 aI997 aI998 aI999 aI1035 asI2 (lp2 I886 aI1035 asI3 (lp3 I887 Or would you mind post the whole pickle.py source, so that we can use it to convert the features into other formats. great thanks for your process on imdb, but i don’t quite understand the two features-map.pickle files.

so would mind explain a little bit on the file, what is the meaning for each line for a segment:

(dp0
I1
(lp1
I737
aI885
aI997
aI998
aI999
aI1035
asI2
(lp2
I886
aI1035
asI3
(lp3
I887

Or would you mind post the whole pickle.py source, so that we can use it to convert the features into other formats.

]]>