Comments on: Collaborative Filtering with Ensembles http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/ A goal is a dream with a deadline. Mon, 15 Mar 2010 00:47:21 +0000 http://wordpress.org/?v=2.7.1 hourly 1 By: Ennuyer.net » Blog Archive » Rails Reading - Sept 10, 2009 http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-211854 Ennuyer.net » Blog Archive » Rails Reading - Sept 10, 2009 Thu, 10 Sep 2009 19:38:37 +0000 http://www.igvita.com/?p=609#comment-211854 [...] Collaborative Filtering with Ensembles - igvita.com [...] [...] Collaborative Filtering with Ensembles - igvita.com [...]

]]> By: Github contest: Winning the bottle of Pappy — Some French Guy http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-211355 Github contest: Winning the bottle of Pappy — Some French Guy Sun, 06 Sep 2009 01:49:00 +0000 http://www.igvita.com/?p=609#comment-211355 [...] Netflix taught us that ensembles win. Ilya Grigorik submitted an entry exploiting that fact and wrote about it Collaborative Filtering with Ensembles. [...] [...] Netflix taught us that ensembles win. Ilya Grigorik submitted an entry exploiting that fact and wrote about it Collaborative Filtering with Ensembles. [...]

]]> By: Ilya Grigorik http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-211153 Ilya Grigorik Fri, 04 Sep 2009 02:17:07 +0000 http://www.igvita.com/?p=609#comment-211153 John: Completely agree with your analysis on the incorrect setup of the problem for GitHub contest. Ultimately, that's why I ended up passing on the contest - couldn't figure out why / how I could optimize the algorithm I had in mind to what seemed to be a random scoring process. John: Completely agree with your analysis on the incorrect setup of the problem for GitHub contest. Ultimately, that’s why I ended up passing on the contest - couldn’t figure out why / how I could optimize the algorithm I had in mind to what seemed to be a random scoring process.

]]> By: John Rowell http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-211138 John Rowell Fri, 04 Sep 2009 00:44:28 +0000 http://www.igvita.com/?p=609#comment-211138 Honored by the fork Ilya! I talk about the crowdsourced entry here: http://bit.ly/ghcontest About the 18% score guys (Jerod), I had two entries. My "regular" one got 18%, the crowdsourced one got 50%+ (or as much as the winning one, just change the weight constant to 2 :P). Honored by the fork Ilya! I talk about the crowdsourced entry here: http://bit.ly/ghcontest

About the 18% score guys (Jerod), I had two entries. My “regular” one got 18%, the crowdsourced one got 50%+ (or as much as the winning one, just change the weight constant to 2 :P).

]]> By: Flow » Blog Archive » Daily Digest for September 3rd - The zeitgeist daily http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-211002 Flow » Blog Archive » Daily Digest for September 3rd - The zeitgeist daily Thu, 03 Sep 2009 08:29:50 +0000 http://www.igvita.com/?p=609#comment-211002 [...] Shared Collaborative Filtering with Ensembles – igvita.com [...] [...] Shared Collaborative Filtering with Ensembles – igvita.com [...]

]]> By: Daniel Haran http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-210893 Daniel Haran Wed, 02 Sep 2009 14:09:15 +0000 http://www.igvita.com/?p=609#comment-210893 Jochen: most of us wrote utilities to extract our own test sets, e.g.: http://github.com/danielharan/github_resys/tree/master data/prepare_training_set.rb The Netflix prize was very well designed. They prevented users from submitting more than once every 24 hours. Further, they had a dev-test-set and *2* test-sets - one for public scoring, another for awarding prizes. It's only participating in the Github contest that I realized how well the Netflix prize was organized; many non-specialists would have made the very same mistakes. Not having a dev-test-set was a small issue compared to the temptation to hill climb and overfit to the test-set. Jochen: most of us wrote utilities to extract our own test sets, e.g.:
http://github.com/danielharan/github_resys/tree/master data/prepare_training_set.rb

The Netflix prize was very well designed. They prevented users from submitting more than once every 24 hours. Further, they had a dev-test-set and *2* test-sets - one for public scoring, another for awarding prizes.

It’s only participating in the Github contest that I realized how well the Netflix prize was organized; many non-specialists would have made the very same mistakes. Not having a dev-test-set was a small issue compared to the temptation to hill climb and overfit to the test-set.

]]> By: Dr. Jochen L. Leidner http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-210892 Dr. Jochen L. Leidner Wed, 02 Sep 2009 13:53:17 +0000 http://www.igvita.com/?p=609#comment-210892 The GitHub competition is a nice idea, however methodologically unsound. In machine learning, data sets are usually split in three buckets, training (sometimes called "development") set, dev-test set and test set. The training set is inspected to learn about the task, class distributions, and to hypothesize about features of potential value. Once a classifier is built (by training it on the training set), it can be tested on the dev-test set, which is used for evaluations multiple times during the development cycle to obtain interim results that are more meaningful than evaluation on the training set would be (cf. Erik's comment about overfitting above), but without touching the test set, which ought to remain "unseen data" until the system is considered good enough, is used only once for a final evaluation, and is never inspected. The GitHub competition is a nice idea, however methodologically unsound. In machine learning, data sets are usually split in three buckets, training (sometimes called “development”) set, dev-test set and test set.
The training set is inspected to learn about the task, class distributions, and to hypothesize about features of potential value. Once a classifier is built (by training it on the training set), it can be tested on the dev-test set, which is used for evaluations multiple times during the development cycle to obtain interim results that are more meaningful than evaluation on the training set would be (cf. Erik’s comment about overfitting above), but without touching the test set, which ought to remain “unseen data” until the system is considered good enough, is used only once for a final evaluation, and is never inspected.

]]> By: Ilya Grigorik http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-210888 Ilya Grigorik Wed, 02 Sep 2009 13:30:22 +0000 http://www.igvita.com/?p=609#comment-210888 bd: When you're making a prediction, there should always be a score, or a confidence interval assigned to it. Hence, if you were to build a general predictor, just sort your predictions in order of their score, and output the top 10. The thing is, the 10 that github omitted may not be the actual top 10! That's where the problem lies. Erik: Overfitting is definitely a pitfall, but that's exactly where Bagging and subspace methods shine. Likewise, if the predictors are 'independent', then you're less likely to run into this problem. Anony: Yes, exactly. Though the more I think about it, the more I'm convinced that it would be very interesting to try a two tier system. On the first tier, teams can build predictors, on the other we can optimize the ensembles. Jerod: Actually, if you just rerun his code, you'll get over 50% accuracy now. I think the low score was representative of the poor predictors at the time when he tried it. Cole: Yep, their blog is a treasure trove of information. Daniel: My thoughts exactly. bd: When you’re making a prediction, there should always be a score, or a confidence interval assigned to it. Hence, if you were to build a general predictor, just sort your predictions in order of their score, and output the top 10. The thing is, the 10 that github omitted may not be the actual top 10! That’s where the problem lies.

Erik: Overfitting is definitely a pitfall, but that’s exactly where Bagging and subspace methods shine. Likewise, if the predictors are ‘independent’, then you’re less likely to run into this problem.

Anony: Yes, exactly. Though the more I think about it, the more I’m convinced that it would be very interesting to try a two tier system. On the first tier, teams can build predictors, on the other we can optimize the ensembles.

Jerod: Actually, if you just rerun his code, you’ll get over 50% accuracy now. I think the low score was representative of the poor predictors at the time when he tried it.

Cole: Yep, their blog is a treasure trove of information.

Daniel: My thoughts exactly.

]]> By: Daniel Haran http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-210852 Daniel Haran Wed, 02 Sep 2009 05:08:49 +0000 http://www.igvita.com/?p=609#comment-210852 For increasing accuracy, there's probably more low-hanging fruit in blending. It would have been cool to have 2 parallel competitions: one for the blenders, one for those developing base algorithms. That way you might have won a bottle of whiskey :) For increasing accuracy, there’s probably more low-hanging fruit in blending.

It would have been cool to have 2 parallel competitions: one for the blenders, one for those developing base algorithms. That way you might have won a bottle of whiskey :)

]]> By: Collaborative Filtering with Ensembles – igvita.com « Netcrema – creme de la social news via digg + delicious + stumpleupon + reddit http://www.igvita.com/2009/09/01/collaborative-filtering-with-ensembles/comment-page-1/#comment-210829 Collaborative Filtering with Ensembles – igvita.com « Netcrema – creme de la social news via digg + delicious + stumpleupon + reddit Tue, 01 Sep 2009 22:45:08 +0000 http://www.igvita.com/?p=609#comment-210829 [...] Collaborative Filtering with Ensembles – igvita.comigvita.com [...] [...] Collaborative Filtering with Ensembles – igvita.comigvita.com [...]

]]>