Interest Still High In the Netflix Algorithm Competition 77
circletimessquare brings us an update to the status of the million-dollar Netflix competition to develop a better algorithm for movie recommendations. We've discussed aspects of the competition since it started two years ago, but the New York Times has a lengthy overview of where it stands now.
"The Netflix competition is still going strong, with a vibrant, competitive roster of some 30,000 programmers around the globe hard at work trying to win the prize. The Times provides a look at some of the more obsessive searchers, such as Len Bertoni, a semi-retired computer scientist near Pittsburgh who logs 20 hours a week on the problem, oftentimes with the help of his children. There's also Martin Chabbert in Montreal: 'After the kids are asleep and I've packed the lunches for school, I come down at 9 in the evening and work until 11 or 12.' The article gets into the history of the search algorithm Netflix currently uses, and explores the hot commodity called 'singular value decomposition' that serves as the basis for most of the algorithms in competition."
Wow! Think about how many free man-hours Netflix (Score:4, Insightful)
got from this, even when it has to pay out the prize it will be very cheap against any going rate.
Re:Wow! Think about how many free man-hours Netfli (Score:2, Insightful)
Re:Wow! Think about how many free man-hours Netfli (Score:2, Insightful)
That's what I was thinking. This shows how a company with good management can really save money. Instead of the standard outsourcing-to-cheap-country thing, you get 30,000 people, many of them very bright and motivated, working on your problem for free, and you only have to pay when they are successful per your definition. What a bargain!
Re:Wow! Think about how many free man-hours Netfli (Score:5, Insightful)
Why is it considered selling yourself short if you do work for free for a commercial entity... but not when you contribute to, say, FireFox, ThunderBird, Apache, the Linux kernel, and so forth and so on?
In both cases you are typically doing work for absolutely zilch as far as cash or prizes go. You may get a fuzzy warm feeling on the inside, you may simply enjoy doing the work (similar to the fuzzy warm feeling), maybe you enjoy the popularity it gives you. On rare occasions, maybe the work you do there lands you a job further down the line but that's not really something you can bank on. In both cases, you are also doing work somebody else -could- have been doing, for actual pay. I won't get into an argument of whether doing work for free means you're 'stealing jobs' - fact simply is that Netflix -is- getting a lot of work done practically for free that they would otherwise have had to hire somebody for; you would have to agree as otherwise "selling themselves short" would not apply.
So yes, you're doing work that should be landing you some cold hard cash when you...
- devise a matching algorithm for Netflix
- create a video for Radiohead
- submit photos to a Canon photo competition that they are then free to use in any and all marketing material aka ads regardless of whether your photo actually won that competition.
But isn't that pretty much the status quo that many here -want- to go to? Those making their money with proprietary programming, creating arts, etc. are dinosaurs in dying business models, no?
(only semi-flamebait)
Re:Wow! Think about how many free man-hours Netfli (Score:4, Insightful)
It's not selling yourself short to work on FOSS for a very simple reason. Work on FF, or Thunderbird, or open-sourcing a script that I wrote to convert music is free at the point of delivery. That is, anyone can use it without paying. Freely given, and freely distributed.
However, in this case the user of the algorithm is paying Netflix. Netflix takes the work that I have done, and closes it off from other people. My work goes not to benefit the community, but merely to benefit one company - a company that has paid me (cheaply) for my work. Since companies by definition only care about the bottom line, their intent is not to benefit the community, but to benefit themselves. You are effectively working for them for cheap, selling yourself short.
If netflix were to give away the algo for use by anyone else too, then it would be very generous and then you may be able to make a comparison with FOSS. I( have no idea if they will do that or not. However, if I were a shareholder, I would not want them to give away a potentially killer feature for which they paid $1m.
Saying that, if you enjoy playing with this, go ahead! Just be honest with yourself about. If you still want to do it, wallow in it. But it's an extremely pernicious thing to do to link this with working on something that is done to benefit everyone. It simply is not the same thing.
Re:almost impossible to really win (Score:3, Insightful)
You recall incorrectly. None of the top teams currently use external data sources (some have tried, but it doesn't help that much once you get up towards the top 10). The last team that (probably, not official yet) won a milestone used the combined predictions of the first and second place teams, interpolated to improve the final score, but nothing external. Same thing in the previous year, the winning team used only computations on the data given.
IAITTT ;)
Re:Algorithm or Human inaccuracy? (Score:3, Insightful)
well, i think that's one of the interesting things about social sciences. it's very difficult--nigh impossible in some cases--to accurately predict the behavior of a single individual. however, it is possible to predict the collective behavior of a large group of people.
this seems counter intuitive at first, but it's kinda like not being able to predict how a particular dice roll will land but still being able to predict the statistical average of 100 dice rolls.
the capriciousness of individuals eventually balances itself out if you use a large enough data set. likewise, fringe opinions also balance each other out when dealing with statistical averages.
Comment removed (Score:3, Insightful)
Re:Wow! Think about how many free man-hours Netfli (Score:1, Insightful)
Oops. One small problem that invalidates most of what you said.
First, a company is not a "self." Second, a company is run to profit the people that put capital at risk (i.e. the owners of the company.) Those are the "selves" that are being rewarded. Why shouldn't people that put capital at risk be rewarded for their effort? Third, the company is providing a benefit to everyone. They are providing a good product at a fair price. If they weren't, then it wouldn't be a successful company.
I still contend that the basics are no longer taught in school.
Re:Crowdsourcing (Score:5, Insightful)
exploitation is exploitation, self imposed or not; as well, contests are, by their very nature, one-sided, there must be a loser, if there is to be a winner, therefore, competition is only valuable within the framework of co-operation, otherwise, yes, it is exploitative
see? oh well probably not....
I think you don't understand the concept of "fun." Read the article and the comments and tell me that the people "working" at this competition aren't getting paid handsomely. If money is the only compensation that means anything to you, you must be an economist. Congrats, you're doing your part to keep up economics' reputation as the "dismal science."
If it's exploitative for a company to provide enjoyment and intellectual stimulation to a lot of people and benefit financially as a result, then I guess publishing companies don't deserve my support, either. And the movie studios... theater companies... restaurants... and of course any bands that get paid for gigs are just a bunch of ruthless exploiters.
I guess the only commercial entertainment that's okay is what I can enjoy completely passively, without any mental effort at all. That way I'm not being exploited, right? Because work is an awful, awful thing ;-)
Wake up; it's not the nineteenth century or even the twentieth century. Everyone has a natural appetite for work, and unlike our unfortunate ancestors, ours is not overwhelmed and sickened by the work required for mere survival. You're addressing a relatively privileged group of people; we earn enough to support ourselves on less work than we have an appetite for. As a result, we don't have to regard all work as a curse imposed on us by necessity. Work freely done and enjoyed is a blessing.
If you insist that our entire appetite for work be channeled through grim-faced contract negotiations, then that blessing is ruined. What's the point of ruining our fun? So you can save us from the misery suffered by our great-grandparents?
Obviously none of what I said applies to call center employees, game company employees, and technical support employees. They should pay careful attention to what you say ;-)
Re:Crowdsourcing (Score:3, Insightful)
it is exploitative for any enterprise to pay less than fair compensation in any exchange
I only demand to be compensated for some pain or loss on my part. I don't need to be compensated for pleasure. Should I be jealous of the benefit another party gets when I engage in a mutually beneficial exchange? If I give a gift to someone, should they be angry if I enjoy giving it more than they enjoy receiving it?
Ah, but you said "enterprise." I do believe that companies are inherently more prone to evil than the people they comprise, and they need careful watching and manipulation (by consumers and by regulators) to make sure they don't abuse their economic power or legal status (which is important when dealing with special legal entities such as corporations.) However, when assessing a mutually beneficial exchange with another party, I don't see any need to add a special penalty if the other party is a business entity instead of a person.
most modern companies enterprises take more from the community than they deserve or return
You are playing a subtle game with words here -- if "taking" is bad then you must be referring to some loss or cost incurred by another party. There is surely nothing wrong with "taking" pleasure from someone without incurring any cost or displeasure for them, so you are strictly using the word "take" to refer to the case where one party gets something and as a result the other party loses something.
In this case Netflix may get something from the work around the prize, but what do they take, and who do they take it from? Do they "take" the time and effort of the participants? The participants don't seem to feel that their time and effort are being "lost" in any way, so I don't see how that is possible.
They may "take" some right to any IP generated by the winner, but they will pay one million dollars for it, and any participant who feels that is unfair is free to keep his algorithm to himself instead of submitting his results and claiming the prize.