Grid Computes 420 Years Worth of Data in 4 Months

Da Massive writes with a ComputerWorld article about a grid computing approach to the malaria disease. By running the problem across 5,000 computer for a total of four months, the WISDOM project analyzed some 80,000 drug compounds every hour. The search for new drug compounds is normally a time-intensive process, but the grid approach did the work of 420 years of computation in just 16 weeks. Individuals in over 25 countries participated. " All computers ran open source grid software, gLite, which allowed them to access central grid storage elements which were installed on Linux machines located in several countries worldwide. Besides being collected and saved in storage elements, data was also analyzed separately with meaningful results stored in a relational database. The database was installed on a separate Linux machine, to allow scientists to more easily analyze and select useful compounds." Are there any other 'big picture' problems out there you think would benefit from the grid approach?

Wikipedia?

[JonathanR]

It strikes me as strange that something like Wikipedia could not be distributed across user's PCs in more of a peer-to-peer fashion. Surely the web itself could benefit from further decentralisation. This issue bothered me some years ago, when I discovered that my desktop PC at work had about 40Gb of unpartitioned disk space. I often wondered about the sense of running file servers in big organisations, when each user probably has a few tens of gigabytes of unused or unpartitioned disk space. If illicit music and video can be distributed by P2P, why not all information?

Ok, how does this apply to patents?

[zappepcs]

If the grid solution finds THE cure for H5N1, will it be patentable? If not, who pays for the R&D to implement it? Who gets the patent? Do the thousands of people who allowed their PCs to be used get anything? Will big drug companies be able to use this and keep the prices low for the final product?

Re:Ok, how does this apply to patents?

[Dr. Spork]

These are all good questions, and every user who volunteers their computer for something like this should find answers to them. I'm quite sure that the stuff discovered by distributed networks does not automatically enter the public domain, but in cases like SETI and protein folding, the organizers explicitly state that it will. But it wouldn't be illegal for a drug company to use volunteers' computers just for corporate profit. You have to judge the merit of each of these projects on a case-by-case basis. Remember also that there is a cost to participating: you have to run your computer at peak power, and this will add several hundred dollars to your utility bills each year while polluting the planet with extra coal smoke and CO2.

How about open source, distributed search

[classh_2005]

This looks interesting:

http://www.majestic12.co.uk/ [majestic12.co.uk]

Grid Computing Projects

[1mck]

I've been donating my processor time for quite awhile now for the Malaria research, and even though the drug companies will probably benefit from my donation, they would not have these breakthroughs if people didn't donate that time, and it is the fact that a breakthrough will be found is what keeps me donating my processor time. It's a great feeling knowing that I've contributed to a possible cure towards this disease! Other projects that could need the services of Grid Computing, I believe that was the original question that was put forth, are imaging analysis (any field), physics (particle research, etc), and I can also see Grid Computing being used also for computer animations where the time to render animations would be greatly reduced, and allowing movies, and shows to be released much faster than before. (With this application, it would be known that you are contributing to a product that a company will be making a profit, and the only reason to do it is get these movies, shows to market faster. I, for one, would love to see a sequel to The Incredibles, and to be a part of that would be fantastic, even to just have my name mentioned in the credits!) One thing that needs to be done for these projects to get the maximum exposure for Grid Computing is to dumb down the process. A Noob would be hard pressed to set up Boinc Manager to do the Malaria research.

Re:Wikipedia?

[Fordiman]

The system would be designed for it.

P2P isn't a good model, but I can think of one:
Data, as it is created, is stored in the users' shared folder. As other users go to access it, a copy is made from the cloud (as long as filename/size/hashes match) and that copy is used so long as the creator's copy hasn't been modified. When writes are done, they're done locally, and a patch is sent to the original copy. If the creator can't be contacted, or his copy doesn't exist, the last-writer becomes 'creator'. The file's creator is identified by his DC user name.

Backing up is simple. For every creation/update that is made, a patch is queued or sent to a backup server. The server ONLY queues the originals and patches, so that past-versions are accessible. As space becomes unavailable (say, below 10%), the backup server alerts the IT guys that it needs to offload some stuff, and condenses changes of the oldest files in the local copy. When a delete is made, that is considered a write and handled accordingly.

In the event of a reinstall (ie: the local copy of the files are deleted, but the world hasn't been notified), the user, upon connection would query the backup server to see where his stuff has gone, and get it back.

One could create this system to act like an SMB share, with access levels and program-independent drive/directory mapping, but with one added benefit: user-creation and auto-mapping. The DC would automatically tell the system which peer-shares are available to him upon login. The user can then filter out what he needs as he uses it, but can index-search it all (a query is sent to the backup server, which, like a good little machine, has been indexing as backups are made).

Lastly, for reverse compatibility, the backup server could provide SMB access to its copies, ensuring that non-updated systems can still access their stuff.

I don't know about most organizations, but I work at Penn, and a system like this could work admirably.

Re:Grid Computing Projects

[AlXtreme]

I can also see Grid Computing being used also for computer animations where the time to render animations would be greatly reduced, and allowing movies, and shows to be released much faster than before.

I'm afraid that that will take quite some time to realize. Rendering CG, besides taking a lot of processing time, also requires enormous amounts of data, which restricts the rendering to render farms, the data being pumped over a high-speed LAN.

Actually the amount of problems solvable by using Grid Computing over the internet isn't very high. You need computationally-intensive problems that can be easily parallized, besides requiring limited amounts of data. There's little point in distributing a problem if it takes longer to distribute the data than that you gain by using multiple nodes.

Re:Malaria?

[Praseodymn]

Worrying about the environment is a luxury. Being able to do something to stop what will probably kill you is a luxury. Living anywhere because you want to is a luxury. Having a choice to take the lucid dream inducing malaria drugs or not is a luxury.

Where malaria flourishes, luxuries are scarce.

Travel as much as you can in your life, preferably to the poorer countries. They are often the happier ones.