Extreme Complexity of Scientific Data Driving New Math Techniques 107
An anonymous reader writes "According to Wired, 'Today's big data is noisy, unstructured, and dynamic rather than static. It may also be corrupted or incomplete. ... researchers need new mathematical tools in order to glean useful information from the data sets. "Either you need a more sophisticated way to translate it into vectors, or you need to come up with a more generalized way of analyzing it," [Mathematician Jesse Johnson] said. One such new math tool is described later: "... a mathematician at Stanford University, and his then-postdoc ... were fiddling with a badly mangled image on his computer ... They were trying to find a method for improving fuzzy images, such as the ones generated by MRIs when there is insufficient time to complete a scan. On a hunch, Candes applied an algorithm designed to clean up fuzzy images, expecting to see a slight improvement. What appeared on his computer screen instead was a perfectly rendered image. Candes compares the unlikeliness of the result to being given just the first three digits of a 10-digit bank account number, and correctly guessing the remaining seven digits. But it wasn't a fluke. The same thing happened when he applied the same technique to other incomplete images. The key to the technique's success is a concept known as sparsity, which usually denotes an image's complexity, or lack thereof. It's a mathematical version of Occam's razor: While there may be millions of possible reconstructions for a fuzzy, ill-defined image, the simplest (sparsest) version is probably the best fit. Out of this serendipitous discovery, compressed sensing was born.'"
Re:We are the ones in need of a network (Score:4, Informative)
This has actually always been the norm. Physics has long driven mathematics research for instance; many areas of calculus were created/discovered specifically to solve problems in physics.
Re:informercial (Score:3, Informative)
What are you smoking? 1877 citations since 2008 isn't a good citation statistic? More importantly, judging someone's research value by absolute citation statistic is quite silly; he is a full Stanford Professor for his accomplishments, intellect, and personality (I hear he is a good advisor).
While the article is quite a promotional piece, you don't know much about the field. Gunnar Carlsson and his group have advanced computational topology moreso than any other. He came up with the concept and way to compute persistent homology, one of the actually useful and computable advancements that has come out of the topology field. It allows you to reason about clustering much better than any adhoc statistical measure.
The current computational topology tools implemented by grad students today, like PLEX, dont scale very well. His group had some proprietary advancement that scales well, and spun off the data science company.
If you would like two links that are actually informative about what Gunnar does:
http://comptop.stanford.edu/
http://www.ams.org/journals/bull/2009-46-02/S0273-0979-09-01249-X/S0273-0979-09-01249-X.pdf