A.I. Advances Through Deep Learning 162
An anonymous reader sends this excerpt from the NY Times:
"Advances in an artificial intelligence technology that can recognize patterns offer the possibility of machines that perform human activities like seeing, listening and thinking. ... But what is new in recent months is the growing speed and accuracy of deep-learning programs, often called artificial neural networks or just 'neural nets' for their resemblance to the neural connections in the brain. 'There has been a number of stunning new results with deep-learning methods,' said Yann LeCun, a computer scientist at New York University who did pioneering research in handwriting recognition at Bell Laboratories. 'The kind of jump we are seeing in the accuracy of these systems is very rare indeed.' Artificial intelligence researchers are acutely aware of the dangers of being overly optimistic. ... But recent achievements have impressed a wide spectrum of computer experts. In October, for example, a team of graduate students studying with the University of Toronto computer scientist Geoffrey E. Hinton won the top prize in a contest sponsored by Merck to design software to help find molecules that might lead to new drugs. From a data set describing the chemical structure of 15 different molecules, they used deep-learning software to determine which molecule was most likely to be an effective drug agent."
Sources of improvements? (Score:5, Insightful)
I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...
Re: (Score:3, Informative)
from TFA
" Modern artificial neural networks are composed of an array of software components, divided into inputs, hidden layers and outputs. The arrays can be “trained” by repeated exposures to recognize patterns like images or sounds.
These techniques, aided by the growing speed and power of modern computers, have led to rapid improvements in speech recognition, drug discovery and computer vision. "
Sounds like both.
Re:Sources of improvements? (Score:5, Insightful)
no, that first sentence pretty much sums up digital neural nets over two decades ago. So more likely the over two orders magnitude processing power per chip improvement since then, with addressable memory over three orders magnitude bigger....
Re: (Score:2, Interesting)
The way they are trained is very different, and it's this change that improves the performance. It's more than just making them faster, a fast idiot is still an idiot.
Re: (Score:2)
we already had the analog / bio version, Cheney's and mega-corporate's meat puppet. But now we've upgraded to Barack the Empty Suit, another mega-corporate bitch.
Re: (Score:3, Informative)
A garden snail has about 20,000 neurons, a cat has 1 billion neurons, a human has 86 billion neurons.
http://www.guardian.co.uk/science/blog/2012/feb/28/how-many-neurons-human-brain [guardian.co.uk]
Yes, but... no. (Score:5, Interesting)
This is a very misleading metric. First, some not-insignificant number of the neurons in the brain are involved in non-cognitive computations. Muscle control, hormone regulation, kinesthesia, vision (not thinking about what is seen, but simply recognizing it), heart rates and other system regulation and so on.
Examples also exist [fyngyrz.com] of low-neuron (and synapse) count individuals who retain cognitive (and all other major) function; these examples cannot be explained away by "counting neurons."
We don't know which yet, but given that high neuron count has been ruled out as the single way to accommodate intelligence, we do know that we need to look to other mechanisms for human cognition. Structure, algorithm, other features known or unknown may be responsible for intelligence; and it may be that something entirely disjoint is responsible for the rise of intelligence; but we know it isn't simply high neuron count.
--fyngyrz (anon due to mod points)
Re: (Score:2)
The example you cite shows images of a compressed brain. It says nothings about "counting neurons"; the person in question could have roughly the same number of neurons as you or I compressed into a smaller space.
Also the guy is said to have an IQ of 75. That's " borderline intellectual functioning [wikipedia.org]", and it's incorrect to say th
Re: (Score:2)
from TFA
" Modern artificial neural networks are composed of an array of software components, divided into inputs, hidden layers and outputs. The arrays can be “trained” by repeated exposures to recognize patterns like images or sounds.
These techniques, aided by the growing speed and power of modern computers, have led to rapid improvements in speech recognition, drug discovery and computer vision. "
Sounds like both.
Well, that doesn't say anything; that just described every neural network for the past couple of decades, except for the "rapid improvement" part. I haven't read TFA, so I don't know if there's more detail, but just describing the basics of how neural networks operate isn't an explanation for why they're suddenly improving.
Re:Sources of improvements? (Score:5, Informative)
The ``new'' (e.g. last decade or so) advances are in training hidden layers of neural networks. Kinda like peeling an onion, each layer getting progressively coarser representation of the problem. e.g. if you have 1000000 inputs, and after a few layers, only have 100 hidden nodes, those 100 nodes are in essence representing all the ``important'' (some benchmark you choose) information of those 1000000 inputs.
Re:Sources of improvements? (Score:4, Insightful)
Article didn't say, but if I had to make a guess, this is where I would start.
http://www.neurdon.com/2010/10/27/biologically-realistic-neural-models-on-gpu/ [neurdon.com]
"The maximal speedup of GPU implementation over dual CPU implementation was 41-fold for the network size of 15000 neurons."
This was done on cards 7 years old now. The massive increase of power in GPUs in the past few years along with more features and better programing languages for them means the performance increase could possibly be many hundreds of times faster. An entire cluster of servers gets crunched down in to one card, multiple cards in one server, and build a cluster of those and you can quickly see that amount of computing power available to neural networks is much much larger now. I'm not even sure how to compare the GT6800 to a modern GTX680 because of their huge differences, but the 6800 did 54 FLOPs and the 680 does 3090.4. A 57x increase. CPU's how far back to we have to go where CPUs are 57 times slower. If everything scales the same in the papers calculations it would mean over a 2000x performance increase on a single computer with 1 GPU. In 7 years.
Re: (Score:2)
Computers have gotten very cheap. Pretty much any prof that wants to pursue something now can build enough hardware to do so with a relatively small amount of money. Neural networks ran into a big wall twenty years ago because the tools weren't there yet.
Once people start having some successes, more funds will be made available, more advances will be made, justifyiing even more funding.. and then we'll turn control of the military over to SkyNet. :)
Re: (Score:2, Insightful)
Don't forget that it's not impossible to build a specially designed processor to do a particular task; such as the digital orrery. Such devices created to do nothing but neural net simulations would be more efficient than using a general purpose computer. It would be linked to such to provide a convenient interface but do most of the heavy lifting itself.
Re: (Score:2)
Why build a special processor when ATI and Nvidia already do. [google.com] Probably at a much lower cost per calculation then a custom machine.
Re:Sources of improvements? (Score:4, Informative)
Why build a special processor when ATI and Nvidia already do. Probably at a much lower cost per calculation then a custom machine.
A GPU can run a neural net much more efficiently than a general purpose CPU, but specialized hardware designed just for NNs could be another order of magnitude more efficient. Of course GPUs are more cost effective because they are mass market items, but if NN applications take off it is likely that everyone will want one running on their cellphone, and then customized NN hardware will be mass market too.
Re:Sources of improvements? (Score:4, Informative)
Re: (Score:2)
Re: (Score:2)
Everything ran into a big wall 20 years ago. There were 680x0, DEC Alpha and SPARC systems, but they were either $10,000 workstations (with no disk drive, server or monitor for the price) or there were embedded systems requiring a rack chassis development kit (manuals cost extra).
Image processing on a PC CPU (= 80386) had to be implemented as a script of image processing command line functions as it wasn't even possible to reliably allocate more than one 64K block. You would load the image in line by line,
Re: (Score:2)
There is no shortage of ideas. There has been a shortage of ways to test them in the real world.
Automatic creation of features (Score:5, Insightful)
I wonder how much of these improvements in accuracy are due to fundamental advances
I was wondering the same thing, and just now found this interview [kaggle.com] on Google. Perhaps someone can fill in the details.
But basically, machine learning is at its heart hill-climbing on a multi-dimensional landscape, with various tricks thrown in to avoid local maxima. Usually, humans detemine the dimensions to search on -- these are called the "features". Well, philosophically, everything is ultimately created by humans because humans built the computers, but the holy grail is to minimize human invovlement -- "unsupervised learning". According to the interview, this one particular team (the one mentioned at the end of the Slashdot summary) actually rode the bicycle with no hands and to demonstrate how strong their neural network was at determining its own features, did not guide it, even though it meant their also-excellent conventional machine learning at the end of the process would be handicapped.
The last time I looked at neural networks was circa 1990, so perhaps someone writing to an audience more technically literate than the New York Times general audience could fill in the details for us on how a neural network can create features.
Re:Automatic creation of features (Score:4, Insightful)
the holy grail is to minimize human invovlement -- "unsupervised learning"
Unsupervised learning is valuable, but calling it a "holy grail" is going a little too far. Supervised, unsupervised, and semi-supervised learning are all active areas of research.
Re: (Score:2)
There is a new thing. It has long been known that "deep networks" could theoretically represent more sophisticated features and concepts, and there were obvious biological examples of this working successfully.
The artificial neural network methods of 1990, as you say hill-climbing on a multi-dimensional landscape, turned out not to work particularly successfully on deep networks, or more correctly, provide little additional benefit vs shallow networks. After this time, resarch in statistical learning move
Re: (Score:2, Informative)
Glad they were able to make it work so quick, but drug discovery has been done like this for over a decade. I worked at an "Infomesa" startup that was doing this in Santa Fe in 2000.
Re:Sources of improvements? (Score:5, Informative)
I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...
I'm sure all of that helped, but the key ingredient is training mechanisms. Traditionally networks with multiple layers did not train very well, because the standard training mechanism "backpropagates" an error estimate, and it gets very diffuse as at goes backwards. So most of the training happened in the last layer or two.
This changed in 2006 with Hinton's invention of the Restricted Boltzman Machine, and someone else's insight that you can train one layer at a time using auto-associative methods.
"Deep Learning" / "Deep Architectures" has been around since then, so this article doesn't seem like much news. (However, it may be that someone is just now getting the kind of results that they've been expecting for years. Haven't read up on it very much.)
These methods may be giving ANN a third lease on life. Minsky & Papiert almost killed them off with their book on perceptrons in 1969[*], then Support Vector Machines nearly killed them again in the 1990s.
They keep coming back from the grave, presumably because of their phenomenal computational power and function-approximation capabilities.[**]
[*] FWIW, M&P's book shouldn't have done anything, since it was already known that networks of perceptrons don't have the limitations of a single perceptron.
[**] Siegelmann and Sontag put out a couple of papers, in the 1990s I think, showing that (a) you can construct a Turing Machine with an ANN that uses rational numbers for the weights, and (b) using real numbers (real, not floating-point) would give a trans-Turing capability.
Re: (Score:3)
using real numbers (real, not floating-point) would give a trans-Turing capability.
What on earth is trans-Turing capability?
Re: (Score:3)
using real numbers (real, not floating-point) would give a trans-Turing capability.
What on earth is trans-Turing capability?
Can compute things that a TM can't.
I think the paper was controversial when it first came out, but I'm not aware that anyone has ever refuted their proof.
Re: (Score:3)
Given that almost every real number encodes an uncountable number of bits of information, I guess this isn't especially surprising in retrospect. The result though should make us suspicious of the assumption that the physical constants and properties in our physical theories can indeed take any real number value.
Re:Sources of improvements? (Score:5, Informative)
The number of bits needed to represent an arbitrary real number exactly is infinite, but not uncountable.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Given that almost every real number encodes an uncountable number of bits of information, I guess this isn't especially surprising in retrospect. The result though should make us suspicious of the assumption that the physical constants and properties in our physical theories can indeed take any real number value.
My intuition is that the difference between the TM's finite set of discrete symbols and the infinite/continuous nature of real numbers is exactly the reason.
I'm not aware of any theory of continuous-state computing along the lines of the Chomsky hierarchy, but maybe there's one out there.
Re: (Score:2)
> (b) using real numbers (real, not floating-point) would give a trans-Turing capability.
Not sure what it means -- a Turing machine is not even capable of storing a single (arbitrary) real number.
Re: (Score:3)
He meant that an ANN with real numbers is a hypercomputer, which is true.
The problem is that like most conceivable hypercomputers neural networks with real numbers would violate natural laws, e.g. the laws of thermodynamics.
Re: (Score:2)
Re: (Score:2)
How so? The math of thermodynamics uses real numbers and does not need any "tricks" to make it work.
I think there is a theoretical minimal entropy production for any computation, so there's a limit to the amount of computation you could do if you used the entire observable universe.
Of course, you can't have the infinite tape required by a TM either.
Re: (Score:2)
Well, real numbers are inherently very problematic from the computational point of view.
Re: (Score:2)
>Aren't support vector machines provably more powerful than ANNs?
In a sense yes. Both (non-linear) SVM's and Neural Nets are universal approximators. However, SVM's can be shown to converge to the ground truth given sufficiently many observations. No such result exists for neural networks. zz
Re: (Score:2)
Re: (Score:2)
It's a naive question. People have been combining evolutionary methods for architecture selection with more traditional gradient/function value optimization since, well at least the late 1970's. It will still evolve only in the space that the human set it up to evolve.
The breakthroughs did not occur because of an automated computer. They occurred through an large-scale evolutionary algorithm known as Smart-Fraction-Of-Human-Civilization-Thinking-And-Working-and-Writing-For-Decades. We needed new ideas a
Re: (Score:3)
Referring to the rapid deep-learning advances made possible by greater computing power, and especially the rise of graphics processors, he added: “The point about this approach is that it scales beautifully. Basically you just need to keep making it bigger and faster, and it will get better. There’s no looking back now.”
I'm sure they've come up with a few incremental advances, but it looks primarily like they've just taken advantage of hardware improvements. You can see from the numbers in the article the results are about what you'd expect from improved hardware (as opposed to actually solving the problem):
[some guy] programmed a cluster of 16,000 computers to train itself to automatically recognize images in a library of 14 million pictures of 20,000 different objects. Although the accuracy rate was low — 15.8 percent — the system did 70 percent better than the most advanced previous one.
Re:Sources of improvements? (Score:4, Insightful)
"As opposed to actually solving the problem"? You brain has about 86 billion neurons and around 100 trillion synapses. It accounts for 2% of body weight and 20% of energy consumed. Do you think these numbers would be large if they didn't need do be?
I think the emphasis in computer science on focusing so exclusively on polynomial-time algorithms has really stunted it. Maybe most of the essential tasks for staying alive and reproducing don't happen to have efficient solutions, but the constants of proportionality are small enough to brute-force with several trillion neurons.
Re: (Score:3, Insightful)
The problem comes when you try larger inputs. Regardless of constant factors if you are playing with O(2^n) algorithms then n will not increase above about 30. If you start looking at really weird stuff (optimal circuit design and layout) then the core algorithms are O(2^2^n) and then if you are really lucky n will reach 5. Back in the 80s it only went to 4, buts thats Moore's law for you.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
The big difference is that biology isn't concerned with finding the optimal solution to problems; any very good solution (optimal or not) will let you live to see another day. A lot of math and computer science is dedicated to finding ironclad proofs that under every circumstance, a particular algorithm will deliver he optimal solution. While that's great when it's feasible, sometimes it's OK to go with something that works well even if it isn't optimal.
The set of good heuristics is a strict superset of t
Re: (Score:2)
You might check your server to see if we took all the magic smoke out of it....
Re: (Score:2)
No, "deep learning" refers mostly to new training algorithms. More computer power helps of course, but the problem previously was that your training became less efficient the bigger your system got. If that doesn't happen, you can scale things up indefinitely.
Just more of the same (Score:2)
They haven't done anything that wasn't already being done by others. They're just doing more of it. Essentially, the approach consist of using Bayesian statistics and a hierarchy of patterns. Prof. Hinton pretty much pioneered the use of Bayesian statistics in artificial intelligence. With a rare notable exception (e.g. Judea Pearl [cambridge.org]), the entire AI community has jumped on the Bayesian bandwagon, not unlike the way they jumped on the symbolic bandwagon in the latter half the 20th century, only to be proven wr
Re: (Score:2)
Re: (Score:2)
Do you have *actual* arguments comparing Bayesian to these hypothetical alternatives
The argument is simple. As Judea Pearl (an early proponent of Bayesian statistics for AI who has since changed his mind) explained, humans are not probability thinkers; they are cause/effect thinkers. If you drop a ball, you know it's going to hit the ground. You don't think that there is a probability that it might not. If you read the word Bayesian in this sentence, you know for certain that you did. There is nothing proba
Re: (Score:2)
Re: (Score:2)
hardware (Score:2)
It's the latter...one could assiduously identify common research buzzwords
From a neuroscience perspective, it's about transmission of signals continuously in a highly complex network...a **hardware limit**
The idea that there will be a 'fundamental advance' that allows for 'artificial intelligence' is really just hype.
All we can ever make is better things to follow our instructions.
Re: (Score:2)
All we can ever make is better things to follow our instructions.
What is the basis for that claim?
In 50 years when we can simulate a brain to any arbitrary level of detail, or build a wet-brain one neuron at a time, why wouldn't it be able to do what naturally occurring intelligence can?
Is there some Special Ingredient that cannot be simulated, even in principle? Or that cannot be understood well enough to try?
It's both (Score:5, Interesting)
In the past few years, a few things happened almost simultaneously:
1. New algorithms were invented for training of what previously was considered nearly impossible to train (biologically inspired recurrent neural networks, large, multilayer networks with tons of parameters, sigmoid belief networks, very large stacked restricted Boltzmann machines, etc).
2. Unlike before, there's now a resurgence of _probabilistic_ neural nets and unsupervised, energy-based models. This means you can have a very large multilayer net (not unlike e.g. visual cortex) figure out the features it needs to use _all on its own_, and then apply discriminative learning on top of those features. This is how Google recognized cats in Youtube videos.
3. Scientists have learned new ways to apply GPUs and large clusters of conventional computers. By "large" here I mean tens of thousands of cores, and week-long training cycles (during which some of the machines will die, without killing the training procedure).
4. These new methods do not require as much data as the old, and have far greater expressive power. Unsurprisingly, they are also, as a rule, far more complex and computationally intensive, especially during training.
As a result of this, HUGE gains were made in such "difficult" areas as object recognition in images, speech recognition, handwritten text (not just digits!) recognition, and in many more. And so far, there's no slowdown in sight. Some of these advances were made in the last month or two, BTW, so we're speaking about very recent events.
That said, a lot of challenges remain. Even today's large nets don't have the expressive power of even a small fraction of the brain, and moreover, the training at "brain" scale would be prohibitively expensive, and it's not even clear if it would work in the end. That said, neural nets (and DBNs) are again an area of very active research right now, with some brilliant minds trying to find answers to the fundamental questions.
If this momentum is maintained, and challenges are overcome, we could see machines getting A LOT smarter than they are today, surpassing human accuracy on a lot more of the tasks. They already do handwritten digit recognition and facial recognition better than humans.
Re: (Score:3, Interesting)
On the other hand there have been some theoretical advances by Hinton and others where networks can be trained on unsupervised data (e.g. the Google cats thing)
Re: (Score:2)
I wonder how much of these improvements in accuracy are due to fundamental advances, vs. the capacity of available hardware to implement larger models and (especially?) the availability of vastly larger and better training sets...
There are limits to what you can achieve with that. I was once surprised to discover how often I actually mishear words (when watching, e.g., episodes of US TV series) and no amount of repeating helps me. After thinking about it for a while, it became apparent to me that I actually interpolate based on the context. This, however, requires understanding what the particular speech is about. The same goes for reading badly printed or (more often) badly scanned text - quite often I reconstruct the word based on
Re: (Score:2)
I used to do some transcription work to make a bit of spare cash. At the beginning of the tape, I really wouldn't understand the accent, not recognising some words, but after going through the tape once and replaying it, I would immediately recognise the words. It's almost as if there were a set of mask images for every word, and these didn't quite fit at first, but after 20-30 minutes they were scaled, rotated, and transformed in some way until they made a better match. Each word would also have a limited
Re: (Score:2)
Yes, we often interpolate from knowing what is being discussed. We can have algorithms to stand in to some extent but there is a limitation when the inference we make is from a representation of things out there in the world and knowledge about how those things work. We can sometimes get a sense of a conversation from very lossy understanding of what is being said.
Re: (Score:2)
Re: (Score:2)
Can their handwriting recognition solve captchas (Score:3)
yet?
Re:Can their handwriting recognition solve captcha (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
There have been several stories about captchas being broken, to the point where secure ones today have to be barely decipherable by humans. That suggests the character recognition algorithms are performing very similarly to humans.
Re: (Score:2)
Re: (Score:2)
Here's one you can try out yourself: http://code.google.com/p/captchacker/ [google.com]
The captcha's now are harder than they used to be but I have no doubt that if you run a few hundred through a breaker you'd get a few hits. Not quite human level, but impressively close considering where we were five years ago. Someone with some serious computer power to put behind it could probably do significantly better.
AI got a bad name because of the promises it made in the 60s and 80s, and there are lots of mystics who are cri
Can You Imagine a Beowulf Cluster of These? (Score:2)
Can You Imagine a Beowulf Cluster of These?
Re: (Score:2)
Neural Network for Machine Learning on Coursera (Score:4, Informative)
I'm doing Prof Hinton course on Neural Network on Coursera this semester. It covers the old school stuff plus the latest and greatest. From what I gather from the lecture, training neural networks using lots of layers hasn't been practical in the past and was plauged with numerical and computational difficulties. Nowadays, we have better algorithms and much faster hardware. As a result we now have the ability to use more complex networks for modelling data. However, they need a lot of computational power thrown at them to learn compared to other machine learning algorithms (random forest). The lecture quotes training taking days on a Nvidia GTX 295 GPU to learn the MNIST handwritten dataset. Despite this, the big names are already using this technology for applications like speech recognition (Microsoft, Siri), object recognition (Google Cat video, okay that's not a real application yet).
Re: (Score:2)
The hardware since the 295 days is around least 3 times as fast too. It seems just about every publication on neural networks has had something about GPUs in the last few years.
http://www.neuroinformatics2011.org/abstracts/speeding-25-fold-neural-network-simulations-with-gpu-processing [neuroinformatics2011.org]
Re: (Score:2)
The hardware since the 295 days is around least 3 times as fast too. It seems just about every publication on neural networks has had something about GPUs in the last few years.
http://www.neuroinformatics2011.org/abstracts/speeding-25-fold-neural-network-simulations-with-gpu-processing [neuroinformatics2011.org]
From the article : Furthermore, to increase the number of calculated time steps increases exponentially the computation time with the CPU while the computation time increases only linearly with the Graphic Processor Unit.
Eh?
Re: (Score:3)
Old News (Score:2, Interesting)
Need some good drugs to believe it? (Score:2)
... wake up people..... its the fucking drug industry looking for any excuse it can to sell you aanother one of their drugs...
And pot remains, for the most part, illegal.....
I think we already have achieved artificial intelligence... in humans...
Neural networks have their limitations (Score:2)
While neural networks do amazingly well for a certain type of problems, they do have their limitations. Neural networks are good for designing reflex machines, that react to their current environment. They aren't efficient when they have to learn on the field or plan ahead.
It had to be asked (Score:2)
From a data set describing the chemical structure of 15 different molecules, they used deep-learning software to determine which molecule was most likely to be an effective drug agent."
So the AI is going to turn some molecules into an FBI undercover snitch? That's some serious DNA-FU there!
Restricted Boltzman Machine (Score:4, Informative)
Deep Belief Networks (Score:5, Informative)
A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning.
It's hard to tell from the article, but they probably are trying to refer to Deep Belief Networks [scholarpedia.org], which are a more recent and advanced type of Neural Network, which incorporates many layers:
Deep belief nets are probabilistic generative models that are composed of multiple layers of stochastic, latent variables. The latent variables typically have binary values and are often called hidden units or feature detectors. The top two layers have undirected, symmetric connections between them and form an associative memory. The lower layers receive top-down, directed connections from the layer above. The states of the units in the lowest layer represent a data vector.
Re: (Score:2)
A lot of vague marketing-speak in this article. "Deep learning"? The article basically talks about neural networks, just one of the techniques in machine learning. Neural networks were hyped for a long time, perhaps because of the catchy name.
You could have answered your own questions with a quick search, rather than assume that that which you are ignorant about is mere "marketing-speak."
deeplearning.net [deeplearning.net]
Deep learning (Wikipedia) [wikipedia.org]
Unsupervised Feature Learning and Deep Learning [stanford.edu]
Re:Deep learning? (Score:4, Insightful)
A lot of vague marketing-speak in this article. "Deep learning"?
Agreed. Why do we need the adjective "deep"? Perhaps it's because a lot of AI jargon uses "learning" when they really just mean "adaptive" (as in, "programmed to respond to novel stimuli in anticipated ways"), whereas normal human "learning" is much more fluid.
The article basically talks about neural networks
Yet another victory for marketing. These things have been around for at least 25-30 years, and the connection to what little we actually have deciphered about how the brain encodes, decodes, and processes information has always been incredibly tenuous. There always seems to be these AI strands of "cognitive science" or "neural modeling," which are often nothing than just somebody's pet algorithm or black box dressed up with words that make it sound like it has some scientific basis in actual neurophysiology or something.
Don't get me wrong -- I'm sure some of the examples in TFA have made great advances, partly due to speed and hardware unthinkable 25-30 years ago. And some of the functionality of the "neural nets" might give significantly better results than previous models.
But I really wish people would lay off the pretend connections to humanity. Why can't we just accept that a machine might just function better with a better program or algorithm or whatever, rather than saying that "our research in cognitive science [i.e., BS philosophy of the mind] has resulted in neural networks [i.e., a mathematical model instantiated into programming constructs] that exhibit deep learning [i.e., work better than the previous crap]."
(Please note: I mean no insult to anyone who works in neuroscience or AI or whatever. But I do question the jargon that seems to make unfounded connections and assumptions that the brain works anything like many algorithmic "models." We may succeed in creating artificial intelligence by developing our own algorithms or we might succeed by imitating the brain, but I don't think we're making progress by pretending that we're imitating the brain when we're really just using marketing jargon for our pet mathematical algorithm.)
Re:Deep learning? (Score:4, Informative)
Why do we need the adjective "deep"?
Because the "deep learning" technologies use artificial neural networks with many more layers than traditionally, making them "deep architectures".
It's widely accepted that the first hidden layer of an ANN serves as a feature detector (possibly sub-symoblic features that you can't put a name to), and each successive layer serves as a detector for higher-order features. Thus the deep architectures can be expected to have some utility for any problem that depends on feature analysis.
Re: (Score:3)
Re: (Score:3)
I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning," though, in the broader world... and even if we equate the jargony connotations of "machine learning" with "learning," it still seems a stretch to use "deep" as an adjective directly applied to that... but perhaps it's just me.
I have a bigger issue with "learning" than with "deep", since with very few exceptions ANNs don't learn anything autonomously, but rather are adjusted by an external algorithm to to perform well on a given problem. "Deep training" would make sense for "deep architectures".
Re: (Score:3)
Actually, it seems your post is the vague one. "normal human "learning" is much more fluid." What does that mean?
Many machine learning algorithms "learn" exactly the way you'd teach a child. They see examples, you tell them what the object, word, etc. is, and the
Re: (Score:2)
I completely agree that you've justified the use of the adjective "deep" in regard to "deep architectures" (and I got that before writing my post). I still don't get how this "deep" has much to do with "learning,"
It's for the same reason as "deep architecture", essentially. Early ANN training methods did not effectively propagate training through many layers of nodes, so the learning was "shallow". New methods allow training each layer at a time, so training can be effectively performed on deep layers. Deep training/learning.
I see your point if you try to interpret the "deep" as implying some sort of more profound learning of the domain. It's not. It's just deep in the sense that deep layers can be effective
deep shit (Score:2)
Why do we need the adjective "deep"?
Because the "deep learning" technologies use artificial neural networks with many more layers than traditionally, making them "deep architectures".
So, you admit 'deep' is a marketing buzzword...thank you. It's *obviously* not a technical term.
It is a discrete, ordinal description of a quantity...that's ALL the word 'deep' in this context means...which means it's a non-technical word...and non-technical words used to make non-existent distinctions in order to gain attention...
well that's a marketing word...
Re: (Score:2)
When people write a paper for publication they have to differentiate their approach from previous approaches. You seem to have latched onto deep as an imprecise description of the number of layers. It is not. It is an accurate distinction in comparison to previous approaches. Because previous approaches were limited to about (not exactly) two layers it makes the definition of the label a little fuzzy, but the partition into shallow / deep approaches is crisp.
Re: (Score:2)
Re: (Score:2)
Then how about something like "multilevel" or "multilayer adaptive networks
Taken.
How about this. First there was the Genetic Algorithm (GA), and then came many variants with different names.. Messy GA, Linkage Learning GA, and so on and on.
Then along comes the Compact Genetic Algorithm, which is actually a variant of an Estimation of Distribution Algorithm (EDA) rather than a variant of a GA, but it was so named because it propagates the same information at the same rate as a simple GA with its crossover parameter set to 0.5 (called Uniform Crossover) so thats the provenance
multilevel shit (Score:2)
I like the suggested nomenclature changes...I think this is an area where any everyday techie or 'nerd' can make the world better for him/herself and everyone *and* make their job easier
make the words we use make sense! It helps **us** signal value to people outside of our in-group...
haha...if I was redesigning computing I'd start with the 'help' tab ;)
fuzzy shit (Score:2)
Because previous approaches were limited to about (not exactly) two layers it makes the definition of the label a little fuzzy, but the partition into shallow / deep approaches is crisp.
I'm glad you mentioned 'fuzzy'...encountering that word partially formed my strong opinions about non-technical language...
See, I started out loving physics, especially astrophysics. An 8 year old in the library science section with a pile of books trying to figure out the sky and learning about the lives of other scientists.
If you look at the history of physics, the idea of rigor is obviously very important. When I encountered things like *Heisenberg Uncertainty* and the consequences of Einstien's theories
Re: (Score:3)
"The term "deep" comes from the idea that the algorithm is trying to learn something deeper than previous algorithms. In fact, the usual set of machine learning algorithms are termed shallow learning now. The difference is that deep learning tries to model P(X) whereas shallow learning (SVM, NN, naive Bayes, etc..) try to learn P(X|Y) where X is your input space and Y is the label space. "
Well, more correctly, the deep learning tries to model P(X) as P(X | H) for some set of "hidden" or latent features H wh
Re:Deep learning? (Score:4, Interesting)
It looks like you are seeing something that is not there. The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works.
As someone who has read a lot of the founding literature of modern cognitive science and the philosophy of mind in the 1950s through 80s, which was hugely influential in setting up the early approaches to AI (including neural nets), I have to say -- this is where the stuff came from.
And frankly, a lot of applications in more obscure disciplines, such as in AI analysis in the humanities, researchers are still making claims about these models and their relationships to the actual brain. Hell, just a few years ago I heard a leading cognitive scientist claim that he found evidence for a sort of musical "circle of fifths" neural network in an actual circular physical structure of neurons in the brain... a made-up musical model grafted onto a made-up AI brain model, supported by noisy data... I admit this is an extreme example, but it's not unique.
I understand that modern researchers in "pure" AI may want to avoid recognizing the history or the implications of the terminology -- but there's a reason why the Starship Voyager was equipped with "neural gel-packs" that could get anxious and cause a warp-core breach at a temporal anomaly... words like "neural" actually mean something, and these "neural nets" have about as much connection to the biological function of actual neurons as Voyager's bizarre "neural gel-packs." Yet the implicit metaphor made in continuing to use the term should not be underestimated, not just in a general audience NYT article, but in the way fields are subtly shaped by their nomenclature.
Re: (Score:2)
One can both acknowledge that something was inspired by something and no longer has any connection to it. And while "neural" means something specific in biology, it can mean something specific but different in computer science. That is the nature of jargon sometimes.
I completely get your point, and if it were just one or two words ("neural" or whatever), I might agree. But the influence in this case is pervasive, and it has shaped and continues to shape the way we talk about the field. New nomenclature often continues to extend the mind metaphors, when there is no necessary reason to. Why call it "deep learning" when "multilevel" or "multilayered" might better describe the process? Etc. That was the point of my original post. And frankly, the nomenclature seems
Re: (Score:2)
Why call it "deep learning" when "multilevel" or "multilayered" might better describe the process?
Re: (Score:3)
"The majority of neural network research is about developing new and/or improved algorithms to solve problems, not to say anything about how the human brain works."
This isn't true---or the connotation of it isn't true. I don't know how to quantify "majority", but there is substantial interest in computational modeling of actual biology across all levels of biological/chemistry fidelity and attention to engineering and statistical problems.
A glance at the work in the journal _Neural Computation_ shows paper
Re: (Score:3)
Advances are in ways of learning hidden layers that are slightly more clever than backpropagation. For example, lets say you have an image, apply some transform to it (dct, wavelet, whatever, neural net layer, etc.) and save all the important features, but at say 10x less space. Then do the same to those features. Every time reducing the amount of data by 10x. After a few such layers, lets say you're left with 10 bits worth of information---the ``most important'' (according to your benchmark used) ten bits
Re: (Score:3)
We need to open all the documentation for everyone who want to learn and investigate about IA.
Absolutely. It's about time we figured out who really won those caucuses -- and what the heck is up with the ethanol subsidies?