How To Evaluate Computers That Don't Quite Exist (sciencemag.org) 27
sciencehabit writes: To gauge the performance of a supercomputer, computer scientists turn to a standard tool: a set of algorithms called LINPACK that tests how fast the machine solves problems with huge numbers of variables. For quantum computers, which might one day solve certain problems that overwhelm conventional computers, no such benchmarking standard exists. One reason is that the computers, which aim to harness the laws of quantum mechanics to accelerate certain computations, are still rudimentary, with radically different designs contending. In some, the quantum bits, or qubits, needed for computation are embodied in the spin of strings of trapped ions, whereas others rely on patches of superconducting metal resonating with microwaves. Comparing the embryonic architectures "is sort of like visiting a nursery school to decide which of the toddlers will become basketball stars," says Scott Aaronson, a computer scientist at the University of Texas in Austin.
Yet researchers are making some of their first attempts to take the measure of quantum computers. Last week, Margaret Martonosi, a computer scientist at Princeton University, and colleagues presented a head-to-head comparison of quantum computers from IBM, Rigetti Computing in Berkeley, California, and the University of Maryland (UMD) in College Park. The UMD machine, which uses trapped ions, ran a majority of 12 test algorithms more accurately than the other superconducting machines, the team reported at the International Symposium on Computer Architecture in Phoenix. Christopher Monroe, a UMD physicist and founder of the company IonQ, predicts such comparisons will become the standard. "These toy algorithms give you a simple answer -- did it work or not?" But even Martonosi warns against making too much of the tests. In fact, the analysis underscores how hard it is to compare quantum computers -- which leaves room for designers to choose metrics that put their machines in a favorable light.
Yet researchers are making some of their first attempts to take the measure of quantum computers. Last week, Margaret Martonosi, a computer scientist at Princeton University, and colleagues presented a head-to-head comparison of quantum computers from IBM, Rigetti Computing in Berkeley, California, and the University of Maryland (UMD) in College Park. The UMD machine, which uses trapped ions, ran a majority of 12 test algorithms more accurately than the other superconducting machines, the team reported at the International Symposium on Computer Architecture in Phoenix. Christopher Monroe, a UMD physicist and founder of the company IonQ, predicts such comparisons will become the standard. "These toy algorithms give you a simple answer -- did it work or not?" But even Martonosi warns against making too much of the tests. In fact, the analysis underscores how hard it is to compare quantum computers -- which leaves room for designers to choose metrics that put their machines in a favorable light.
Re: (Score:3)
I think I will live long enough to see a production, retail robot car.
I suppose 1 out of 30 things ain't too bad.
Re: (Score:2)
Quantum computers exist, but general purpose quantum computers don't... or do they?
Climate change, on the other hand, is tired of hearing about you, and is in the process of doing something about it. Sadly, the rest of us will go, too.
Re: (Score:2)
Quantum computers exist, but general purpose quantum computers don't... or do they?
They exist, however the problem is that you can't run a qubit through very many gates before the errors become too great to get anything useful out. This is because the external environment interacts with the qubits altering them from the desired state. Trapped ions seem more stable, however their stability comes at the cost of more time to go through a gate - so right now any gains in stability are wiped out by their gate operation times.
The dream right now is to get to a point where the error rates are
uh huh (Score:3)
"which leaves room for designers to choose metrics that put their machines in a favorable light"
Which differs from classical computing how?
Re: (Score:2)
It's a practiced science at almost any vendor whose product can be measured in terms of speed, to optimize against all known usable benchmarks in any class suited to their product.
Lookin' at you $everyhardwarevendorusingacpugpuasic_or_fpga!!
Gaming benchmarks is a point of honor among those thieves, and it gets worse if you're a German diesel car vendor trying to game the EPA.
We need a better measurement metric (Score:3)
Re: (Score:3)
History repeating (Score:4, Insightful)
In fact, the analysis underscores how hard it is to compare quantum computers
Strike "quantum" and replace with "the most recent new breed of", and you have a truism for the ages.
It could have been said for the initial tube-based computers, and then the first transistor-based ones, and then the IC based ones, and fully integrated ones, ones based on field-programmable arrays, the first supercomputers, etc. As long as each machine is part of a bespoke vanguard and thus there is no extensive commonality among them (like reasonably comparable operating systems), there is, fundamentally, no good way to compare performance except for almost meaningless toy problems.
We will cross that bridge when we get to it. (Score:2)
LINPACK was developed when the computers to evaluate already existed*.
Why do we need to develop benchmarks for computers that do not exist? Let the computers come to fruition first, and after we understand enough about them, and there is some sort of architectural commonality across them (all supercomputers nowadays are more or less Von-Newman machines, no mater if they are Arm, X86/AMD64, PPC or some other arch). That way we at least have a fightng chance to get decent benchmarks for a while.
JM2C
* LINPACK
Re: We will cross that bridge when we get to it. (Score:2)
Why do we need to develop benchmarks for computers that do not exist?
So that researchers with no ideas of value to society can continue to suck on the government teat. Why else?
Vapourware (Score:2)
I find this unsurprising. There used to be whole industries based on software that didn't exist.