Advanced Version of Gemini With Deep Think Officially Achieves Gold-Medal Standard at the International Mathematical Olympiad (deepmind.google) 63
An anonymous reader shares a blog post: The International Mathematical Olympiad is the world's most prestigious competition for young mathematicians, and has been held annually since 1959. Each country taking part is represented by six elite, pre-university mathematicians who compete to solve six exceptionally difficult problems in algebra, combinatorics, geometry, and number theory. Medals are awarded to the top half of contestants, with approximately 8% receiving a prestigious gold medal.
Recently, the IMO has also become an aspirational challenge for AI systems as a test of their advanced mathematical problem-solving and reasoning capabilities. Last year, Google DeepMind's combined AlphaProof and AlphaGeometry 2 systems achieved the silver-medal standard, solving four out of the six problems and scoring 28 points. Making use of specialist formal languages, this breakthrough demonstrated that AI was beginning to approach elite human mathematical reasoning.
This year, we were amongst an inaugural cohort to have our model results officially graded and certified by IMO coordinators using the same criteria as for student solutions. Recognizing the significant accomplishments of this year's student-participants, we're now excited to share the news of Gemini's breakthrough performance. An advanced version of Gemini Deep Think solved five out of the six IMO problems perfectly, earning 35 total points, and achieving gold-medal level performance.
Recently, the IMO has also become an aspirational challenge for AI systems as a test of their advanced mathematical problem-solving and reasoning capabilities. Last year, Google DeepMind's combined AlphaProof and AlphaGeometry 2 systems achieved the silver-medal standard, solving four out of the six problems and scoring 28 points. Making use of specialist formal languages, this breakthrough demonstrated that AI was beginning to approach elite human mathematical reasoning.
This year, we were amongst an inaugural cohort to have our model results officially graded and certified by IMO coordinators using the same criteria as for student solutions. Recognizing the significant accomplishments of this year's student-participants, we're now excited to share the news of Gemini's breakthrough performance. An advanced version of Gemini Deep Think solved five out of the six IMO problems perfectly, earning 35 total points, and achieving gold-medal level performance.
Re: (Score:3)
Oh, so you must have found it easy when you got your gold medal?
Re: (Score:2)
The point is we have a myriad of "tests that are hard for humans, but don't necessarily translate to anything vaguely useful". In academics, a lot of tests are only demanding of reasoning ability because the human has limited memory. Computers short on actual "reasoning" largely make up for it by having just mind boggling amounts of something more akin to recall than reasoning (it's something a bit weirder, but as far as analogies go, recall is closer).
It's kind of like bragging that your RC boat could ge
Re: (Score:2)
The five states of grief are denial, anger, bargaining, depression, and acceptance.
You seem to be stuck on denial.
Re: (Score:2)
I'm not in denial, the LLMs and other forms of AI have utility, but expectations have to be mitigated.
Was in a discussion with a software executive a couple weeks back who said he fully anticipates he can lay off every one of his software developers and testers in the next year and only have to retain the 'important' people: the executives and sales people.
People see articles like this show how LLMs enable computing to reach another tier of 'stupid human tricks', which is certainly novel, but people overext
Re: (Score:2)
It's kind of like bragging that your RC boat could get a gold medal in the Olympic 1500 meter freestyle. It didn't complete the challenge in the same way, and that boat would be unable to, for example, save someone that is about to drown, because the boat can just go places, it can't do what that human swimmer could do. That person swimming 1500m itself is a useless feat of interest, and not really directly useful in and of itself.
Nice example.
Re: (Score:2)
It's kind of like bragging that your RC boat could get a gold medal in the Olympic 1500 meter freestyle. It didn't complete the challenge in the same way, and that boat would be unable to, for example, save someone that is about to drown, because the boat can just go places, it can't do what that human swimmer could do. That person swimming 1500m itself is a useless feat of interest, and not really directly useful in and of itself.
Let's extend your analogy a bit more. An actual boat, not just a little RC, with an engine can go faster and farther than any human. Yes the boat cannot rescue a person by itself, but it doesn't need to. And if you are drowning two miles off shore I strongly suspect you'd rather the rescuers get in a boat and motor over to you and then rescue you rather than trying to swim out two miles. The question is not "Is this technology identical to a human?"
Re: (Score:2)
I chose RC boat because some people are using this competing at "stupid human trick" as an example of intrinsic proof of LLMs being able to supersede humans.
In the scenario of an olympic swimming competition, an autonomous boat vs a manned boat would show no difference to each other, both would compete the task much better than a human. It's a useless test to measure general utility. Just like a person swimming a 1500 meter distance is not really a useful indicator on its own of how useful they are. Thes
Re: (Score:2)
AI Training (Score:1)
I have to wonder. Did "Gemini Deep Think" solve the problems or simply regurgitate the answer from the billions of sucked up webpages, math research papers, etc. used to train the model? Actual competitors don't have the complete history of https://math.stackexchange.com... [stackexchange.com] at their fingertips.
Re:AI Training (Score:5, Informative)
Re: AI Training (Score:1)
Thank you
Re:Ok (Score:5, Interesting)
Yes, they are high school students, but the students who get gold medals are students who frequently started studying for the IMO in 9th or 10th grade, and sometimes even earlier. And yes, mathematicians aren't always going to see the trick that a given problem relies on. And it is true that IMO problems often involve tricks or approaches that one can study for. The problems are not research mathematics, and a lot of very good mathematicians never did well at the IMO as a young person. At the same time, some problems which very hard in the IMO are things where mathematicians in specific areas would not have trouble. For example, P3 of 2007 is a tough graph theory problem https://artofproblemsolving.com/wiki/index.php/2007_IMO_Problems/Problem_3 [artofproblemsolving.com] but if one has done a lot of graph theory it may seem simpler. But the IMO also frequently involves some problems which require a degree not just of pre-existing knowledge, but also something we would normally call creativity or involve concepts that are just not standard. For example, P6 from 2014 involves a problem where the idea of the problem is essentially a creative thematic connection between geometry and Ramsey theory https://artofproblemsolving.com/wiki/index.php/2014_IMO_Problems/Problem_6 [artofproblemsolving.com]. P6 from 2009 is another interesting way things can go https://artofproblemsolving.com/wiki/index.php/2009_IMO_Problems/Problem_6 [artofproblemsolving.com] and is a curious one because although very few people got it right, once one has seen the solution it feels completely obvious (unlike some other IMO problems where even after having seen a solution it isn't clear where it came from). (Note that traditionally they aim it so that Problem 6 is the hardest problem.)
Your last point that these problems are more suited to solving with AI on a whole probably has some validity. A lot of the geometry-style IMO problems are highly narrow in framing, and there's been a lot more success with AI with those problems. And all IMO problems are in an important sense easier than genuine research problems, because you know either what you need to prove or very close to it, whereas one of the big issues in research is that you are often spending a massive amount of time trying to prove something that turns out to be false. And the IMO problems are also selected so that they do not require any "advanced" techniques such as calculus, which also drastically reduces what one's functional search space looks like. In fact, one issue some LLM AIs had early on when trying to do IMO problems is that the AIs would sometimes try to jump to developing solutions which used high-powered techniques which just weren't useful in that context. So no this isn't research math, but it remains extremely impressive and shows how fast these systems have gone. My own opinion has been from where things have gone in the last two years that the current AI systems would likely fizzle out in the sense discussed here https://scottaaronson.blog/?p=7266 [scottaaronson.blog]. This is evidence that I was wrong in that assessment.
In terms of where AI and research mathematics is going, we're still not there. But we are getting closer to the point where AI systems can be genuinely useful. For example, not too long ago, I was looking for a specific result of a type I had seen before, and I asked an LLM about it. The LLM hallucinated a bunch of junk, but it also kept hallucinating papers by a specific mathematician, and it turned out that there was an actual paper by that person that had the sort of result I was looking for. But more direct use of AI for not just looking up things like that but to do actual research is being developed. The major hope is that we'll use systems like Lean http [wikipedia.org]
Re: (Score:3)
Re: (Score:2)
That's the basics of what the Google system
Re: (Score:1)
But what about putting them into a Computer Algebra system? We have had these for decades now. In fact, I used one when I started my CS studies 35 years ago.
The very point of such a competition is to have a human do it, not a machine.
Re: (Score:2)
Re: (Score:2)
but the usefulness of the IMO as essentially a natural metric of how effective these AI systems are at difficult reasoning problems.
Since LLMs have zero reasoning capability (the math does not allow it), it is obviously a failure at this.
Re: (Score:3)
Re: (Score:2)
Or that watches based on quartz crystals are just an idle curiosity because they aren't built using intricate gear assemblies.
Re: (Score:2)
No, they are not designed to reason. You're arguing from ignorance of the subject just using rhetorical techniques. You're mistaking technical questions with real answers for philosophical questions where you can just blow any horseshit you want out your ass and then use rhetoric to "argue" it.
And your rhetoric is mostly red herrings and strawmen.
Re: (Score:2)
You're mistaking technical questions with real answers for philosophical questions where you can just blow any horseshit you want out your ass and then use rhetoric to "argue" it.
What technical questions do you think I'm mistaking for philosophical questions? I must confess also that I find it somewhat strange that you make a claim about strawmen and then say "No, they are not designed to reason" as if that somehow responds to what I wrote where I explicitly said that the question about whether an LLM AI can reason is a distinct question from whether they can solve problems humans solve via reasoning. So it seems very odd to tell me that they are not designed to reason. So what straw positions are there here other than yours?
Re: (Score:2)
You're mistaking technical questions with real answers for philosophical questions where you can just blow any horseshit you want out your ass and then use rhetoric to "argue" it.
What technical questions do you think I'm mistaking for philosophical questions? I must confess also that I find it somewhat strange that you make a claim about strawmen and then say "No, they are not designed to reason" as if that somehow responds to what I
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
You can't do that. You have to actually look at what the machine is doing.
Re: (Score:2)
Re: (Score:2)
That nicely sums it up. Most of these people (like the one you responded to) have zero clue how LLMs actually work and what limits that imposes. They just want to believe like fucking morons in a cult.
Re: (Score:2)
You are so utterly without insight here, it is embarrassing. If it is a benchmark for difficult reasoning questions, it must measure reasoning or it is a failure. Take chess: It is not a reasoning problem to a computer. It is, in part, to a human. But because it is not to a computer, it is not suitable as a benchmark for reasoning capabilities.
Re: (Score:3)
You are so utterly without insight here, it is embarrassing. If it is a benchmark for difficult reasoning questions, it must measure reasoning or it is a failure. Take chess: It is not a reasoning problem to a computer. It is, in part, to a human. But because it is not to a computer, it is not suitable as a benchmark for reasoning capabilities.
Please reread the last two sentences of my previous reply since they address exactly this sort of issue. Whether humans use reasoning to do something doesn't mean something cannot be solved any other way, hence the analogy with the birds. If you think there's a problem with that analogy, by all means please explain it. That said, given that in the other subthread you are actively refusing to test a claim you've made that should just take you a few minutes, I'm guessing that this is not going to be a terribl
Re: (Score:2)
To justify that assertion you must have a very specific definition of what "reasoning" entails. Would you care to share it?
In the definition of reasoning that I use, e.g., alpha-beta pruning would count as reasoning. Clearly you disagree.
Re: (Score:2)
In the definition of reasoning that I use, e.g., alpha-beta pruning would count as reasoning. Clearly you disagree.
That's a really good point, and I would love to hear the GP's response! Why isn't alpha-beta pruning considered reasoning? Why isn't A* search over a generated graph considered reasoning?
I've said it before, but I truly do think many of the most vitriolic anti-LLM voices come from a place of being worried about what LLM capabilities say about the nature of human intelligence and reasoning. Perhaps the bottom line is that human intelligence is not so special as we make it out to be?
Go stands out to me as an
Re: AI Training (Score:2)
At what point do humans become unable to reason, according to your rather strict requirements?
Re: (Score:2)
Since we do not know how the human mind works (or often fails, as this AI hype nicely shows), I have no idea. But I am a PhD level CS type and I have followed AI research for 35 years now. At one point I was considering making it a career myself, but the constant lying and overpromising turned me away.
And no, I do not use a narrow definition. I use one that makes sense. Unless you want to sell something and then a vacuum cleaner becomes "intelligent" and borderline AGI.
Re: (Score:2)
Maybe you've seen this paper, maybe not:
"Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, Gemini-2.5-Pro, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical ta
Re: AI Training (Score:2)
In the mean time maths and physics students are using them at masters level to explain questions, give step by step explanations of proofs and create test questions.
The whole "can it reason" is arguably a moot issue if LLMs are better at maths and physics than 99% of the population.
Re:AI Training (Score:4, Insightful)
Re: (Score:1)
If you had spent even a few minutes actually learning about LLMs you would understand that training data is not directly stored in an LLM. In fact, you don't even need to research it, you can just use your brain and look at open releases. You can load the 8B parameter release of deepseek into 24gb of ram. But it was trained on 14.8 Trillion tokens (approximately 50TB of data). You cannot store 50TB of data in 24GB.
Re: (Score:1)
You're not even wrong. Perhaps start by taking a first year course on data compression. Once you get to a point where you understand how JPEG works, you will have everything you need to conceptually understand how it is possible for an LLM to represent its training data inside the parameters.
If you're lazy, then I'll just point you to the fact that LLM exploits have been found that make the LLM spit out its training data in clear text. [theregister.com]
Re: AI Training (Score:2)
Only some locally coherent snippets. And that only after poking it for a long time. If you claim the LLMs will puke up their entire training set, just take that to the lawyers for the authors in the case vs OpenAI. They'll be quite happy to see you.
A computer that can do math? (Score:2, Funny)
What will they think of next?
Re:good (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
Unfortunately, the economic justification for AI assumes problems of the caliber of the Riemann Hypothesis will be solved
Does it? It would be fascinating if AI could eventually solve problems like that, but I think the economic justification is more along the lines of "we have billions of dollars of capital available, billions of people who want to use this technology, tens of thousands of other companies working on AI technology, and we don't want to be left behind when the next big discoveries are made."
Re: good (Score:2)
Yet they are already answering some long standing unsolved problems. So while the Riemann hypothesis may be out of reach, the tools required to crack it may still come out of an LLM.
Re: (Score:2)
And other AI engines? (Score:1)
I have in mind, the latest Grok 4 Heavy, was he tested? I didn't saw any information about other engines doing that test to compare.
So? (Score:1)
I have no doubt that Maple or any other decent Computer Algebra system could have done the same ... 30 years ago. Or Wolfram Alpha.
This is a completely meaningless stunt. The only purpose is to deceive the stupid about what these systems can do, or rather cannot do.
Re: (Score:2)
Re: (Score:1)
You're replying to slashdot's local resident AI retard! Don't waste time reasoning with him. It won't work. For some reason, he feels it is his duty to shit on anything related to AI. He will never give any explanation or reasoning behind his comments and doesn't care what anyone says.
Re: (Score:2)
Why would I waste time in what is clearly a failed technology? The stupid always need years and years to find out a hype is just a hype and never has any of the substance claimed. I can do it directly.
Re:So? (Score:4, Insightful)
Re: (Score:2)
Not interested. You are deep in delusion and cannot be reached.
Re: (Score:2)
Not relevant to your post, but I appreciate your insights as a mathematician on this article (says the guy who tapped out after 3rd semester calc!). For what it's worth, I don't _think_ gweihir is trolling, though he could be be, but he does seem to spend a substantial number of his waking hours posting ad hominems and just outright calling people stupid if anyone suggests any possible productive usage of an LLM. He doesn't engage in good faith, so the conversations are never particularly interesting.
Re: (Score:2)
Ah, the gaps close even tighter.
"I'm totally sure X or Y could do that."
"Okay, try it!"
"Why would I waste my time?"
Why indeed. You might find out that you're wrong.
Re: (Score:2)
You might find out that you're wrong.
Or I could waste a lot of time. Which is the more likely case.
NOTHING OFFICIAL AT ALL (Score:1)
The IMO is pretty pissed off at Google because the results are embargoed from publication until the 28th of July.
Apparently Google did their own metrics and claimed the Gold-Medal. IMO is not happy,
https://arstechnica.com/ai/202... [arstechnica.com]
But hey, good on Goog for demonstrating some level of success, whether IMO gives them a gold medal or not.
Shame on them for violating the publication embargo, but I haven't read that contract or those T&Cs so I only
judge by what IMO says and what Goog says.
Complete Fail (Score:2)
What's worse than the self-grading is that the rules include not using calculators.
How can an AI do math without using a calculator?
There is no way a computer anything can score even 1% using the same rules as humans.
Re: (Score:2)
Re: (Score:2)
I think when people talk about calculator assistance for LLMs, they are conflating a general class of approaches to bolt on reasoning capabilities into those LLMs by delegating arithmetic, and delegating logic, to specialized, provably correct software tools. These tools are not AIs themselves and do not use ML algorithms in their implementations, so should be considered traditional software. The tools are necessary though, as the Chatbot/language models alone are incapable of performing arithmetic, and th