Cisco Blamed A Router Bug On 'Cosmic Radiation' (networkworld.com) 145
Network World's news editor contacted Slashdot with this report: A Cisco bug report addressing "partial data traffic loss" on the company's ASR 9000 Series routers contended that a "possible trigger is cosmic radiation causing SEU [single-event upset] soft errors." Not everyone is buying: "It IS possible for bits to be flipped in memory by stray background radiation. However it's mostly impossible to detect the reason as to WHERE or WHEN this happens," writes a Redditor identifying himself as a former [technical assistance center] engineer...
"While we can't speak to this particular case," Cisco wrote in a follow-up, "Cisco has conducted extensive research, dating back to 2001, on the effects cosmic radiation can have on our service provider networking hardware, system architectures and software designs. Despite being rare, as electronics operate at faster speeds and the density of silicon chips increases, it becomes more likely that a stray bit of energy could cause problems that affect the performance of a router or switch."
Friday a commenter claiming to be Xander Thuijs, Cisco's principal engineer on the ASR 9000 router, posted below the article, "apologies for the detail provided and the 'concept' of cosmic radiation. This is not the type of explanation I would like to see presented to the respected users of our products. We have made some updates to the DDTS [defect-tracking report] in question with a more substantial data and explanation. The issue is something that we can likely address with an FPD update on the 2x100 or 1x100G Typhoon-based linecard."
"While we can't speak to this particular case," Cisco wrote in a follow-up, "Cisco has conducted extensive research, dating back to 2001, on the effects cosmic radiation can have on our service provider networking hardware, system architectures and software designs. Despite being rare, as electronics operate at faster speeds and the density of silicon chips increases, it becomes more likely that a stray bit of energy could cause problems that affect the performance of a router or switch."
Friday a commenter claiming to be Xander Thuijs, Cisco's principal engineer on the ASR 9000 router, posted below the article, "apologies for the detail provided and the 'concept' of cosmic radiation. This is not the type of explanation I would like to see presented to the respected users of our products. We have made some updates to the DDTS [defect-tracking report] in question with a more substantial data and explanation. The issue is something that we can likely address with an FPD update on the 2x100 or 1x100G Typhoon-based linecard."
That's my go to excuse, too (Score:1)
Re: (Score:2)
Re: (Score:1)
Maybe cosmic radiation is effecting them too?
Re: (Score:2)
Yet she still had time to write the Realm-Jumper Chronicles, 5 volumes and counting. Colour me impressed.
Alternate explanation (Score:1)
I'm not saying it was aliens, but...
It was aliens!
Re: (Score:1)
99.99â... of SEUs have notihing to do with cosmic radiation.
Let me guess, you are a millennial, and to cool to use Google?
https://en.wikipedia.org/wiki/ECC_memory#Problem_background [wikipedia.org]
Re: (Score:1)
https://en.wikipedia.org/wiki/ECC_memory#Problem_background [wikipedia.org]
Actually, there was a packaging material that caused this problem, back in the late '60s. They reaserched it and stopped using that, but were astonished to find it was still not completely cured. That's when the "Science-fiction" idea of cosmic ray impacts started to be believed.
(Strange how new people assume the "ancients" were wrong and ignorant, when they were usually smarter than the newbies.)
Bad memory... (Score:2)
Re:Bad memory... (Score:5, Insightful)
Or perhaps something like a design flaw in memory [wikipedia.org] that's provable and repeatable, and has even been used for conceptual security attacks.
Still, when you start looking at crash reports from millions of customers (I used to work on a fairly well-known MMO), you see stuff that simply shouldn't be possible, and you start wondering about things like cosmic radiation. We had to filter out what we figured were hardware-based errors due to overclocked CPUs, bad RAM, etc, or else you get flooded with impossible crash stacks.
x = 3 * y; // Crash here! WTF?
Re: (Score:2)
x = 3 * y; // Crash here! WTF?
Maybe y ~ 2^31 and the CPU doesn't support overflow...
Re: (Score:1)
x = 3 * y; // Crash here! WTF?
Quite a few opportunities that doesn't involve external causes:
Overflows.
y is a pointer to an address that can't be read.
y is a macro which causes a divide by zero or other exceptions.
Using a C compiler that doesn't take C++ style comments
Re: (Score:2)
Lol, I knew I was going to get comments like that. Christ, it was just an example. Assume this is simple integer arithmetic in a well-defined range, ok? And I'm still trying to figure out what sort of C compiler that doesn't understand C++ comments would generate runtime code that crashes instead.
Honestly, there are seriously whacked PCs in the world (especially badly overclocked gaming PCs) that try to argue that 1 + 1 == 3.
Re: (Score:1)
overflows in C don't cause immediate crashes
Where in any C standard does it say that.
"overflows in C [ compiled by most compilers ] [ run on most common hardware ] don't cause immediate crashes" is probably true.
But an overflow in signed integer calcutaltions on a Harris 800 or a Tandem TXP, to cite two examples I have real experience with, would cause an immediate crash.
Re: (Score:1)
Quite right. And by way of actual observation on Cisco TAC engagement with a top tier Service Provider... both bad memory and cosmic radiation have surfaced in the debug and diagnostic phase of the trouble ticket. It is not easy to then get the customer to ignore mention of 'cosmic radiation' and make progress on the real issue (timeline, early 2000's). And guess what surfaces in the next service contract negotiation.
Van Allen radiation belts (Score:1)
Re: (Score:1)
The Earth's magnetic field is great for deflecting most of the stuff that come from the Sun, but cosmic rays, as in the stuff from outside the solar system, includes a lot of high to very high energy particles that are not much deflect by the same field. It can actually be worse in the atmosphere than in orbit too, depending on your exact setup. A single high energy particle directly hitting something in space might deflect a single atom, but a similar particle hitting the upper atmosphere deposits enough
Re: (Score:2)
Re: (Score:2)
I can't claim to know the cause, but I have seen one proven case of a bit flip affecting processing on a machine that ran fine before and after the incident. Since it was doing batch processing I was able to re-run the job with identical inputs. It never made the error again.
Could have been a cosmic ray, alpha decay, power glitch that oddly didn't affect the other machines on the same circuit, who knows?
Re: (Score:2)
Re: (Score:1)
I have always attributed things you mention to non-ECC ram in the past.
Non ECC ram has exactly the same number of bit-flips as ECC ram. It's just that ECC lets you know it happened (as would parity) and can fix it for you (if only one bit flipped).
Re: (Score:1)
Yup, that's obvious. I'm dumb not to have thought of that: ECC == more cells / usable bit == more events / usable bit.
We had a fault in a high altitude aircraft (>60K feet) that we are pretty confident occurred when a cosmic ray flipped a bit inside the error correction circuitry.
Hilarious. As you add protection you add places to be broken :-( Need error correction circuitry for the error correction circuitry.
Re:Van Allen radiation belts (Score:5, Interesting)
Re:Van Allen radiation belts (Score:5, Interesting)
Because working with geosynchronous satellites I learned that it mostly affects them, and satellites in LEO aren't affected very much, and the ground has quite a bit of atmosphere for additional shielding.
While satellites and space vehicles may be hit with a lot more sub-atomic particles, some particles do reach the earth's surface, e.g., muons, which results in about 13 to 14 neutrons/cm^2/hr with sufficient energy to flip a bit on a chip (depending on latitude, longitude, altitude, etc.) All chip companies that care to be concerned about it know the estimates for errors/Mbit for their process technology. Some extra effort is needed to estimate the amount of error masking due to the micro-architecture and software. Many/most companies design in sufficient error detection or correction to bound the expected chip error rate to an acceptable level. However, that acceptable error rate depends on the customer. For supercomputing, nuclear control systems, and self-driving cars, the acceptable error rate is quite small, but for networking ASICs, the chip error rate may be allowed to be higher with software and network protocols providing additional error mitigation.
Re: (Score:2)
Re: (Score:2)
Of course it depends where you are on the ground. I used to work in a data center in Colorado Springs, at about 6000 feet altitude. We saw quite a few correctable memory errors in the logs (and a few random crashes).
Might have been cosmic rays, might have been radiation from the mountain of granite (Pikes Peak) we were in the shadow of.
Either way, if the errors occurred several times in the same DIMM, it was probably bad memory and we replaced it. The odds of cosmic rays hitting the same DIMM every few d
Re: (Score:2)
Of course it depends where you are on the ground. I used to work in a data center in Colorado Springs, at about 6000 feet altitude. We saw quite a few correctable memory errors in the logs (and a few random crashes).
The soft error rate do to cosmic rays is an order of magnitude higher at Colorado springs than at ground level. For aircraft it is another order of magnitude higher but they have a very small capture area; I assume safety critical aircraft systems are designed with this in mind. Placing systems in a deep basement has the opposite result do to greater shielding.
A lot of memory*hours are needed for it to become a significant problem which is why systems with either lots of memory or systems which run contin
Re: (Score:3)
Re: (Score:1)
You presented no argument because you have none. That's why you're being attacked as the fool you are. Hint: It is better to remain silent.
Re: (Score:3)
Not buying it (Score:3)
Re:Not buying it (Score:5, Interesting)
Sorry, but cases such as this exist.
Back around 1999/2000 I was with Cisco engineering on the GSR 12000 (the first Cisco service provider class router).
We did send a system to a POP in Denver (altitude 5000+ ft) and saw on this system a statistically significant increase in recoverable memory ECC errors.
When the affected board was returned to San Jose and retested (basically sea level) the errors could not be reproduced.
So we returned the hardware back to the Denver POP, and the recoverable ECC errors returned. No amount of swapping memory DIMMs (various vendors) made a difference.
Any satellite hardware designer will tell you that cosmic radiation is a big deal for satellite design. And lead shielding is not a cost effective option in space.
Re: (Score:2)
Sure, it could be that. But it could also be:
Re: (Score:2)
A cleaning person plugging a vacuum cleaner into the power strip on the rack instead of into the wall outlet that's on an external circuit (combined with improper power filtering in the equipment).
Even shitty chinese powersupplies filter this out to an acceptable level to make this a non issue.
Electrical noise caused by some other crappy piece of equipment in the rack (combined with improper power filtering in the equipment).
Even shitty chinese powersupplies filter this out to an acceptable level to make this a non issue.
Errors caused by higher operating temperature.
Unless the equipment has an appreciable difference in operating envrionment it would be insignificant. It's also one of the first things you do when checking failures is do a quick check of the installation equipment, especially if you're in a data centre or other envionmental controlled situation.
Errors caused by emissions from natural Uranium or other radioactive elements in the soil.
You mean just lik
Re: (Score:2)
But that is not cosmic radiation.
That wold be occurring regardless where you are.
My guess is a batch of the chips in those routers has contaminated silicon. The radiation is likely coming from inside. Does not need to a health risk high contamination, just a random increase in a phosphor isotope or something.
Re: (Score:2)
Sorry you are incorrect. The EXACT SAME hardware did NOT malfunction at sea level altitude, but when relocated to Denver at 5000+ ft above sea level it displayed a statistically significant increase in single bit ECC errors.This behavior has been studied by numerous organizations, including IBM, Sun Microsystems, and others. See this IEEE technical talk: http://www.ewh.ieee.org/r6/scv... [ieee.org]
Re: (Score:2)
5000ft difference in hight has no influx on cosmic radiation.
Much more likely the buildings had.
Re: (Score:1)
Re: (Score:2)
Re: (Score:3)
It is a reasonable explanation. Memory has parity bits. There are random faults from the various sorts of noise you can get in semiconducting circuits, but if you have some safety-net that will catch the occasional flipped bit. Your computer will be catching these sorts of errors all the time. The problem with cosmic rays is that they are very energetic, so they can pass through a lot of matter, but when they collide they generate a tight cone of ionising particles that knocks out electronics in a small re
Re: (Score:2)
Some Memory has parity bits.
Fixing that for you.
I bet your PC or Laptop has no memory with parity bits.
Re: (Score:1)
It is calculated that the average personal computer will be hit by this about once a year. It's has been about the same since the 1960's. Cell size gets smaller, which reduces the chance, but number of cells gets greater, increasing the chance. The chips are about the same overall size, which is what makes the "target" area.
But operating systems such as Windows crash much more often than that, so nobody notices unless they have high-reliability equipment and track faults.
It's not an excuse or a fairy tale,
'Cosmic Radiation' can corrode credability (Score:1)
even if you have a strong support organization, one slacker responding with this to a customer, and the entire brand is tarnished.
Re: (Score:2)
Is there more detail in the logged in bug report, as the one you linked to without login says nowhere cosmic radiation, just SEU, which can be caused by many things and is just a generic term for a bit flip.
happened to me (Score:4, Funny)
Re: (Score:2)
On a related note, there is the infamous (in narrow circles) issue of the serial consoles on old Unix systems. Many had an option to "press any key for boot menu" on the serial console. The problem was that the serial consoles would get enough static interference to occasionally detect a character while this option was available, and it would halt the boot process. On a datacenter reboot (usually due to power loss), a handful of servers would never come up because of this. It was far more reliable to re
Solar Flares? (Score:3)
I'm guessing that they've read the BOFH [andrews.edu], but realized that there's much more reporting on solar-induced radiation ... so just decided to go with 'galactic' instead. .... completely forgetting that if this were the case, it would happen more frequently at high latitudes, due to the magnetosphere. And we'd also see a higher incidence rate after solar x-ray flares and solar particle events.
(and the disclaimer: I work for the Solar Data Analysis Center, but I'm not a scientist, and don't speak for my place of work, etc, blah blah blah)
strong elevation effect too.. (Score:3, Informative)
As discovered by IBM back in the 70s, if it is a radiation induced upset, you'd see higher rates in places like Colorado vs Sea Level, and on upper floors of building vs lower floors.
Uh huh... (Score:1)
Sun Microsystems already pulled this bullshit back about 15 years ago... I don't really recall if it was a bad batch of processors or if it was bad non-ecc cache memory or whatever... but I do remember plenty of folks giving them a ration of shit and generally refusing to buy hardware from them after that... though once they fessed up to the problem and replaced all of our defective systems(and gave us a couple of free systems) we never had any further issues.
Re: (Score:2)
The SRAM structures used for integrated high performance processor cache are orders of magnitude more sensitive than discrete DRAM to radiation induced soft errors. Some of this is simply because the bandwidth is so high which exposes a greater capture area of logic. And so high performance processor cache has included parity and ECC protection for a long time.
Trying to clam acts of god to get out of being (Score:2)
Trying to clam acts of god to get out of being responsible?
Re: (Score:2)
It reminds me of people blaming compiler bugs for non-working code. While it does happen that a compiler generates incorrect code (I've encountered a few instances over the years), unless you either have reduced the problem to a minimal test case or examined the generated assembly and located the problem there, it's far more likely that it's a case of not digging deep enough to find a bug in your own code.
Cisco is getting worse... (Score:1)
Re: (Score:1)
They will be losing market share... us included... we're a global company, with a size akin to a fortune 100. We're pulling the plug and moving to Juniper. Too much horse shit from Cisco. It's a fucking nightmare to get a quote, or even get an order filled CORRECTLY. We get the wrong shit sent all the time, and Cisco says the internal RMA process is so tedious, they tell us to throw the wrong equipment in the garbage or just keep it.
They don't stock shit. Everything takes weeks. RMA'd equipment takes weeks
Re:Cisco is getting worse... (Score:5, Interesting)
One of the VP level Engineers (title is "distinguished" or something exalted like that) told me over lunch a couple of years ago that Chambers had said to him he wasn't interested in R&D. If there was a technology he needed, he'd buy it.
The problem is that Cisco climbed to the top using IBM strategies and thinking which were focus on delivering "end to end" solutions to customers.
They had no interest in box shipping. Those were just lego bricks and logistics. You can imagine how soul destroying that was to be a Cisco engineer.
Bugs were a bonus to them as they sold annual maintenance contracts for roughly the same cost as the gear they sold.
Now that the router/switch market has matured and commoditised they care even less about the quality of those boxes they have to ship.
Their focus is entirely on the "service" level.
They will eventually become another IBM. I was trying to think of a real tangible product that IBM made and sold just the other day. Do they?
Re: (Score:2)
I think they're still selling POWER-based systems.
Re:Cisco is getting worse... (Score:4, Interesting)
They will eventually become another IBM. I was trying to think of a real tangible product that IBM made and sold just the other day. Do they?
They sell more mainframes now than they did in 1970, believe it or not. One z13 can support 8,000 Linux VMs simultaneously. Cool looking box, too.
Re: (Score:2)
I seem to run across a fair number of places with AS/400 stuff. What's kind of interesting is the AS/400 stuff and people seem to run in this parallel universe IT department with their own staff somehow immune from the other pressures of the rest of the IT department.
Once in a blue moon I'll hear mention that some kind of AS/400 update or installation is happening, so it's not like they're strictly legacy systems. And at longer term clients with AS/400 I occasionally see something new/different in the "AS
Head cover protest (Score:2)
This makes me wish I was still working for them in IOS Engineering for the opportunity just to stir some shit.
I'd have gone into the office on Monday morning with my head covered in tinfoil.
I use to get some good laughs in the Cisco office, I seem to be getting more on the outside these days.
Oh how the mighty have fallen....
Not bloody likely (Score:5, Informative)
Re: (Score:2)
Given that alpha emissions are trivially blocked by something as thin as a sheet of paper I take that citation with a grain of salt.
Re: (Score:2)
"Alpha radiation" is always from a nuclear decay. That is how the discoverers "named" it.
An Alpha particle, is a helium core, an atom without electrons, an ion.
An cosmic alpha particle has energies that go far beyond your imagination. You don't shield them with a sheet of paper. Hence we gave them a different name "cosmic ray".
Re: (Score:2)
Interesting. Do you know how long ago that study was done? I"m curious if smaller manufacturing geometries have made newer processors more vulnerable.
Re: (Score:2)
Interesting. Do you know how long ago that study was done? I"m curious if smaller manufacturing geometries have made newer processors more vulnerable.
The sensitivity of DRAM actually leveled off a few generations ago. I think what happened is that there is a minimum capacitance needed per DRAM cell so as the cells became smaller and the dielectric constant was increased to make up for it, the charge stored in a given volume became *greater* so an ionizing radiation impact spreading charge over a greater number of DRAM capacitors without enough charge to affect them individually.
High performance SRAM used for integrated caches became more vulnerable and
Wake up sheeple! (Score:1)
Santa Claus spread chemtrails in the sky with which the easter bunny got stoned and confused causing the routers to crash!
Hey, it's not impossible!
The BOFH now works for Cisco? (Score:2)
That was in his excuses rolodex.
Stray Gamma Rays.. (Score:1)
I used to use that all the time. Now I'll have to think of something else..
Of course this happens (Score:2)
Flips of a single bit in a memory or register are that few modern systems would run for long without error correcting memory. Even ECM has its limitations and most systems eventually crash/panic/blue-screen or whatever and require a reboot.
The costs to improve error resilience go up rapidly and don't have a meaningful upper bound. My working trade off was to design for a mtbf comparable to how long I wanted to keep that job.
I remember in the 70s some memory manufacturer use (Score:1)
Re: (Score:2)
I guess in this case it is "the same thing" ... the silicon from which they made some of the chips involved was not pure enough, or the material for doting was contaminated.
Are sure they didn't mean... (Score:2)
...a Cosmic Brownie?
http://cosmicbrownies.littlede... [littledebbie.com]
On a router that expensive (Score:2)
It shouldn't be a huge expense to build in some form of error correction to catch that sort of thing.
Re: (Score:1)
It shouldn't be a huge expense to build in some form of error correction to catch that sort of thing.
Otherwise known as ECC memory?
Global Warming next? (Score:2)
My wife was looking over my shoulder when the "Cisco Blamed A Router Bug on 'Cosmic Radiation'" headline went by, and asked:
"What's their next excuse? Global Warming?"
Next week (Score:1)
Radiation can indeed cause memory errors (Score:2)
Total bullshit, SEUs are fixable (Score:3)
There has been assloads of research on mitigating soft errors going back to the 1970’s. I’ve published some myself. There is no shortage of workable methods on masking transient errors in logic and bit flips in DRAMs. SEUs are a major problem for supercomputers, so their memory systems have sophisticated mechanisms for catching them.
If Cisco is blaming this on SEUs, that just proves their incompetence, since they obvious didn’t spend 5 minutes with Google Scholar looking at hundreds of GOOD papers (in the top conferences and journals) on this topic. Seriously.
PLUS, if something goes wrong, even if it IS a transient error, it’s FAR more likely to be a fixable bug than radiation. We had a weird bug in a DRAM controller whose state kept going invalid. We had to add another circuit to fix that. We *called* is a cosmic ray deflector, but the more likely causes, in order were (a) another bug we couldn’t find, (b) a timing violation caused perhaps by voltage or temperature fluctuation, or (c) crosstalk in the circuit. We would have kept looking, but this deflector circuit made it robust to hundreds of hours of slamming the memory system, so we let it go. (Also, it was graphics memory, so even if it did ultimately suffer a glitch some day, it would go unnoticed.)
Not new (Score:2)
I have had Cisco tell me this many times any time a router reboots from a parity error for over 15 years now, so they have been using this for a long time now.
Reasonable - but not enough data (Score:3)
Bwahahahahah.... (Score:1)
My reaction when I first heard the "cosmic radiation" excuse for misbehaving electronics.
With decades of experience in tech implementations in radiation fields I can personally attest to the fact that the radiation flux levels needed to cause reactions in electronics could only be high enough due to cosmic radiation at elevations higher than 20,000 feet. The levels need to be in Rad per hour rather than the microrad per hour that you get from cosmic radiation. (i.e. background at s
Re: (Score:1)
It only takes a single "cosmic ray" particle to flip a bit in memory. The readings that are averages, are no good for this.
And all this has been known for about 50 years... ;-)
This paves the way to new ways for DDOS attacks (Score:2)
"Cosmic Radiation Bug"? (Score:1)