Crowdfunded, Solar-powered Spacecraft Goes Silent 366
Last week saw the successful launch of the Planetary Society's LightSail spacecraft, the solar-powered satellite that runs Linux and was crowdfunded on Kickstarter. The spacecraft worked flawlessly for two days, but then fell silent, and the engineering team has been working hard on a fix ever since. They've pinpointed the problem: a software glitch. "Every 15 seconds, LightSail transmits a telemetry beacon packet. The software controlling the main system board writes corresponding information to a file called beacon.csv. If you're not familiar with CSV files, you can think of them as simplified spreadsheets—in fact, most can be opened with Microsoft Excel. As more beacons are transmitted, the file grows in size. When it reaches 32 megabytes—roughly the size of ten compressed music files—it can crash the flight system." Unfortunately, the only way to clear that CSV file is to reboot LightSail. It can be done remotely, but as anyone who deals with crashing computers understands, remote commands don't always work. The command has been sent a few dozen times already, but LightSail remains silent. The best hope may now be that the system spontaneously reboots on its own.
Seriously? (Score:5, Insightful)
I’m usually the first to defend others when some bug like this makes it through testing. Hindsight always being 20/20, only takes one bug amongst a million good bits of code, etc. But this just seems like something that even basic testing should have caught.
Did they not run this thing on the ground for a few weeks? That’s just basic testing, especially for something that is going to be inaccessible for a while. Also that some critical bit of processing relies on stuff being written (and then presumably read back from) a csv file is very worrying.
This sounds like some very shoddy work.
Re:Seriously? (Score:5, Insightful)
Re:Seriously? (Score:5, Informative)
Their current plan is to wait charged particles to affect electronics so that it forces a reboot.
Spacecraft are susceptible to charged particles zipping through deep space, many of which get trapped inside Earth’s magnetic field. If one of these particles strikes an electronics component in just the right way, it can cause a reboot. This is not an uncommon occurrence for CubeSats, or even larger spacecraft [planetary.org], for that matter. Cal Poly’s experience with CubeSats suggest most experience a reboot in the first three weeks; I spoke with another CubeSat team that rebooted after six.
Re:Seriously? (Score:5, Funny)
Their current plan is to wait charged particles to affect electronics so that it forces a reboot.
Watchdogs are for wimps. Real designers use supernovas in a distant galaxy to reset their boards.
Re:Seriously? (Score:5, Funny)
Re:Seriously? (Score:5, Insightful)
Testing might have found it, but I'd say that regardless of testing they should assume something bad will happen with the software and have a mechanism in place to force reboot & update on a locked up system. Maybe they thought they did. Its a shame if they can't get it fixed.
Speaking as an embedded developer, this is completely inexcusable.
Not having a Watchdog, PLUS not making the limited-filesize log file "roll-over", is clearly Amateur-Hour stuff. Who wrote this code, anyway? An eight year old???
Next we're going to hear that they bricked it with a software update, because they didn't think they needed to checksum the uploads, or provide enough RAM to hold the updated code before they re-flashed the OS, or something similar.
Pathetic. They deserve to lose their spacecraft.
Fortunately, if extraterrestrials discover the floating hulk of this abomination, they will (rightly) conclude that there is no intelligent life worth exploiting on this planet, and will decide not to enslave us...
Re:Seriously? (Score:5, Insightful)
Not having a Watchdog, PLUS not making the limited-filesize log file "roll-over", is clearly Amateur-Hour stuff. Who wrote this code, anyway? An eight year old???
It's not even who wrote it, it's who designed it. Reading the summary actually made me angry that there is a group of people out there somewhere with the ability to build, launch, and track a satellite but without the common sense to recognize that they're creating a system that will grow infinitely in size without a mechanism to clear that data out. Does the satellite have unlimited storage space available? No? Then how about designing a way to monitor and clear the data other than saving it in /tmp?
Pathetic. They deserve to lose their spacecraft.
They definitely do. And no amount of descriptions of a CSV file meant for a grade school kid, or saying that 32MB is about the size of 10 songs, is going to minimize the schadenfreude that I'm feeling. Such a basic design error and they never even bothered to run tests for a significant period of time before putting the damn thing in space.
Way to go, LightSail team. I dub thee LightFail.
Re:Seriously? (Score:4, Informative)
These guys did not launch a satellite, ULA did. Basically LightSale simply took a ride on an Atlas 5 that was deploying the X-37B and was thrown out as a secondary payload. Pretty much anybody can do that. A lot of CubeSats are often made by college students.
Also, describing CSV and measuring files in songs makes me want to punch Bill Nye, and I love Bill Nye.
Re:Seriously? (Score:5, Informative)
One report I read made it sound like they were aware of the bug for a while. It's possible that they had to launch with an old version of the software because the patch wasn't ready yet, and being a secondary payload on a launch you have no say whatsoever as to the launch date. They probably expected to be able to upload the patch after launch, but the log filled up faster than expected.
That being said, it is shoddy programming to blindly write to a log on a resource-constrained embedded platform (or any platform, really. Just especially so on something like this), so somebody definitely goofed. All I am saying is that it probably was caught by testing, but couldn't be fixed in time due to various constraints. It was a dumb move on the developer's part to not do enough diligence and to rely too heavily on QA in the first place.
Re: (Score:2)
That being said, it is shoddy programming to blindly write to a log on a resource-constrained embedded platform (or any platform, really. Just especially so on something like this), so somebody definitely goofed.
Maybe they did not blindly write log on a resource-constrained embedded platform. Depending on how much memory they had, and how short the mission duration was, they could have computed that 32MB/(2d) * length_of_mission was sufficiently smaller than available_space
But as a test engineer in the space industry (having working on big and expensive and too much paperwork satellites, not on the new space version of cheaper, less doc and simpler tech, which really looks interesting, but I don't know to what exte
Re:Seriously? (Score:5, Funny)
32MB is all that anybody with a satellite would ever need.
Re: (Score:2)
32MB is all that anybody with a satellite would ever need.
You beat me to it!
But I think it would be more funny as "32 megabytes should be enough for anybody."
Systems Administration 101 (Score:5, Insightful)
Roll your log files. I smell a DevOps debacle.
Re:Systems Administration 101 (Score:4, Insightful)
Any competent software designer and developer should have known basic rules of embedded systems. One of them: Do not use dynamic memory (and files are just alike). If you need space all the space you need must be determined at compile or design time. BTW why store all this data in the device? This should have been (if at all) implemented as an round robbing database. Yes that overwrites old data, but who cares? If you need all the data you should have calculated the amount for the complete mission and reserved enough memory for that. And why did they use a CSV file? Are their physicists?
Re: (Score:3)
Today, though, dynamic memory allocation is a reasonable thing. Granted you want to make sure it can't fail, and that "out of memory" is handled appropriately.
I don't completely disagree but you might watch the CPPCON 2014 presentation on the Curiosity rover for some insights into how the industry actually does things. One thing I noticed right off; rad hardened hardware is way behind the latest thing from Intel.
Re: (Score:3)
some people still do code for the Z80 in 16KB of memory.
It is still in widespread use from robotic control systems to hardened consoles to law enforcement (it's the processing unit in portable breathalysers). Some modern mobile phones (some Ericsson models) still use the Z80. Some musical synthesisers use the Z80 in realtime voice processing. The Harvard Zed SBC uses a Z80 core.
Re: (Score:3)
But this just seems like something that even basic testing should have caught.
Did they not run this thing on the ground for a few weeks?
It was tested by the same guys that tested the Boeing 787 for only 247 days [engadget.com] ...
Re:Seriously? (Score:5, Insightful)
As a retired QA guy, I can tell you that checking that no files can grow without bound is standard fare. Same with exercising all code for long periods of time, as you pointed out. That means there was not a single experienced QA guy on the team.
By the way, CSV was the golden standard for many years. Given the tight compactness/memory budget that space projects have, CVS with it's small foot print might well be the logical choice.
Re: (Score:3)
By the way, CSV was the golden standard for many years. Given the tight compactness/memory budget that space projects have, CVS with it's small foot print might well be the logical choice.
We're talking about telemetry beacon data written once every 15 minutes. CSV is NOT the ideal format for that, and is nowhere near compact. Naive CSV parsers are trivial, but also break very very easily (ex. embedded new lines in a quoted field; quotes in a quoted field; mixed quotes; etc). Also, while CSV can be read in a
Re: (Score:3)
We don't need no stinkin' testing (Score:5, Funny)
when some bug like this makes it through testing
Testing? what testing? If it compiles, it works. Every hacker knows this.
I have to say, when I read that the spacecraft ran Linux and had died, I naturally assumed that someone had left the auto-update enabled and it was busy trying to apply about 50 million kernel patches.
Re: (Score:3)
Really? If a system dies after sending it a reboot command I would blame systemd.
Re: (Score:3)
Re: (Score:3)
This is something that should have been done with any number of RTOSes. FreeRTOS [freertos.org] is a good start, I prefer ChibiOS/RT [chibios.org].
Re: (Score:2)
In embedded systems, you do not go for testing (alone), you verify the system. And you certainly do not use dynamic data management, neither in RAM nor in storage (which is often the same). So you do not append something to a file. You can overwrite something in a file, but not append. And why store that data on the device anyway. An incredible whackjob, an epic fail.
Re:Seriously? (Score:4, Informative)
it's not so much the capacity of local storage, plus you have to consider that this is a system which is unlikely to be touched by a human being ever again so whatever goes up has to be physically resilient - super-compact flash storage such as micro/SD would be out, I'd go a couple generations back and use Compact Flash with slightly lower capacity to take advantage of larger dies - this is why NASA went on a shopping trip very recently for Pentium I and Pentium Pro chips for space systems, they're by virtue of their architecture, fairly hard against the environment. Back to topic, a cursory search around and it apepars that it's an issue with the kernel, sysvinit and/or php, all of which at some point or another default shared and allocated memory spaces for various purposes (including scripting and logging) to 32MB.
Re:Seriously? (Score:5, Funny)
issue with the kernel, sysvinit and/or php
So, what you are saying is that I can blame systemd? Or did I miss it and systemd is our savior?
I am confused and unsure how to be outraged by this. I am going to go eat ice cream.
Re: (Score:2)
me too. This is highly confusing.
Re: (Score:2)
Re: (Score:3)
space is not the issue, it's the way the kernel and/or apps handle memory.
Think of it like running win32 on a 64-bit chip with 8GB of RAM. It's nice having 8GB of RAM but Windws can't actually address it - it's a 32-bit kernel which means it can only address 4GB.
Some logging processes particularly those configured to write to volatile memory are constrained by default configurations (in eg. php) which *allocate* memory space for scripting in 32MB segments. This can also include the scripts themselves and th
Re:Seriously? (Score:4, Funny)
Something slashdot readers can relate to
Re: (Score:2)
Re: (Score:2)
Re:Seriously? (Score:5, Funny)
Nah, a Microsoft OS would reboot much sooner.
Punishment for using Linux: now they'll have to wait for a decade to see a reboot.
Re:Seriously? (Score:5, Informative)
No, csv sure as hell is NOT a Microsoft format [wikipedia.org].
This has nothing whatsoever to do with Microsoft, as much as you seem to want to blame them.
Re: (Score:2)
.csv is a microsoft format though.
I lol'd when I read that!
Re: (Score:3)
CSV (Score:5, Insightful)
I know the average IQ at /. has gone down over the years, but I think the explanation of what a CSV file is is slightly too much dumbing down.
Re:CSV (Score:5, Insightful)
I think the "32 megabytes—roughly the size of ten compressed music files" part is even more insulting.
Re:CSV (Score:5, Insightful)
Honestly, I'm surprised they didn't try to define space, Linux, and solar.
This sounds like someone failed to run a bench test where the system was up and running for an extended period of time.
Which strikes me as utterly bizarre.
Re: (Score:2)
Sounds more like someone forgot to put their thinking cap on.
Re: (Score:2)
Yep
Slashdot news for ?
That being said really? It crashes at 32Gigs? That is a odd bug to not get caught in testing.
Re:CSV (Score:5, Funny)
Re:CSV (Score:5, Funny)
Bill Nye the Science Guy is their CEO so...
Re: (Score:2)
Can I get that in Libraries of Congress? I mean damn, how am I supposed to really know how big it is?
Re: (Score:2)
Oh, 32MB would be like pico or fempto-LOCs (possibly even less) ... it's slightly less than 4 empty .xlsx fles (which are like CSV files, but different ;-)
You can buy about 1000x that amount of storage in the express checkout at Wal Mart for under $10.
Re: (Score:2)
yeah but the track width on a 32GB SD card would last about 3 seconds in space before a charged particle zaps across it and blows the whole deal.
Re:CSV (Score:5, Interesting)
To be fair, that was copypasta from TFA. And they carefully omitted the next sentence: "The manufacturer of the avionics board corrected this glitch in later software revisions. But alas, LightSail’s software version doesn’t include the update."
That still doesn't excuse a problem that would have been found by bench-testing the thing for a few days before sending it up. Nor does it excuse constantly appending one file to store data in an unattended system. Also, anything that JPL sends up has a backup channel that can push that little red button on the main computer. All they can do now is hope for cosmic rays to reboot it randomly. At least it's in LEO and not zipping off into interplanetary space.
In the meantime, the team is looking at several fixes to work around the software vulnerability once contact is reestablished. One is a Linux file redirect that would send the contents of the troublesome beacon.csv file to a null location, a sort-of software black hole. Lab testing on this fix has been promising—over a gigabyte of beacon packets have already been sent into nothingness without a system freeze.
Well, isn't that special. Now they test it. So if they can just link it to /dev/null, did they really even need that data? It's always fun to cause a mission to fail by recording data that wasn't even needed.
Re: (Score:2)
I think the "32 megabytes—roughly the size of ten compressed music files" part is even more insulting.
True that. It should have been in Libraries of Congress.
Re: (Score:2)
I think the "32 megabytes—roughly the size of ten compressed music files" part is even more insulting.
Especially when the -- well known -- standard unit of measure is "Libraries of Congresses".
Re: (Score:2)
It's just a copy/paste from TFA which is apparently targeted at someone with no technical knowledge what so ever.
Re: (Score:2)
what's a music file...doesn't everything just stream in as needed?
Re: CSV (Score:2)
I assume you mean the part where they think any of us listen to MP3s of music at such a shitty bitrate? ;)
Re: (Score:2)
For those of you who are too young to have heard of it, inna gadda da vida at 256kbps is 32,727,315 bytes.
Pink floyd's meddle at 320bps is 56,465,408 bytes
Disk Jockies would spin one of these when they wanted to have a 17 to 22 minute break.
Re: (Score:2, Insightful)
Re: (Score:2, Interesting)
Actually this particular failure wasn't as obvious of an oversight as you may think. The reason it happened was because in an existing system one particular set of parameters were logged in miles since they weren't responsible for flight control (which NASA mostly uses metric for). Later on portions of this design were reused and an engineer
Better example (Score:3)
How embarrasing (Score:4, Insightful)
You'd think that something as small as 32MB would have been tested before they launched the thing... It doesn't sound like it takes very long to fill up 32MB either
Re: (Score:2)
Sounds like it's pretty much a transponder in an airplane.
I wonder why they are even logging this information?
Re: (Score:2)
It doesn't sound like it takes very long to fill up 32MB either
Nope. Just ten compressed audio files would do it.
Re: (Score:2)
How many long versions of "Freebird" is it?
Re: (Score:2)
21,000 pages of plain text (assuming "average" page of 80col, 20 row), going by my head math.
Re: (Score:2)
Re: (Score:2)
UAT (Score:2)
I'll never understand how groups (Especially NASA) can spend millions, or even BILLIONS on projects like these and not even complete the sorts of rudimentary testing that those of us in the professional software fields have to do every day. Ok, this computers going into space and going to run for days/months/years... whatever... so hey, maybe we should boot it up while it's still on the ground and see if it'll run for a couple of months without crashing first?
One of the mars rover had the same problem. It w
Re:UAT (Score:5, Insightful)
Well, how do you test it before you're happy ? If the beacon is 40 bytes, and transmitted every 15 seconds, it would take half a year before you fill up 32 MB. That's a long time for testing.
This is the kind of mistake you shouldn't even make in the first place.
Re: (Score:2)
One way to test that is to simulate time. A simulation wouldn't need to wait 15 actual seconds, it could speed up time such that transmissions run immediately after the last, until the test has surpassed the expected lifetime of the mission.
If this were able to be done once every millisecond instead of once every 15 seconds, they would have run across the bug within 14 minutes.
Re: (Score:3)
You write your test so that it sends the 40 bytes to the csv file every 10 milliseconds instead of every 15 seconds.
The moment that you think of doing that, is the moment that you realize that the file will grow too big.
Re: (Score:2)
The problem is they hire programmers from EA.
it compiled! OMG! Launch it!
Re: (Score:3, Informative)
First off.. LightSail isn't a NASA mission.. it's a low budget cubesat and cubesats tend to trade risk for rigor.
NASA does run stuff for days/weeks/etc in testing. And you'll note that the Mars rover flash file system thing was able to be recovered from, thanks to smart people at JPL realizing that you always need a way to recover. This is not necessarily the case for cubesats, often built by enthusiastic grad students whose hair is not yet grey from living through near and actual disasters in flight proje
Re:UAT (Score:5, Informative)
Speaking as an engineer working on software that is on the Orion spacecraft, I can say that rigorous testing is budgeted into the project from the beginning because it helps to avoid most of the problems like this. The testing that goes on with flight software is orders of magnitude more than you find for a traditional commercial product. You have to. The consequences of failure are, obviously, a lot more significant.
That being said, it's impossible to catch every single possible bug, especially as systems get more and more complex. But there are strategies that help reduce your risk. For example, you don't just run off to kernel.org and throw the latest stable release on a board. You pick operating systems that are maybe a bit harder to use (i.e. limited in what they can do) but are far better suited to real-time embedded work. And you certainly don't blindly append to a file without verifying that you're not going to overflow your space. And you always have an automated recovery plan for any dynamically allocated space in the event of an overflow.
This kind of failure is caused by amateurs making amateur mistakes. It was caused by application programmers who don't understand the consequence of failure in a constrained environment where you can't just click a mouse to restart the program. It was caused by poor planning and a lack of understanding of the environment in which they were designing. This was caused by hiring coders instead of experienced engineers. It was caused by trying to do it cheap rather than spending the money to do it right. They got what they paid for.
Re: (Score:3)
I'll never understand how groups (Especially NASA) can spend millions, or even BILLIONS on projects like these and not even complete the sorts of rudimentary testing that those of us in the professional software fields have to do every day.
This is not a NASA project, so you've made a stunningly basic error in your first sentence. Not looking too good for attention to detail for someone "in the professional software field".
Regardless, if you want to see how NASA does software, or for anyone even remotely interested in how the best practices for true mission-critical software gets written, you can't find a more interesting story on the creation of space shuttle software [fastcompany.com]:
tachyon eddy (Score:2)
Re: (Score:2)
What the computer needs is ... (Score:2)
... the ability (small code here) to power cycle and come backup in maintenance mode where it doesn't do anything on its own except receive diagnostic commands.
The computer also needs a sibling for fail-over.
There may be reasons those were left out that I would agree with.
I sure hope they can get this puppy lined out.
Re:What the computer needs is ... (Score:5, Insightful)
No. They need programmers and sysadmins that knew that they were doing. E.g. roll log files and/or put logs on a non-critical partition. Systems Administration 101 for systems where memory and disk space are at a premium. It was a rookie mistake.
Accident code. (Score:2)
cat beacon >> beacon.csv
instead of....
cat beacon > beacon.csv
oops.
Maybe (Score:2)
SpaceX can retrieve it long enough to hit the reboot button...
At least the post used "hope" as a noun (Score:3)
Mirab, his light-sails unfurled! (Score:4, Funny)
Shaka, when the walls fell
For those who don't know what Linux is... (Score:3)
Coming up next on Slashdot... Linux is an operating system, kinda like Windows or Mac OS, but built by a bunch of neckbeards, and uses about the same amount of space as 10 compressed music files. Some versions use less, some use more depending upon how it's configured.
Wow; I think it's time to move on from Slashdot. Taco would be spinning in his grave, assuming he was dead.
Oh great (Score:2)
A satellite running Linux is contingent upon a spontaneous reboot to function again? Great, now we'll never hear from that satellite again.
Clearly, the plan should have been to run the device on Windows 98. That way, it would only be out of commission for 49.7 days.
The most interesting spacecraft. (Score:2)
We don't normally test our spacecraft systems, but when we do, we do it after launch.
Updated Summary W/ Tech Terms Explained (Score:5, Funny)
Re:Should have used apps! (Score:4, Funny)
Mebibyte is an idiotic term (Score:2, Insightful)
and you are an idiot for using it.
Re:Mebibyte is an idiotic term (Score:4, Insightful)
Just because you don't like the term doesn't make it wrong. Highjacking SI prefixes [wikipedia.org] and changing their meaning is wrong and has led to countless problems.
And historical meanings shouldn't be changed simply so that marketing speak can be used to sell less at the same price.
I love how 1 MB of RAM is 1048576 bytes but 1 MB of storage is now 1000000 bytes of storage, simply because the hard-drive industry decided that they could make more money by using the same term, change the historical meaning in the computing industry from base-2 to base-10 (essentially downsizing the actual storage), and charging the same amount.
Either convert totally to GiB, MiB, etc. for everything computer related or stick with the old convention. It's when you are mixing the two in a particular context (i.e. computers) where you run into problems.
Re: (Score:2)
I love how 1 MB of RAM is 1048576 bytes but 1 MB of storage is now 1000000 bytes of storage
Makes perfect sense. RAM is addressed with a N address lines, giving access to 2^N cells, so base 2 makes sense. For everything else, base 10 makes more sense, especially when you're talking about speeds.
simply because the hard-drive industry decided that they could make more money by using the same term
They didn't make more money because everybody was doing this.
Re:Mebibyte is an idiotic term (Score:5, Informative)
I wish people would stop thinking that hard drive manufacturers are the "source" of this so-called "problem". Digital communication speeds never used base 2, clock speeds didn't either.
People are simply stuck on terms like "Mebibyte" because they either don't want to accept the fact that mega is an SI prefix or because they don't like how the IEC units sound. Get over it [wikipedia.org].
Re: (Score:3)
No they mean MB because they even though they've crowdfunded a tiny satellite launch they are still not as autistic as you.
Re: (Score:2)
Ok, so they have a problem when going over 32 000 000 bytes.
Re: (Score:2)
If it makes you feel better, sure think that the rest of the world is as autistic as you.
32MiB (Score:2)
They couldn't afford to pay Will Smith for that many sequels
AllWorkAndNoPlayMakesJackADullBoy.csv (Score:2)
How much is that in library of congress?
Please, I'm no nerd, I don't know this "technology" stuff.
6 Shakespeares [gutenberg.org]... or
16.5 gzip-Shakespeares... or a whopping
22.6 bzip2-Shakespeares.
The Bard fares well by the Burrows-Wheeler algorithm for his works are so oft-repeated he even runs on and repeats himself. "...So all my best is dressing old words new, Spending again what is already spent" as RLE (run length encoding) and "To smother up the English in our throngs, If any order might be thought upon..." as MTF (Move to Front) Transform. "We render you the tenth; to be ta'en forth! Before the common distribu
Re: (Score:2)
Re: (Score:2)
Re:Whew! I nearly funded that one... (Score:5, Funny)
Well, Jason. What have you got to say?
Well, Mr Nye...
Doctor! It's Doctor Nye.
But I thought those were honourary degrees.
It is DOCTOR Nye. Say it! SAY IT!
Y..Yes. D..D..Doctor Nye.
So, what happened to our bird, Jason?
As you know, um... Doctor Nye... We used a kickstarter campaign to fund the satellite's development and testing.
Get to the point, Jason.
We ran out of funds. If we had one more donor, we would have been able to complete the final testing.
So we lost the satellite and now face public humiliation because one anonymous person was too cowardly to donate?
Yes. Um.. Doctor Nye. That's about the size of it.
Well, Jason. That fellow had best pray that he and I never cross paths. You may go.
Re: (Score:2)
. . . have you tried turning it off and on again? :-)
Ok. but I can't reach the power switch....
Re: (Score:2)
It's called "logrotate" and shame on you if you stared the logger and didn't configure it first.
Re: (Score:2)
Bill Nye was a TV guy who got sucked into a science gig where he became famous. He apparently is under the delusion this makes him a top-notch scientist. I hope his role in the project was Fundraiser and not Chief Engineer.