Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Linux Software Science

LinuxBIOS, BProc-Based Supercomputer For LANL 189

An anonymous reader writes "LANL will be receiving a 1024 node (2048 processor) LinuxBIOS/BProc based supercomputer late this year. The story is at this location. This system is unique in Linux cluster terms due to no disks on compute nodes, using LinuxBIOS and Beoboot to accomplish booting, and BProc for job startup and management. It is officially known as the Science Appliance, but is affectionately known as Pink to the team that is building much of it."
This discussion has been archived. No new comments can be posted.

LinuxBIOS, BProc-Based Supercomputer For LANL

Comments Filter:
  • by Anonymous Coward
    Imagine a Beowulf cluster of these!
    • Beowulf? [slashdot.org] by czardonic (Score:4, Funny) Monday October 08, @12:07AM
    • Re:Obligatory Beowulf Post [slashdot.org] by Anonymous Coward (Score:-1) Monday October 08, @12:08AM
    • Greatest joke ever! [slashdot.org] by conner_bw (Score:1) Monday October 08, @12:10AM
  • Anybody know if this is a reference to Pink Floyd, if so then I appreciate this team all the more :)
    • Re:Floyd (Score:1, Interesting)

      by Anonymous Coward
      I think a more fitting musical allusion would be 'Music from Big Pink', by The Band.
  • Wait... (Score:1, Redundant)

    by hkhanna ( 559514 )
    Pardon my ignorance, but would this be considered a Beowulf cluster? I mean everyone on /. talks about them so is this it, finally? A real live, Beowulf cluster? If so, imagine a beowulf cluster of beowulf clusters.

    Or, a beowulf cluster of beowulf clusters of beowulf clusters. The possibilities are infinite (literally.)
  • by Anonymous Coward
    Which one is pink?
  • Just like the Iraqis and the Chinese, I do all my nuclear weapons testing on my Playstation 2 Supercomputer! [beowulf.org]
  • Uses (Score:1, Interesting)

    by esac17 ( 201752 )
    Let's just hope they do something good with this. I'm tired of reading about how supercomputers are used for military war simulations.

    Does anybody know other applications that supercomputers are being used for. I know some do weather predictions.
    • Re:Uses (Score:2, Informative)

      by saveth ( 416302 )
      Let's just hope they do something good with this. I'm tired of reading about how supercomputers are used for military war simulations.

      LANL tends to do projects that are focused much more on science and engineering than military applications. It's very likely that Pink will end up analysing spectral emissions of bombarded protons or something like this.

      The military simulations you mention probably don't happen at LANL.

      • That opinion is, shall we say, not very informed.

        You do realize that Los Alamos is the child of the Manhattan Project, don't you? The former home of Wen Ho Lee? Ringing any bells yet?

      • You obviously don't know much about LANL.
    • Medical (Was:Uses) (Score:3, Interesting)

      by srw ( 38421 )
      A former client who worked at a Cancer Center used a cluster to simulate radiation treatments.

      • It is called Monte Carlo simulation of the radiation transport. Basically one tracks propagation of the high-energy particles as they progress trough the matter (human tissue). e.g. if you simulate a brachytherapy source (radioactive "seed" implanted in tissue), the code "creates" photon with energy characteristic for a given isotope. The direction vector is chosen by a random number generator (RNG). The RNG also decides at what point along the photon's trajectory an interaction with surrounding matter should occur according to the physical probabilities.
        After the interaction, there is a bunch of scattered particles (photons, electrons and positrons) and the code continues to track them until the energy of the n-th generation particle drops below certain energy (10 keV for electrons) at which moment particle's energy is deposited as a dose.
        The object of the simulation is to obtain precise dose distribution. In order to achieve good statistics one needs to run millions and millions of histories.
        Beowulf clusters are ideal for this job because histories are independent and there is no need
        for the fast shared memory and a fancy interprocess communication.
        I had a pleasure to assemble 24-node 1.6 GHz AMD cluster and we achieved sub-minute simulation times, a result that makes this technique suitable for everyday clinical practice.
    • On the contrary, I wouldn't mind seeing more military war simulations being done on supercomputers; so long as they are carried out as an alternative to actual military war.

      Think about it: Instead of wasting all the money, resources, and lives of actually invading another country, we just get a few supercomputers into a network, and duke it out online.

      First thing, of course, would be to allow the export of supercomputers to blacklisted countries. (Is Afghanistan still on the list, I wonder?) Then get a UN resolution that all member countries will abide by the outcome of any virtual war.

      And hey, the US has already got a head-start in training soldiers for it: "America's Army"!
      • Hell no. If that happened, Korea would take over the world. Starcraft has taught us that we should never mess with the Koreans when it comes to virtual war!
      • On the contrary, I wouldn't mind seeing more military war simulations being done on supercomputers; so long as they are carried out as an alternative to actual military war.

        Wasn't that a Star Trek episode? I don't think it turned out very well...

      • So you're saying.. like, GW 'n Sadam should play Return to Castle Wolfenstein?
      • Come on, like there was anyone on Earth who thought that Afghanistan was going to destroy the US... or that Iraq has an ace in the hole that's going to bring us down... or that anyone, anywhere, has enough strictly military might to mess with the West. You've got to be kidding.

        The only place where a military conflict is actually a contest is in between the second and third world, now that many third world nations have nuclear weapons. The second world is not of a mindset that any type of computer simulation would be an acceptable substitute, and even simulating the morale of billions of starving Communists is a staggering proposition.

        So we're right back to the third world countries. Ignoring the export ban, do you really think India and Pakistan (who are hardly third world countries anymore) are the types of people who will say, "Look, the computer says we'll win, so why don't you surrender now?"

        And I am using the accurate definitions of first, second, and third world. No need for anyone to get offended by whatever false implications they attach to terms like "third world".

    • Re:Uses (Score:5, Interesting)

      by marm ( 144733 ) on Tuesday October 08, 2002 @01:10AM (#4408214)

      Does anybody know other applications that supercomputers are being used for. I know some do weather predictions.

      Ok, non-military uses, off the top of my head:

      • mathematical research - simply complicated maths on big numbers
      • fluid dynamics modelling - traffic flows, or aerodynamics, or hydrodynamics - this is also tied in quite closely with weather/climate prediction
      • statistical modelling - wouldn't you like to know if the stock market is going to go up or down tomorrow, before it happens?
      • computational chemistry/biochemistry - protein folding is just the tip of the iceberg - imagine being able to design a molecule and then simulate the effect it will have on the human body, without that substance ever having been actually synthesized or going near a human... this is the future of drug development
      • quantum mechanical simulation - related to computational chemistry, imagine taking all those complicated quantum mechanics equations to their logical conclusions, predicting as-yet undiscovered subatomic particles and their behaviour, or to design better magnetic containment fields so that practical fusion energy generation is possible
      • good old-fashioned databases and signal processing - when you have hundreds of terabytes of data that you wish to mine for interesting patterns, speed matters

      I'm sure there are plenty more applications for supercomputer power - any kind of complicated or chaotic system is a good candidate for modelling, especially when there's more than one unknown variable (multivariate analysis is complicated, to say the least).

      • Re:Uses (Score:2, Insightful)

        by abulafia ( 7826 )
        statistical modelling - wouldn't you like to know if the stock market is going to go up or down tomorrow, before it happens?

        I agree with you on most respects (even if much of what you're talking about is very, very far beyond most realistically imaginable systems in the near future), but simple economics shows why the above is silly.

        Simple question: someone uses a tool to make a killing on a pre-existing market. How does everyone respond (not counting RIAA, et al, who depend on regulation)? They either curl up and die, or figure out what the winners are doing, and quickly. Learning what people are doing is even easier in markets like finance, where there's a lot of transparency in actions, a very close knit group of participants, people who like to brag, and a lot of people staring at the winners.

        Fact is, any new innovation in trading quickly becomes used by everyone who has a serious enough stake. It is just market economics. Once everyone gets an innovation, it is no longer an advantage, because everyone is doing it (bonus points for those who see past and potential systemic failures lurking in this behaviour).

        Of course, keeping your traders free of risks like sharing information and regulatory oversight can extend an advantage, and that works in a very few situations. But hell, even Warren Buffett took a fairly serious beeting recently due to things he couldn't predict (and this is an insurance guy!), not to mention Soros when he attacked Asian currencies a few years ago.

        Not only is there no silver bullet for the folks who run finance, there's just no way in hell peons in the game (anyone with less than a few hundred million invested) will profit from raw computational power. Sorry.


        • by Isle ( 95215 )
          Actually this was widely used before the tech-bubble. The idea is that a computer can generate a few parts of a percentage better predictions than most humans. Since the percentage is so low, you let the computer invest huge amounts of money.

          Ofcause when the entire market crashes, these machines loose money a lot faster than humans, since no one has tought them to pull out, and because they have to so much money invested.

          So while much of the research in this area died suddeny sometime in early 2000, it still proves your theory wrong.
          • Most of your statements prove my point exactly correct, if we are to belive you..

            Yes, various practices that fall under the moniker of 'technical trading' have been around a long time. By some counts, since right after the 1930s. By others, before then. Software assisted trading is in some ways new, but in the past the same result happened, aleit slower, through agents.

            To give you a point...

            Sure, ill tuned risk management systems fuck up. Plus, they're extremely important to the world economy. That's why Greenspan bailed out a certain well known hedge fund very recently.

            I was not asserting that "much of the research in this area died suddenly". On the contrary, research in risk management is hitting a rather furious pace. Please re-read what I wrote, and this time pay attention.

            The volume of trades taking place without human interention causes huge swings at the moment. We're seeing this now, and have been for a couple of years.

            Bonus points if you come up with a theory why seeking short-term gains are going to cause exactly the "double dip" so many cheerleaders at Bussinessweek and Fortune are trying to recant.(

            If you get bored, you could actually respond to what I was asserting, which was that a commercially viable trading system would rapidly stabilize any advantage it had, because it would spread to everyone who had a serious interest in tracking new developments in this area. That's what's still oddly on topic for the parent post.
            Thank you, drive though.


        • simple economics shows why the above is silly.

          Silly it might be - that doesn't stop people from attempting it though, and buying and using supercomputers to do it with.

          Fact is, any new innovation in trading quickly becomes used by everyone who has a serious enough stake.

          Sure. But every time you improve the model, in theory at least you get a short period of having an advantage over everyone else - until they improve their statistical model to match or beat yours. Even if your advantage only lasts a day, and even if it's only a minor advantage, that's easily long enough to make up the cost of the supercomputer and the programming time, and then some, at least if you're a major investor.

          No-one said this was something you or I would benefit directly from - at least, not unless you have a stake in the investors doing the market analyses - but to the major market investors, if it gives them an advantage, even a temporary one, good luck to them.

        • New trading innovations are not necessarily adopted widely by everyone quickly. Technological innovations, sure, but not all innovations are technology based.

          Example: Benoit Mandelbrot has been working on using the concept of fractals to predict market changes. His intellect is unique and not likely to be duplicated easily by others. Unless of course he decides to publish his results.
      • "good old-fashioned databases and signal processing - when you have hundreds of terabytes of data that you wish to mine for interesting patterns, speed matters"

        I agree with most of your points, but as someone with only a few dozen terabytes to worry about, i can say, speed does matter. DISK speed. MEMORY access speed. With that much data, Cpu speed beyond "decent" is really only useful as something you can trade for less disk access via compression, and even that hits a wall pretty fast. Supercomputers like this one are great for big calculation; not so much for big data.
        • Supercomputers like this one are great for big calculation; not so much for big data.

          If all you're doing is storing data and then retrieving subsets of it on demand, then sure, I agree. Most databases are like that - but not all. Some databases do more complicated processing than search, sort, split and join, some have to do some heavy manipulation of every record, and in these cases, CPU speed is just as important as memory or disk speed, if not more so.

          Case in point: SETI. Terabytes of raw data collected from radio telescopes, no doubt stored in a large database in Berkeley, divided up into manageable records indexed by time and position in the sky. The data is absolutely worthless though, without colossal amounts of processing. So they build their own network of donated CPU time - a global data processing system - in order to turn that worthless data into something worthwhile. It isn't the disks or memory that limits speed of data processing, it's the CPU.

          Of coure, if they'd had the money they could have simply bought a single machine to do all the data storage and processing, but it would still have been CPU-bound rather than disk or memory-bound. There are plenty of more conventional systems where everything is done on one physical machine, but still have the same problem of being bound by CPU.

          I guess it depends on how you define a database - is a system that does complex processing of data as well as storage and retrieval still a database, or is it something else?

      • "any kind of complicated or chaotic system is a good candidate for modelling"

        Actually, perhaps the chief feature of chaotic systems is that they are NOT good candidates for modeling.
      • To add to that, seismic interpretation by Oil companies. Shell have a 1024 node AMD cluster in the Netherlands for this purpose.
    • Re:Uses (Score:3, Interesting)

      by afidel ( 530433 )
      The largest (largest by a long shot it outpowers the rest of the top10 combined) supercomputer in the world is the NEC Earth Simulator in Japan. It is being used to do the most detailed climate modeling ever attempted. Not only that but they are attempting a complete system model which AFAIK has never before been possible. In addition the last couple clusters that I have read about have been for biomedical research, maybe it's just what I read but I believe bioinformatics is going to be one of the biggest pushers of HPC going forward. Genomics is nothing compared to proteonics, mapping the genome probably takes about as much computing power as simulating the folding of one large protein series!
    • Re:Uses (Score:2, Funny)

      by DasBub ( 139460 )
      Well, Hollywood has used supercomputers and large clusters to do effects for movies like Star Wars: Episode II, Resident Evil, and the upcoming Terminator 4.

      So, no, there haven't been any good uses.
      • Er, 3. Terminator 3...

        But we all know there'll be a T4.

        SkyNet probably sends Vin Diesel back in time to beat a 27-year-old John Connor at a game of pick-up-sticks, thereby destroying his ego.
      • Re:Uses (Score:3, Informative)

        by foobar104 ( 206452 )
        Wrong. Render farms are neither clusters nor supercomputers. At best, a render farm might be considered an array.

        A supercomputer is a single system image. Some people call large clusters "supercomputers," but technically they're wrong.

        A cluster is an interconnected group of computers that can communicate with each other. Usually a cluster depends on some kind of software layer to allow programs to run across multiple systems, something like MPI. Clusters are tightly interconnected many-to-many systems.

        An array has a single job control system and a number of job execution systems. Batch jobs are submitted by users to the job control system, which doles them out to the various execution systems and then collects the results. The execution nodes don't talk to each other, and one job runs on one execution node at a time. Render farms are basically arrays; each execution node works on rendering a single frame of a multiframe animation. Because each frame can be rendered independently, without any dependencies on the previous and subsequent frames, rendering is particularly well suited to array computing.
        • Don't be so sure (Score:3, Insightful)

          by marm ( 144733 )

          A supercomputer is a single system image. Some people call large clusters "supercomputers," but technically they're wrong.

          Says who?

          Once upon a time 'supercomputer' meant 'any computer made by Seymour Cray', and this was reasonable, because he (probably) invented the concept. Then there was the mid-80's loose but widely-accepted definition 'any computing system that can do more than 200 MIPS'. Then MIPS went out of fashion and processors got faster and it was 'anything that does more than a GigaFlop'. Or there's the US Department of Commerce definition which was 'any computing system that does more than 195 Mtops (Million theoretical operations per second)' during the 80's, which then got changed to 1500 Mtops and is probably something different now.

          Note that most Linux cluster systems would meet the requirements of most of these - indeed, most single-CPU computers today would meet most of these requirements, which is how Apple manages to get away with calling the G4 a 'supercomputer'.

          Really, these days 'supercomputer' means absolutely anything you want it to be, although if I had to define it, I think probably the fairest definition would be 'anything that can run the LINPACK benchmark suite and get on the Top500 list' [top500.org].

          Nice try at creative redefinition though.

          • Really, these days 'supercomputer' means absolutely anything you want it to be, although if I had to define it, I think probably the fairest definition would be 'anything that can run the LINPACK benchmark suite and get on the Top500 list'.

            Personally, I'd refine that to "get onto the first page of the Top500 list"; but with either definition, the point remains that while some clusters can run the linpack benchmark, most render farms can't, and SETI@Home or distributed.net *certainly* can't.
          • Re:Don't be so sure (Score:3, Informative)

            by foobar104 ( 206452 )
            The important thing to notice about the word "supercomputer" is that it's singular. A supercomputer is a single system image; this is implicit in the definition. This is not to say that supercomputing clusters aren't worthy; it's just that they're different in important ways from single-system-image supercomputers.

            Some classes of problems aren't suited for cluster computation. I won't pretend to be educated enough to tell you exactly which problems can and can't be adapted for cluster computation, but consider the nature of clusters to see my point. Clusters are highly scalable, but the inter-node latency is huge. An interconnect like Myrinet can get your remote messaging latencies down to the microsecond range, but the far more common MPI/PVM-over-Ethernet solution is a thousand times slower than that. This makes it somewhat inefficient for node N to try to access a bank of memory on node M. In order for a cluster to be efficient, each node should have sufficient physical memory to hold it's entire data set, and each node should be able to operate more-or-less autonomously, without having to contact other nodes.

            Supercomputers are fundamentally different from clusters. In some cases, you can do the same job with either a supercomputer or a cluster. Some jobs are better suited to clusters, while some are better suited to supercomputers. Some jobs, as I mentioned above, are better suited to arrays than to either clusters or supercomputers. It just depends on the job.
            • The important thing to notice about the word "supercomputer" is that it's singular.

              Ok. The word 'cluster' is also singular. Big deal.

              A supercomputer is a single system image; this is implicit in the definition.

              Like I said, whose definition?

              I already know that some types of supercomputer work better for certain types of problems than others, all I'm doing is nitpicking at your pulled-from-the-air definition of a supercomputer that somehow magically defines that a cluster of machines cannot also be a supercomputer. Sure, they may only be individual machines connected via a network, but you use such a cluster as a single large, powerful supercomputer, even if it's not a single system image. What about MOSIX clusters? Whilst strictly speaking they are not single system image, they behave like they are - indeed, from a programming point of view they're all but indistinguishable from a single image NUMA machine, and using switched Gigabit Ethernet (cheap these days) the internode latency and bandwidth isn't too horrid either.

              Your overstrict definition of 'supercomputer' makes you sound like you work for one of the old-guard manufacturers like SGI or Cray threatened by the rise of cluster supercomputers.

              My argument is that clusters, arrays, NUMA machines, SMP machines, with both conventional and vector processors - these can all be 'supercomputers', because although the hardware design and programming techniques are quite different for each type, the end result for all of them is a single system (not necessarily a single system image) that solves numerical problems very quickly. They are just different types of supercomputer. If I could convince all the world's population to do maths with an abacus all at once, and I could somehow divide up the work sensibly and then collate the answers, then yes, that would be a supercomputer too - a human supercomputer, although not a particularly fast one.

              • Ok. The word 'cluster' is also singular. Big deal.

                The word "cluster" is collective. Like "dozen." It refers to a group of things as a unit. It's not possible to have a cluster of one.

                Clusters of computers are not supercomputers. They're different. You can try all you like to say that the definition of "supercomputer" is arbitrary; I suppose it is, in the sense that all definitions are fundamentally arbitrary. A supercomputer is a single computer. A cluster of computers is not a single computer. Ergo, a cluster is not the same thing as a supercomputer.

                If I could convince all the world's population to do maths with an abacus... that would be a supercomputer too.

                No, it wouldn't. A computer-- and, by extension, a supercomputer-- is a mechanical device. A group of people could perform the same job as a supercomputer, but the group wouldn't be a supercomputer.

                I think you're possibly confused by metaphors. When you say, "A group of people is a supercomputer," you're employing a metaphor to say, "A group of people shares many important characteristics with a supercomputer." Don't be fooled by this. It's not literally true, and shouldn't be accepted as such.
      • There's at least one movie that I can use to thumb my nose at you. The Last Starfighter. It used a Cray X-MP to do 36000 frames.

        In any case, I was trying to be a funny little shit with regards to the crap that they've produced. I wasn't trying to be technically correct. But thanks for the education.
    • I think someone else has mentioned it but one of the largest govt uses outside weapons simulation is weather. You just can't do medium to long range weather without a lot of computer power,

      Oil companies like to have serious computer power too for prospecting and resevoir modelling.

      In organic chemistry, you can do some serious molecular simulations ranging from pharmaceuticals through to the actions of enzymes and catalysts.

      The fluidics side can even extend through to air-flow modeling (aircraft to cars) and combustion.

  • as it is an OS by Apple and IBM (well gone but still)
    • >as it is an OS by Apple and IBM (well gone but

      AFAIK, "Pink" was just the internal code name for the eventual OS planned for the PowerPC.

      And we all know how THAT turned out.


    • Los alamos is run by the gub'ment and they dodn't have to abide by patents or copyrights. The millitary, for example, reverse engineers software all the time to make it more secure. As to your second question its really coarse it tears easily and it sucks if you make bed sheets out of it...oops. Sorry, thought you said muslin
    • Yes, because people are _bound_ to confuse a 2048 processor cluster with an operating system when LANL start marketing them.......duh.
    • And what about the pop star who needs a belt for her pants (but I hope she doesn't get one)?
    • as it is an OS by Apple and IBM (well gone but still)

      Maybe they mean this Pink [pinkspage.com]? I know which one I prefer ;-)
  • Diskless X clients have been attractive due to the lack of remote configurations and disk/data failures.

    Clusters suck up a lot of electricity because of the hardware they support.

    I'm sure the next step involves skipping the extra motherboard components (IDE, USB, AGP, etc.) and making the CPU/Memory mount to a TCP/IP Switch backplane. Better yet, drive the thing with low power CPU's so it won't sound like a helicopter prior to take off/reqire a new nuclear plant to power it up/create a new market for Frigidair.
  • LinuxBIOS (Score:5, Insightful)

    by Anonymous Coward on Tuesday October 08, 2002 @12:35AM (#4408114)
    I wonder why LinuxBIOS hasn't taken off. I've debated ordering one of their "kits." It seems to me the 3 second boot time of LinuxBIOS should be a selling point for some obscure Linux vendor, but no one really offers it yet.

    I really imagine a machine with an 8MB EEPROM/ROM that can be updated as needed, but provides a boot environment and login screen - while spinning the disks in the background. This would make an excellent product.

    Why hasn't anyone done this yet?

    • Re:LinuxBIOS (Score:2, Interesting)

      by brsmith4 ( 567390 )
      Probably the same reason we aren't on IPv6 yet: not enough need to insite change. I agree with you though, I would love to have 2-3 second boot times.
    • The problem is that motherboard vendors still tend to only use 2M-bit flash devices and only route the address lines from the chipset to the flash for 2M bit flash BIOS devices.

      An 8M-bit flash BIOS kit would need to have its memory footprint paged into the 2M-bit memory space that is routed on the motherboards or include schematics, solder, wire and soldering iron for the installer to tie the needed memory address lines back to the chipset.

  • by Chirs ( 87576 ) on Tuesday October 08, 2002 @12:46AM (#4408147)

    This sounds like some kind of dual-processor rackmount type solution. Why not go all the way and use something like compactPCI? You can fit 21 cPCI blades into 8U of rackspace.

    A standard blade could have up to a couple gigs of ram, a powerpc or p3/p4 cpu, 100BT or 1000BT ethernet, etc, etc.

    You boot the things using bootp/tftpboot and then run linux off a ramdisk.

    We're using cPCI at work to run VoIP softwitches. Currently we're at over a million calls an hour on a wimpy 450MHz processor.
    • At 8U you get to use standard (cheap!) ATX Xeon motherboards and put them in your cases. Xeon motherboards give you very fast CPUs with lots of memory (8GB-12GB) and usually one or two GigE built-in. This is what the high-end-computing customers want - concentrated computing power. VOIP is much less demanding. If you want to use Xeon with CompactPCI, you currently need to make your own motherboard ($$$$ in initial engineering costs) and figure out how to cool it (small fans don't work well).

      Does anyone know any good CompactPCI Xeon manufacturers? Doubt it.
      • Unless I missed the announcement, linux is still stuck with 2GB RAM max, at least on x86.

        Also, if you can fit enough P4s in blades, you can beat the Xeon's processing power for less cost and floorspace; this is, after all, why x86 clusters are beating the big iron servers from Sun & IBM on price/performance. Of course, you have to have a load which can be distributed in this way; if you have fewer, heavy duty compute requirements, Xeons may be the best fit for the job.

    • Because that's more specialized and not as mass produced, it's going to end up costing a bit more. I, personally, have never played with cPCI, and I've played with some esoteric, technical stuff. I'm not sure that they'd have the experience necessary with that. They might need to hire someone or train someone. Once you start getting into the embedded world, you need need training than the average guy on Slashdot has.

      cPCI with PowerPC processors would be just too damn cool. I've looked at them at Motorola's web site. I just wish I could find an application for them!
    • Good question.

      Why not cPCI? In a word, performance/price on our apps. We looked at all sorts of cPCI blades (e.g. http://www.cluster-labs.de) but the peformance just is not there. Also, no existing ethernet will do the job for our apps, so we have to use Myrinet, and again, the fastest Myrinet is going to be in the PCI 64/66 slots on plain old motherboards.

      Other folks have asked about 1.0 Ghz G4. I like PPC. But on our applications the PPC, with the best compilers we can find, is actually slower in absolute terms than a PIII/800. So as much as I would have wished to use a PPC, it's not cost-effective.

      Note that our software runs fine on G3 and G4 macs however -- our standard CD distribution from http://www.clustermatic.org will boot either PC, PPC, or Alpha just fine. In fact, the standard Linux distribution from http://www.terrasoftsolutions.com/products/blackla b/components.shtml features some of our software, including bproc.

      Also, if you look at the PPC offerings from synergy and CSPI you'll find they run their own kind of "Linux in flash" -- not LinuxBIOS, but pretty much the same function. They've been doing this for years.

  • by Anonymous Coward on Tuesday October 08, 2002 @12:47AM (#4408151)
    I don't envy the developers... After every revision of LinuxBIOS, they get to reflash 1024 motherboards, which could take a while...
    • by Anonymous Coward
      Not really, a new revision can be flashed with a single utility that can be run on all the nodes in parallel.
  • And I thought the new 48 node cluster at work will soon be able put out some flops... outclassed, outgunned and outperformed. I used to get excited about hearing of other's beowulfs. Now I am only jealous. :)

    BTW, if you see a post that says 'Damn...' and nothing else, thats cuz this damn keyboard has this enter key that gets in the way.
  • ....RMS rants about pink and demands everyone call it GNUpink.

  • by goombah99 ( 560566 ) on Tuesday October 08, 2002 @01:27AM (#4408247)
    I've been a beta tester on the prototype for this system. It works great. I've seen diskless systems before they all were NFS nighmares, could not scale and had horrible tendencies to cause rippling crashes as one computer after the next timed out on some critical disk based kernel operation it could not complete across a wedged network.

    This one, brpoc, is different it is completely stable. You never get NFS wedges. Jobs launch in flash. Plus if you do reboot the whole thing is back up in seconds (literally).

    Bproc is an incredibly light weight job submission system. It is so light weight and fast that it changes how you think about sumbitting jobs. Rather than designing long duration jobs and tossing them on queue, you can just run tiny short jobs if you want with no loss to overhead. It makes you re-think the whole idea of batch processing.

    when the jobs run they appear in the process list of the master node. That is if you run "top" or "ps" the jobs are listed right there. In fact from the users point of view the whole system looks like just one big computer.

  • by FyRE666 ( 263011 ) on Tuesday October 08, 2002 @01:33AM (#4408258) Homepage
    I will personally track down and slaughter the first person to mention a popular clustering architecture, and how one might imagine it...
  • It will be interesting to see if anyone builds a massive HyperTransport switch (probably a local switch for a blade frame with 1000bt between blade frames). The opteron looks like it could run without much in the way of chipset support (build in memory controller), and skipping all the unnecessary I/O would be pretty simple.
    Of course, dumping all the heat would be an issue...
    • Yup, I dont think that anyone will take AMD seriously for any fault tolerant systems until they deal with the heat issue.

      Personally I dont even think its so much a matter of running hot, which although a nuisance isnt deadly, but more so that there are no safegaurds against overheating. Intel on the other hand, besides running cooler, downgrades the CPU if its overheating. AMD's XP does the same unless the temperature rises more than 1C per second, in which case it craps out anyway.

      Deal with the heat and they may be taken seriously, they certainly have the performance, not to mention it should total up to a nice savings for a couple thousand processor system.
      • Re:AMD Opteron (Score:3, Informative)

        Ugg.. I do WISH that people would stop reading "Tom's Hardware", or at least that they would get a clue first and realize that Tom doesn't know dick-all about what he's talking about most of the time.

        His comments about heat rising more then 1C/second make NO SENSE AT ALL! It's flat-out wrong! I don't know what orafice he pulled that comment from, but it certainly had no technical backing to it. The chip uses a thermal diode. It will tell you the temperature whenever you poll it. It doesn't matter how fast or slow you poll it, it will give you the temp. You would really have to go out of your way to try to break this sort of data to get it to only be able to handle a 1C/s temp increase.

        As for the heat "problem". AMD's AthlonXP chips have a maximum power consumption of roughly 50-70W. Intel's P4's have a maximum power consumption of roughly 50-70W (yes, they consume almost the exact same amount of power, check the data sheets).

        For comparison, Intel's Itanium has a maximum power consumption of around 100-130W, and IBM's Power4 is also on the high-side of 100W.
  • by MalleusEBHC ( 597600 ) on Tuesday October 08, 2002 @01:42AM (#4408280)
    Cluster Overview:
    * 2050 Intel 2.4GHz Xeon processors

    Now when people complain about the United States government being responsible for global warming they will have some good hard facts to use.
  • Good Stuff (Score:3, Interesting)

    by Perdo ( 151843 ) on Tuesday October 08, 2002 @03:54AM (#4408561) Homepage Journal
    "The Science Appliance" as it is dubbed will use dual processor AMD based nodes [linuxnetworx.com].

    Scary part is that this will be one of the top 5 supercomputers in the world.

    Scary because you could buy all the hardware off the shelf for about half a million dollars.

    On a lighter note:

    "The Linux NetworX cluster will be used solely for unclassified computing, including testing on ASCI-relevant unclassified applications."

    I think they mean text mode quake.

    I guess they got tired of "Global Thermo-Nuclear War"
    • Re:Good Stuff (Score:2, Insightful)

      by gregorio ( 520049 )
      Scary because you could buy all the hardware off the shelf for about half a million dollars.

      Scary? Why? Oh, and the interconnect hardware and installation is going to cost you more than 4x this value if you want good latencies and reliability.
      • Re:Good Stuff (Score:3, Interesting)

        by Perdo ( 151843 )
        Good dual amd boards come with gigabit ethernet. With prices as they are, the nodes can be put together for about $350,000. That would leave $150,000 for 512 ports of gigabit switches. Cisco gigabit 48 port switches run $5,000. Double that and add an additional nic to each box and use a flat neighborhood network [aggregate.org] (.pdf)

        That should give each node about 200 MB/s aggregate bandwidth (the best gigabit ethernet runs at 800 Mb/s or 100 MB/s), easily exeeding what can be achieved with much more expensive solutions.

        About the cost of a nice house.

        Put into perspective, a cluster that could outperform Japan's earth simulator would cost 2 million in hardware costs. Outperforming Seti@home's 3,000,000 users would require $10,000,000.

        I know where my lotto money is going :P

  • I think that mauve has the most RAM.
  • But doesn't it say " build, integrate and deliver a 1,024-processor Linux cluster " rather than 2048 cpu cluster.

Most people will listen to your unreasonable demands, if you'll consider their unacceptable offer.