Open Source Experiment Management Software?

Alea asks: "I do a lot of empirical computer science, running new algorithms on hundreds of datasets, trying many combinations of parameters, and with several versions of many pieces of software. Keeping track of these experiments is turning into a nightmare and I spend an unreasonable amount of time writing code to smooth the way. Rather than investing this effort over and over again, I have been toying with writing a framework to manage everything, but don't want to reinvent the wheel. I can find commercial solutions (often specific to a particular domain) but does anyone know of an open source effort? Failing that, does anyone have any thoughts on such a beast?"

"The features I would want would be:

• management of all details of an experiment, including parameter sets, datasets, and the resulting data
• ability to "execute" experiments and report their status
• an API for obtaining parameter values and writing out results (available to multiple languages)
• additionally (alternately?) a standard format for transferring data (XDF might be good)
• ability to extract selected results from experimental data
• ability to add notes
• ability to differentiate versions of software

In my dreamworld, it would also (via plugin architecture?) provide these:

• automatically run experiments over several parameters values
• distribute jobs and data over a cluster
• output to various formats (spreadsheets, Matlab, LaTeX tables, etc.)

Things I don't think it needs to do:

• provide a fancy front-end (that can be done separately - I'm thinking mainly in terms of libraries)
• visualize data
• statistical analysis (although some basic stats would be handy)

The amount of output data I'm dealing with doesn't necessitate database software (some sort of structured markup is ok for me), but some people would probably like more powerful storage backends. I can see it as experiment management 'middleware'. There's no reason such software should be limited to computer science (nothing I'm contemplating is very domain specific). I can imagine many disciplines that would benefit."

Piracy is Your Only Option

[use_compress]

1. You cannot (?) afford commercial software.
2. It is impractical for you to continue writing your own software.
3. You cannot find open source software.
-------
Conclusion: Steal commercial software! -)

Uh-huh

[Ryvar]

I don't mean to sound cynical, but this seems to come across to me as a very nicely written:

Ne3D H3lp WIt M4H H4x0RiN!!!!!

I mean, let's face it, much of what modern hacking closed-sourced software consists of is throwing a variety of shit against a variety of programs in a variety of configurations and seeing what breaks and then following up to make an exploit out of it.

While this probably isn't the case here, it's very hard to read that note and not snicker just a tiny, tiny bit . . .

Oh that's easy....

[Anonymous Coward]

The Academic Community, especially those strange AI people, have long sought complicated programs and machinery that could automate all of their work and projects, keep track of complicated "parameter sets, datasets, etc....".

But what you are looking for, sir, is the cheap labor commonly known as a Graduate Student

• Many of these "grads" [as they are commonly known] have INDEED been able to " 'execute' experiments and report their status", as well as "writing out results (available in multiple languages)".
• The Graduate Student is often known for their abilities to create and distribute notes in lieu of bringing that onerous burden upon more high-ranking academic officials
• ...you don't even have to dream about doing "clustered work" or "outputing results to spreadsheets, Matlab, LaTeX tables, etc....". These fancy machines can definately do that...
• Of course, there are several "graduate students" that provide a fancy front end (and rear end, for that matter). I think that I would agree with your assesment that they do not need to have that feature, although it might make your days a bit more... ermm... *pleasant* :-)
• As well, most graduate students have the capability of performing "basic stats", although most don't have an extensive faculty for performing such calculations...
• And don't you even worry about the price -- you'll see that they're quite affordable.

To conclude, you say that "There's no reason such software should be limited to computer science (nothing I'm contemplating is very domain specific). I can imagine many disciplines that would benefit". I would wholeheartedly have to agree with you: just about every discipline can do more and see farther by standing on the backs of their graduate students.
In fact, I'm afraid to report that you are a bit behind the times in this department as these "Graduate Student" devices are quite common at universities and research labs.

Re:Oh that's easy....

[Alea]

Ah, you see... there's the problem... I am, in fact, a cheap alternative to the much vaunted "Graduate Student". I'm a "Lazy Graduate Student" (TM), with slow update rates, poor accuracy, and long downtimes. Eventually, I'll probably break down completely into a "Professor", in which case someone will have to find some "Graduate Students" to get the work done...

Re:Perl is only useful for maintaining your job

[asciirock]

Just admit it. Perl slept with your wife. That's what this is really about, isn't it?

Re:Oh that's easy....

[quantaman]

Of course, there are several "graduate students" that provide a fancy front end (and rear end, for that matter). I think that I would agree with your assesment that they do not need to have that feature, although it might make your days a bit more... ermm... *pleasant* :-)

That does have it's advantages though you should be cautious. In my experience those models often have a large number of bugs in their systems and tend to be a lot more likely to pick up viruses as well.
This shouldn't be a problem for most operations but ocassionally if you try to interface them with your other components you may find your other systems becoming infected as well. In extreme cases you may also find interfacing with these systems can cause additional child processes to be created. These child processes are extremely hard to get rid of, early on you may be able to simply kill them but this command becomes extremely impratical after a few months of operation. These processes are known to take up huge amounts of resources and maintainance and often take the better part of 2 decades to subside (they're still present but resource demands drop considerably). Of course many of these risks can be alliviated by using a proper wrapper class while working with this "graduate student" systems.

you need...

[Alpha_Nerd]

It looks like you need - da da da da! - [b]EXTREME[/b] PROGRAMMING!