This page is linked from the goodies section.

GridStuffer

Download

GridStuffer 0.4.7 - Recommanded for Mac OS X.5 Leopard (not tested on OS X.4 Tiger)

GridStuffer 0.4.5 - Recommanded for Mac OS X.4 Tiger

It is recommended that you download and use the latest version. See change log. Older versions are available at the bottom of this page.

Description

GridStuffer is a Cocoa application to submit multi-task jobs using a very simple format that resembles very much the xgrid command-line format.

The idea is to emulate the bacth plist format without the complexity, and to provide a simple GUI and easy result retrieval. The application uses Core Data to store information about job progress and can restart where it left, for example after a reboot. GridStuffer is what is being used for the Xgrid@Stanford project to submit large-scale jobs over several days, and is thus being heavily tested in that configuration. If you are interested, you should read more details about the input file format before using GridStuffer.

If you use it, thanks for sending your feedback to charles•parnot@gmail•com. In particular, if you have some specific applications in mind, describe them to me, and I would try to propose an input file compatible with your goal. My primary goal for GridStuffer is to use it for Xgrid@Stanford, but I would want it to be as general as possible, and useful to other people too.

Disclaimer

The next versions of GridStuffer may not be able to read the information stored by version 0.4.0. This is mostly due to a limitation in Core Data, that makes files incompatible with each other after modifications in the data structure. Core Data is otherwise great, and I don't want to give it up just because of that particular issue. This limitation is thus also due to my laziness and to my own inability to make the good decisions early enough in terms of data structure. Hopefully, at some point, things will be stable enough that this won't be an issue anymore. Importantly, though, I have designed the application in such a way that several versions of it can be run concurrently, so that you could finish jobs started with an earlier version, while starting new jobs with a newer one. Ah!

Finally, this is just version 0.4.0, because I feel like it is quite far from a 1.0 version, in terms of features, testing and user interface. So use at your very own risk, and I shall not be liable for any direct, indirect, incidental, special, exemplary, or consequential damages arising in any way out of the use of this software, or out of the thought of the use of this software.

Source code

The download includes the source code, released under a GPL license. The Xgrid core functionality is in a separate framework, called GridEZ.framework, that can also be embedded in other apps that want to use the same input format for jobs. This framework is released under LPGL.

Screenshots

GridStuffer defines the concept of MetaJobs, that may represent several Xgrid jobs. A MetaJob consists of a simple list of shell commands defined in an input file (see format). To add a MetaJob, click the '+' icon to open a 'New MetaJob' window. Then, select an input file or load some of the demos to see examples. These demos will clutter your desktop a little bit, but don't worry, you can just trash them after use, with no harm to GridStuffer.
A MetaJob consists of a simple list of shell commands defined in an input file (see format). To add a MetaJob, click the '+' icon to open a 'New MetaJob' window. Then, select an input file or load some of the demos to see examples. These demos will clutter your desktop a little bit, but don't worry, you can just trash them after use, with no harm to GridStuffer. For example, if you load the Calendar demo, you will see that the input file simply consists of a list of cal commands.
One MetaJob can consist of thousands of commands. The status for each command can be checked. Each command can be run several times (or just once), can be validated based on its output and will be automatically rescheduled in case of job failures. For each command, GridStuffer keeps track of the number of successes, the number of failures, and will consider it 'Completed' or 'Dismissed' based on user-defined thresholds (set in the 'Submission' and 'Validation' tabs of the Inspector panel).
One MetaJob also consists of one or several Xgrid jobs, which themselves are made from one or several of the commands listed in the input file. GridStuffer automatically creates new Xgrid jobs as needed. The number of commands in one job is defined by the user in the Inspector. More Xgrid jobs are sent only as needed, on a regular basis. For instance, a MetaJob can consist of 1,000,000 commands, but only 1000 are really submitted at a given time to the Xgrid controller. The Xgrid jobs are conveniently named using the indices of the submitted commands (this is the name that will appear in Apple's Xgrid Admin application). The Xgrid jobs are automatically removed from the grid when they finish or fail.
The results of each command are automatically loaded to disk. There are two options. One is to let GridStuffer do all the work, and organize the results neatly in a hierarchical tree in the Finder. The other option is to use the -se, -so and -out flags in the input file (see format) to save the different pieces to costum locations. Importantly, files are NEVER overriden, and you may get several versions of the same files, if a command was submitted and completed several times. GridStuffer add indexed suffixes to the file names, or create result subdirectories.
GridStuffer allows you to connect to several controllers at the same time. The controllers found in the local network (via Bonjour) will automatically be listed as available. You can also add a remote controller by using the '+' button in the toolbar and type the address in the text field (you will only have to type that address once, because GridStuffer will remember it for the next time). At present, the jobs are automatically sent to all of the grids, trying to use all the resources at their best. In a subsequent version, you might be able to select a specific subset of grids to submit to.