How you organise your computational projects is important. If you’re not careful you can find yourself with hundreds of computer-hours worth of data, without being exactly sure which parameters or scripts you used to generate it. One way to keep track of your scripts and your data is with a JOBID number, which you attach to records of input and output files, and lists of parameters you used for each job.
On server-type systems such as high-performance computers, or research facilities like the Diamond Light Source synchrotron, which uses GDA, there is often a system set up to automatically generate a unique JOBID each time a job or script is run. This makes it much easier to keep track of exactly which simulation, calculation or operation was carried out. Inspired by these server-type systems, I’ve made a simple jobid generator script for running jobs on my own machine.
Each time you call the script it increments a log file with the JOBID, a time stamp, and records any comments you wish to make to the job. You can incorporate the JOBID output from this script into your own jobs.
Once you’ve got the JOBID it opens doors for you to make copies of your input scripts, your output files, and make a record of the parameters used for each job. It also makes it much easier and quicker to link your computational work to notes in a physical paper notebook, as one JOBID number immediately connects you to all work related to a particular calculation or experiment. I like to use the search program “Everything” to quickly find files relating to a particular JOBID.
Once you’ve got your unique JOBID, here’s a few things you could try.
- automatically save output files with jobid in the filename
- save outputs in a directory named after the jobid
- create a copy of your script file with the JOBID prepended to the filename
- create a separate parameters file, named after the JOBID.