Automating spectral reduction with the "reduce" script

The preceding pages in this document have described how to debias, flat-field, track and optimally extract a grating spectrum from a CCD image. The bad news is that this process has taken some time to set up. The good news is that once you've done this for one image it is quick and easy to do it for all other similar images, which is a great advantage of using starlink and pamela and molly.

The processing commands for an image can all be put into a script which will run automatically. This is also very complex, but it has already be done. The pamela command reduce runs a perl file which has all the appropriate commands in and requires one (big) input parameter file and a few other bits and pieces.

skymov:

First you must run the run the pamela command skymov -- on the same image as you used for regpic to get some parameters for the input datafile used by the reduce script. Give reasonable parameter values as prompted, and record the pixel value of the object position as outputted by skymov. This value is required in the reduction datafile.

Making the input datafile for reduce

Firstly, get hold of an example datafile from the online documentation on pamela or from here. Example datafiles also exist in this set of webpages. The datafile contains example values of all the parameters which have been used up to this point. You must now change the parameter values in the datafile to match those which you have been using to reduce your dataset. Remember that observations covering different wavelength ranges, or using different gratings, should be kept in separate directories and will in general require different parameter values in their version of the datafile.

The example datafile is heavily commented so it should be fairly easy to insert the appropriate parameter values for your dataset. There are several additional things to note:

Once you have set up the datafile, run the pamela script reduce on your images:

reduce [datafile] [list of files]

where the [list of files] can specify the images individually (but the usual UNIX wildcards are allowed) or alternatively, an @ and the name of a text file containing a list of images to reduce. For example:

reduce  datafile.in  2004-11*.sdf
reduce  datafile.in  @files.lis

This script goes through the images you have given it and does type identification, spectrum tracing, sky calculation, arc extraction, and spectrum extraction. Once it has finished, there will be a set of files for each input science image, such as:

2004-03-19_05-49-43.sdf
2004-03-19_05-49-43skyn.sdf
2004-03-19_05-49-43skyo.sdf
2004-03-19_05-49-43nor.sdf
2004-03-19_05-49-43opt.sdf

Here the original image is 2004-03-19_05-49-43.sdf. The normally extracted spectrum from it is 2004-03-19_05-49-43nor.sdf and the optimally extracted one is 2004-03-19_05-49-43opt.sdf. In addition, the measured sky level is given for the normal and optimal extraction in the files 2004-03-19_05-49-43skyn.sdf and 2004-03-19_05-49-43skyo.sdf. There will also be arc file associated with each science image, if appropriate arc images were found.

Sorting out the idtype algorithm

pamela originally used header items to determine whether each image was a science spectrum, flat field, bias or arc lamp image, but the inaccuracies of some headers meant that this needed to be improved. The reduce script now uses the idtype algorithm, which studies each image to find horizontal lines (indicative of arc spectra) and vertical lines (science spectra), so enabling the images to be classified without resorting to header information. This works well in general, but can sometimes need careful tuning to get the desired result. The relevant parameters are easy to find in the reduce datafile.

reduce also includes options to allow the user to specify the type of each image. Running reduce with the -t option:

reduce -t  datafile  @[listofimages]

causes it to produce a text file of datatypes, which can then be edited by the user. Specifying the -s option causes reduce to read the datatype file instead of running idtype on each image:

reduce -s  datafile  @[listofimages]

Troubleshooting

The first time you run the reduce script, it will probably fail for some reason. The reduce script is basically a parameter passer, which creates and runs a shell script to do all the work. This means that any problem is generally easy to track down. reduce also outputs several files containing details on what it is doing, and problems will show up in these: