Spectra from the blue optical wavelength range will not normally need to have telluric lines removed from them, whereas spectra covering redder wavelengths will. If you can avoid using regions with telluric lines then this is the best option, as they cannot be completely removed.
I will go through flux calibration first, then combine this with telluric removal afterwards.
Firstly, you will have had to have observed a spectrophotometric flux standard star. Discussions and the absolute spectra of several flux standards can be found in Oke (1990, AJ, 99, 1621) and Oke & Gunn (1983, ApJ, 266, 713). Tabulated data for the standards can be obtained from several sources, such as ESO and ING. These are available in two different types:
molly differentiates between these by the number of columns in the datafile: three columns means Oke-type data and two columns means Filippenko-Greenstein.
Once you have selected your star and obtained the tabulated data it will need a slight modification before use. If the measurements are given in absolute flux units (milly-Janskys) you must add a line at the start of the datafile with Fluxes=mJy. If the measurements are in AB magnitudes you should add the line Fluxes=AB.
Start molly and read in your spectra. Choose the flux calibration star and give the flux command
flux [flux-star-slot] [telluric-star-slot]
[telluric-star-slot] should be put to zero at present (we will deal with this below, as it is important for data in the red and infrared).
Various other arguments now need to be specified:
[pixels] will be needed for Fillipenko-Greenstein data. A good value is 20 pixels, but this should be increased if the data have low signal to noise.
[number of splines]: start off with 5 (you will probably add more later).
[lower reject threshold] and [upper reject threshold]: give large values for these (about -20 and 20) to avoid rejecting data at the ends of the spectra where vignetting is important.
[number of reject cycles]: 10 is normally fine.
[linear fit] normally choose 'y', but choose 'n' to do a logarithmic fit if your data have a very large dynamic range.
[weight factor]: normally choose '0' for no change or '1' for unit weighting.
Now choose to plot the fit: you will see a plot of the number of counts per pixel in the spectrum (white circles), a set of green dashed vertical lines indicating where the knots of the cubic spline have been placed, and a white line indicating the spline fit to the spectrum. The knots will be spaced uniformly, and the resulting spline will probably fit very well where the data are changing slowly but fit less well if the data have waves or other structure.
Unless the fit is already very good, choose to edit the spline knots and refit. You can now add extra spline knots at points where variations in the data values cause the current spline fit to be poor, and remove additional any spline knots which are not needed. To add a spline knot, move the mouse cursor the the point where you want it and press A. To remove a spline knot, point at it with the mouse and press R. You can of course move knots by removing the current knot and adding a new one at a different point. You will not normally want to change the spline knots too much at each iteration, or end up with more than about 10 to 12 knots. Don't add splines right at the ends of the data as this can cause the fit to fail. Once you have adjusted the knot positions, press Q to see the new fit. You can keep doing this until you get a good spline fit.
Flux calibration is not very accurate, and a very good fit can be difficult to achieve. Even an OK fit can be hard or impossible in regions where there are many spectral lines (e.g. the Balmer jump around 4000Å). You should be aware that this is not a particularly reliable process.
If some strong lines are causing big problems in the fit, then try masking them out (using the mask command) of the spectrum before flux-calibrating.
Once you have a good fit, enter 'n' to stop iterating. You can now inspect the result by plotting the spectrum using first counts and then milli-Janskys as the y-axis quantity. If your spectrum is in slot 101 then type
ax a c ; pl 101 101 ax a m ; pl 101 101
several times. Remember that you can easily blink the plots by using the up-arrow key to recall previous commands (i.e. up-arrow, up-arrow, enter for each replot). If the result is good you now have a flux calibration which can be applied to your other spectra.
Now you have a flux calibration you can apply it to your science spectra using the fcal command:
fcal [first-slot] [last-slot] [flux-cal-slot] [telluric-cal-slot]
As you have not yet made a telluric-correction star, put 0 for [telluric_cal_slot]. At the remaining prompts put 0.5 for the [grey extinction coefficient] (this is not important normally) and 1.0 for the [Rayleigh factor] (you could choose other values if you know what you're doing). The flux calibration will then be applied to the target spectra.
For red-wavelength spectra it can be very important that telluric lines are removed. In molly the flux and fcal commands are set up to do these two things at the same time, and it is also possible to use the same calibration spectrum both as a flux standard and a telluric standard ("water star" in the molly terminology).
First fit a function to your flux standard spectrum and divide by the fit so its continuum level is roundabout 1.0. Polynomial fitting is usually the best, using the command pfit. Specify your target slot, output fit slot, use a polynomial order of 3 to about 7 (depending on the shape of the spectrum) and apply a mask. Assuming your spectrum is in slot 101 and the fit will be put in slot 102, the command is:
pfit 101 101 102 [order] -3 3 10 1.0 YES
When the mask menu comes up, plot your spectrum and mask out all the telluric and stellar spectral line regions. Quit the mask menu and the fit will be calculated. Plot the spectrum and its fit in different colours to check that the fit is good:
pcl ; pl 101 101 ; dc 3 ; pcl ; pl 102 102 ; dc 1
If the fit is good then rectify the spectrum to the continuum by dividing by the fit:
div 101 101 102 102 103
Now we want to set most regions of the rectified spectrum to exactly 1.0 to avoid introducing the noise in it to the target spectra later. Regions with stellar spectral features should also be set to 1.0 but the telluric regions should not be. Use the cset command:
cset 103 103 1.0 YES
When the mask menu comes up, plot slot 103 and then type m to mask. Choose to do it with the cursor and then mask out the regions not containing signifcant telluric features. Quit the mask menu and then plot the result to check that everything but the telluric features has been set to 1.0. It is also usually a good idea to limit the spectrum values to between 0.0 and 1.0 to make it smoother. To do this use the limit command:
limit 103 103 0.0 1.0 YES
and plot the mask out the whole spectrum. Plot the result - it should be very similar to before but any noise spikes are no longer poking up above 1.0 or below 0.0. You now have a telluric spectrum.
Now we use the flux command with a telluric spectrum:
flux 101 103
For water exponent the best value seems to be 0.6. For fudge watermass?, 'yes' means that it will try to fit the telluric spectrum directly to the stellar spectrum to find the best fit, and 'no' means that it would calculate the telluric strength according to the airmass of the flux spectrum; it is probably a good idea to try both. If you do fudge watermass, the filters are to remove large- and small- scale variation and it is best to accept the default values of 31 and 3. You will then need to use the mask menu to indicate a small area of spectrum where the telluric lines are strongest, with which it find the best-fitting strength of the tellurics. If you do not fudge watermass, then accept the default of 1.0 for the ratio of airmass to watermass.
Now fit the continuum with a spline as before to get your flux calibration.
We use the fcal command as before, but now we also have a telluric spectrum:
fcal [first-slot] [last-slot] 101 103 0.5 1.0
Give a value of 0.6 for the telluric scaling exponent. For rebinning (needed to make the wavelength scales of the telluric and other spectra match), the usual choices are q (quick) or s (slower but better). Then choose 'y' or 'n' to fudge the spectra (if you choose 'y' then it will output the best-fitting airmass values compared to actual ones) and follow the additional steps resulting from this as in the paragraph above. Once the flux calibration has occurred you will find that the target spectra are flux calibrated and that their telluric lines have been (approximately) removed.