How to extract calibration spectra using the offline pipeline.
Last edited 17 Feb 05

1.  Create and prepare the working directory

> mkdir /home/sloan/irsdata/test/
> chmod 775 /home/sloan/irsdata/test/
> cd /home/sloan/irsdata/test/

Copy setup.bat from the most recent calsfx campaign
(e.g. /home/ioc/campaign_12/IRS_070/setup.bat).
Run setup.bat to generate the directory tree and set permissions.

2.  Create the file target.lst

target.lst:

There are several ways to create this file, but they all involve 
editing.  The file could be generated by hand, or by editing
results from qdb results.

A general approach is to use the AORNAME field to limit a qdb 
search to the desired files.  The following example would find all 
AORnames containing the string "mc-00" (the MC_DUST program).

> qdb "select object,'  ',aorname,'  ',aorkey from onlinedata where expid=0 and dcenum=0 and aorname~'mc-00'" | sort | uniq > target.lst

Edit this file so each line has the form:

    hv2310    mc-0003   9528320 9544704 9546240 9636864

    Note that (1) there are no spaces or upper-case letters in column 1;
    (2) the second column does not contain suffixes like _1, _2 etc.;
    (3) all AORkeys for a given target appear on one line, separated
    by spaces.

See the file /home/sloan/irsdata/test/notes.txt for the commands 
used to generate the target.lst file for this test.

3.  Create the target and file listings needed to generate the batch
files used to extract and process the spectra.

a.  Generate file.lst in the mc_dust directory.

The most straighforward way to do this is to use an awk file 
created for this purpose to generate file.lst from target.lst

> cat target.lst | awk -f /home/sloan/tools/aorfile.awk | csh > file.lst

The example below could be used if all AORnames had a unique string
(i.e. the MC_DUST example from step 2):

> qdb "select subdir,localfile,' ',fovid from onlinedata where subdir~'10.5' and aorname~'mc-00' order by aorkey,expid,dcenum" > file.lst

It may be necessary to limit the search to one version number:

> qdb "select subdir,localfile,' ',fovid from onlinedata where subdir~'10.5' and aorname~'mc-00' and localfile~'2.b' order by aorkey,expid,dcenum" > file.lst

    Check file.lst to see if it contains duplicates for each DCE with
    different pipeline version numbers.  The pipeline version number 
    appears just before the "bcd_fp" or "bcd" field in the file name.  
    If there are multiple versions, rerun the above command with the 
    string "localfile~'N.b'" before "order", where N is the highest 
    version number.

b.  Generate Sl/sl.lst, SL/file.lst, SH/sh.lst, etc. from file.lst and
target.lst.

    IDL> mk_bat_lst,'file.lst','target.lst'

    Check the information mk_bat_lst prints to the screen.  It gives
    the target, pointing, AOR, and number of DCEs for each observation.
    The number of DCEs should match those expected from the AOR file.

4.  Generate the batch files to process spectra in the module 
subdirectories.

In IDL in SL:

mk_bat_cp,'file.lst','sl.lst','runcp.bat'
mk_bat_file,'sl.lst','imclean.idl',option=2
mk_bat_file,'sl.lst','imdiff.idl',option=35
mk_bat_file,'sl.lst','extract/runex.bat',option=41
mk_bat_file,'sl.lst','convert.idl',option=5
mk_bat_file,'sl.lst','offset.idl',option=6
mk_bat_cc,'sl.lst','cc.a1.lst'
mk_bat_join,'sl.lst','join.a1.idl',option=11,/plot

Edit cc.a1.lst to replace "##" with the correct version number for 
the correction file.  Copy the appropriate correction files into the 
SL directory (probably from the most recent calsfx campaign directory).
The most recent correction files are tr_slA_N_20.fits
  (A=1,2 for LL1 and LL2, N=1,2 for Nods 1 and 2).

In IDL in LL:

mk_bat_cp,'file.lst','ll.lst','runcp.bat'
mk_bat_file,'ll.lst','imclean.idl',option=2
mk_bat_file,'ll.lst','imdiff.idl',option=34
mk_bat_file,'ll.lst','imdiffx.idl',option=37
mk_bat_file,'ll.lst','extract/runex.bat',option=42
mk_bat_file,'ll.lst','convert.idl',option=5
mk_bat_file,'ll.lst','offset.idl',option=6
mk_bat_cc,'ll.lst','cc.a1.lst'
mk_bat_join,'ll.lst','join.a1.idl',option=11,/plot

Edit cc.a1.lst to replace "##" as for SL.
The most recent correction files are tr_llA_N_22.fits
  (A=1,2 for LL1 and LL2, N=1,2 for Nods 1 and 2).

In IDL in SH:

mk_bat_cp,'file.lst','sh.lst','runcp.bat'
mk_bat_file,'sh.lst','imclean.idl',option=2
mk_bat_file,'sh.lst','extract/runex.bat',option=41
mk_bat_file,'sh.lst','convert.idl',option=5
mk_bat_file,'sh.lst','offset.idl',option=6
mk_bat_cc,'sh.lst','cc.a1.lst'
mk_bat_join,'sh.lst','join.idl',option=5,/plot
mk_bat_join,'sh.lst','trim.idl',option=61

In IDL in LH:

mk_bat_cp,'file.lst','lh.lst','runcp.bat'
mk_bat_file,'lh.lst','imclean.idl',option=21
mk_bat_file,'lh.lst','extract/runex.bat',option=41
mk_bat_file,'lh.lst','convert.idl',option=5
mk_bat_file,'lh.lst','offset.idl',option=6
mk_bat_cc,'lh.lst','cc.a1.lst'
mk_bat_join,'lh.lst','join.idl',option=5,/plot
mk_bat_join,'lh.lst','trim.idl',option=61

In lores:

> cp ../SL/sl.lst .

In IDL in lores:

mk_bat_join,'sl.lst','join.idl',option=2,/plot
mk_bat_join,'sl.lst','clean.idl',option=3
mk_bat_join,'sl.lst','spplot.idl',option=4


5.  Run the batch files which copy and generate the image files from
which spectra will be extracted and complete the preparations for
extraction.

In each module subdirectory, in IDL where necessary:

> chmod 764 runcp.bat
> runcp.bat
IDL> imclean.idl
IDL> imdiff.idl          (SL and LL only)
IDL> imdiffx.idl         (if creating cross-diff files)

Go into the extract subdirectory and ensure that the necessary
calibration and namelist files are in place.  The subdirectory "qa" 
should be present and empty.  The subdirectory "cal" should contain 
the wavesamp file and psf_fov file.  The latter is not needed in the 
hi-res directories.  Make sure that the version numbers match those 
called in the namelist files: profile.nl, ridge.nl, extract.nl.  If 
any of these files are not present, copy them from a previous
campaign.

6.  Run the extractions.

Log onto one of the Solaris computers supporting the offline pipeline 
(for the moment, these are isc1-5).  Source the pipeline code:

> source /usr/local/pipeline/S11/env.csh

Note that you may need to type "S9" or "S10" instead of "S11" to run
older versions of the pipeline (which only run on isc4 and 5)

Go into the extract subdirectory of each module directory and run the
extractions:

> chmod 764 runex.bat
> runex.bat

7.  Complete the calibration process.

Move up to the module directory and put the data in the appropriate 
locations.

IDL> convert.idl
> mv *.sp.tbl tables
> mv *.sp.fits spectra
> mv *.pro.tbl profiles
IDL> spccbat,'cc.a1.bat',/plot
IDL> @join.a1.idl
> mv *.sp.fits coadd_a1

Combine and clean the lores spectra in the lores directory

IDL> @join.idl
IDL> @clean.idl

The results from coadd.idl are stored in clean.txt, giving the
normalization for each segment.