swsmake documentation (as of 10 Apr 03) all of the following are IDL procedures or functions: version last modified swsmake1.pro 4.0 29 Mar 02 segment.pro 4.1 16 Dec 02 segmentread.pro 4.1 20 Jan 03 segmentmed.pro 4.0 17 Jul 02 segmentmrg.pro 4.1 17 Apr 02 segmentmod.pro 3.0 17 Jul 02 segmentmad.pro 3.0 17 Jul 02 segmentpass.pro 4.0 25 Apr 02 segmentsort.pro 2.0 3 Oct 00 segmenthigh.pro 3.0 27 Feb 02 segmenterr.pro 3.0 27 Feb 02 spike.pro 2.1 30 Apr 02 gridlo.pro 3.1 31 Jan 02 swsmake2.pro 4.4 6 Sep 02 seg2segin.pro 4.2 8 May 02 seg2segout.pro 4.2 16 Jun 02 seg2segtwo.pro 4.3 18 Dec 02 segmentavg.pro 2.1 21 Jul 02 splice_in.pro 4.1 4 Apr 02 splice_in2.pro 4.1 18 Apr 02 splice_out.pro 4.2 26 Apr 02 splice_two.pro 4.3 23 Jul 02 cut.pro 3.0 19 Mar 02 swsread.pro 4.0 28 Mar 02 sws6read.pro 4.0 7 Apr 03 curvefit.pro IDL supplied We also use the following programs and batch files to automate the use of swsmake: flslws - executable (obtainable from the IPAC website) isotrim.c - C program isosort.c - C program runall - text batch file mk_sws_bat.c - C program The full procedure starts with a FITS file produced by OLP 10.1 and produces a spectral file in 4-column format (along with 2 header lines) Step 1. Convert the swaaNNNNNNNN.fits file to ASCII text using flslws; NNNNNNNN is the TDT number flslws -o all.dat swaaNNNNNNNN.fits Step 2. Use the batch file runall to call isotrim and isosort runall is written to read in file "all.dat" calls isotrim 24 times (12 detectors, 2 directions) saves isotrim output by redirecting it to files fNNa.dat NN = segment number (1-7, 9-13) a = "p" for +1 scan direction, "n" for -1 dir'n calls isosort 24 times saves isosort output by redirecting it to files fNNas.dat "s" for sorted deletes all files unnecessary for further processing i.e. all of the original isotrim output files only the isosort output files (24) are saved isotrim.c - C program to extract from file all.dat each spectral segment and scan direction (24 total) usage: isotrim input_file [filter direction] isotrim output is written to screen this should be redirected to a file isosort.c - sorts each fNNa.dat file into order by (1) detector (2) wavelength usage: isosort input_file input_file should be an output file from isotrim isosort also writes output to screen this should be redirected to a file Step 3. Run swsmake1.pro in IDL swsmake1 produces one combined spectrum for each of the 12 spectral segments and writes these spectra to disk as 1 file swsmake1.pro calls segment.pro 12 times, once for each segment a typical call to segment.pro segment,'f07ns.dat','f07ps.dat',l07,f07,e07,flag07,seg=7,wt_n=0.5 note that the -1 scan direction is first results saved as l07, f07, e07 flag07 (wavelength, flux, error, flag) segment.pro reads in two fNNas.dat files for each segment and processes the spectra from the 24 individual detectors into one spectrum for the segment segment.pro parameters set in call seg - segment number, used for bookkeeping only wt_n - weight of -1 scan direction used to combine median_n and median_p into median wt_p = 1.0 - wt_n; both default to 0.5 fixed in procedure, can be tuned, but not recommended npass=2 - for spike.pro as called by segmentmed.pro range=3 - ditto threshold=1.0 - ditto ndegree=3 - order of polynomial in segmentmod (and mad) sigpass=3.0 - factor to multiply pass array by (after segmentpass) mpass=0.2 - for segmentmrg.pro segment.pro calls the following: segmentread,file_a,raw_a,cnt_a,l_lo_a ; "a" = n or p this procedure reads in one FNNa.dat file loads raw_a - raw data array (2,12,len) loads cnt_a - 12-element array with length of each spectrum creates l_lo_a - low-resolution wavelength grid - matches resolution of typical detector spectrum returns status_a=0 if no data in segment, which will cause segment.pro to cancel remainder of calls before the next call, swsmake1 loads the lo-res grid with more wavelength elements as the lo-res grid to use for all other calls segmentmed,raw_a,cnt_a,l_lo,grid_a,flag_a,npass,range,threshold produces grid_a and median_a grid_a - the 12 individual spectra regridded onto lo-res grid median_a - the median of the 12 detectors on the lo-res grid flag_a - set to 1 if any median value below limits of 15-20 Jy call includes three parameters for spike rejection npass - number of passes to be made range - number of detectors to either side of test position to consider threshold - number of standard deviations to take as rejection criterion algorithm call spike.pro (with parameters) to reject and replace all spikes and narrow lines before regridding the spectra this will give a pseudo-continuum for spectrum NOTE - the spike rejected data are only used in the lo-res grid regrid all 12 spectra onto the lo-res grid use gridlo.pro to do this - weighted gaussian convolution time consuming produce median spectrum simple median at each element of lo-res grid if all median data < 15 Jy or max < 20 Jy, sets flag=1 this will change following corrections from multiplicative to additive segmentmrg,median_n,median_p,flag_n,flag_p,wt_n,median,flag,mpass combines median_n and median_p into median if either flag_n or flag_p = 1, returns flag=1, else 0 if flag=0 or wt_n set to use only one scan direction then segmentmrg returns a simple weighted average for median i.e. the weights are set by wt_n and wt_p=1-wt_n and they are the same at all wavelengths but if flag=1 and both scan directions have non-zero weights then segmentmrg checks to see that both media are consistent i.e. the difference between them is less than mpass*average call includes one parameter mpass - maximum allowed difference between media in two scans defaults to 0.20 (20% of average of median_n and median_p) where the difference exceeds mpass over 3+ wavelength elements segmentmrg changes the weights to use only the better-behaved scan thus the weights can vary with wavelength this can lead to discontinuities in the returned median but it will stop poor data from propagating errors if the flag for segment was not set to 1 by segmentmrg, then swsmake1 next calls segmentmod; otherwise it calls segmentmad segmentmod,raw_a,cnt_a,l_lo,grid_a,median,mod_a,ndegree generates a corrected spectrum for each detector raw_a - raw data generated by segmentread cnt_a - length of spectra for raw_a (segmentread) l_lo - lo-res grid (segmentread) grid_a - regridded and spike-rejected spectrum from an individual detector (segmentmed) median - generated by segmentmrg mod_a - the corrected spectrum produced here (on original grids) ndegree - parameter: degree of polynomial fit 1 for linear, 2 for quadratic, 3 for cubic should add 0 for simple shift algorithm: compare individual regridded spectrum to median this produces an array of multiplicative offsets vs wavelength use this offset vector to correct full unregridded spectrum make a polynomial fit to this offset vector use polynomical coefficients to interpolate correction to wavelength on individual detector grid apply this correction to generate one element of mod_a NOTE - the correction is determined from grid_a (i.e. the regridded spike-rejected data), but it is applied to raw_a (the raw data, complete with spikes) these steps effectively correct for different curvatures in spectra from individual detectors segmentmad operates identically, except that the offsets and corrections are additive calls to segmentmad are made with ndegree=1 segmentpass,mod_a,cnt_a,l_lo,median,pass_a generates the pass array - the typical deviation about the median at each wavelength - to be used to reject bad data in segmentsort algorithm: shift curvature-corrected spectra from segmentmod (or mad) to the lo-res grid using gridlo compute an array of additive differences between these and median (a difference spectrum for each detector) use the difference array to find the typical deviation take max of (1) median of abs value or (2) standard deviation at each wavelength result: typical deviation at each wavelength on lo-res array segment now combines pass_n and pass_p into a minimum pass vector at each wavelength, the minimum of pass_n and pass_p is used this will force the rejection of apparent atomic lines which only appear in one scan direction the result is multiplied by the sigpass parameter (typically 3) segment next merges the count and moddata arrays count - merger of cnt_n (first) and cnt_p (following) moddata - merger of mod_n (first) and mod_p (following) some care taken here to keep array lengths correct because some entire spectra from individual detectors may be missing, the total might not be 24 segmentsort,moddata,count,l_lo,median,pass,sortdata sortdata - result of segmentsort bad data rejected, sorted by wavelength algorithm: flag bad data in the spectrum from each detector shift pass and median (called continuum) to actual wavelength grid for detector - both shifts require gridlo.pro unfortunately compare actual difference between moddata and median (i.e. continuum) to pass flag the failures copy moddata sequentially into 1-D arrays now have lambda, signal, flag (generated in previous step) use IDL sort algorithm to put these into wavelength order copy the sorted data into final sortdata array 2-column format - wavelength and flux skip data which have been flagged, count good elements segmenthigh,sortdata,seg,l_hi,final produces l_hi and final from sortdata l_hi - high-resolution wavelength grid with uniform spacing resolution fixed, based on segment and speed range of new grid based on range of actual data final - fluxes in sortdata regridded to uniform grid final = gridlo(sortdata,l_hi) segmenterr,grid_n,grid_p,cnt_n,cnt_p,error produces error estimates on lo-res grid, regrids to hi-res grid based on std deviation of regridded data at each wavelength element note that this is before any detector-to-detector corrections error contains uncertainty in mean at each position swsmake1.pro calls segment.pro 12 times segment then writes the results out to file "all.pws" "pws" = preliminary SWS file pws format: line 1 - 12 integers - lengths of each spectral segment line 2 - 12 integers - flag for each segment (set by segmentmed) following lines in 4-column format wavelength, flux, spectroscopic error, photometric error units are as given in starting file: um, Jy, Jy, Jy as generated by swsmake1, the two error columns are equal that will change in swsmake2 swsmake1.pro also generates a log file ("all.log") the heart of the log material is produced by segmentmod/mad it prints two sections of coefficients for each detector format: detector number, polynomial fitting coefficients the detector number runs from 0 to 11 for all segments the number of coefficients depend on the degree of the fit segmentmod will specify that these are multiplicative segmentmad will specify they are additive two sets of coefficients are written for each segment first set: negative scan direction second set: positive scan direction Step 4. Run swsmake2.pro in IDL swsmake2 performs two functions (1) normalization of spectral segments to eliminate discontinuities (2) elimination of bad data from the ends of each spectral segment swsmake2 has one command-line parameter - type type determines the nature of the normalizations if type > 10, then we are avoiding emission lines in certain regions of overlap, calls seg2segin.pro with 4 wavelengths instead of 2, which results in a call to splice_in2.pro instead of splice_in.pro type also determines the nature of the Band 3D/4 normalization see the splice_out discussion for details swsmake2 first reads in the all.pws file normalization runs from the brighter of Band 1E or 3D to the fainter using calls to seg2segin.pro, which calls splice_in.pro splice_in uses a region of overlap between adjacent spectral segments to normalize one to the other seg2segin,fixed,fseg,adjust,aseg,ll,flag,nfa fixed,fseg - fixed data and segment number adjust,aseg - adjustable data and segment number ll - array of wavelengths flag - 0 for multiplicative normalization, 1 for additive nfa - correction factor and uncertainty in correction if ll contains 2 wavelengths, then splice_in called but if ll contains 4 wavelengths, then splice_in2 called splice_in,la,fa,ea,lb,fb,eb,l0,l1,MODE la,fa,ea - wavelength, flux, and error of reference segment lb,fb,eb - wavelength, flux, and error of segment to be normalized l0,l1 - wavelength range for normalization mode=0 for multiplicative normalization mode=1 for additive normalization averages two segments over l0-l1 range to find normalization errors are photometric and include propagated errors from previous calls to splice_in splice_in2,la,fa,ea,lb,fb,eb,l0,l1,MODE l0,l1,l2,l3 - wavelength ranges for normalization as splice_in, except averages over l0-l1 and l2-l3 skips l1-l2 (used to avoid em lines in regions of overlap) normalization continues from Band 2A to Band 1A always from red to blue, always using seg2segin.pro the Band 1E/2A interface may require some attention due to Br-alpha normalization of Band 4 to Band 3D requires extrapolation calls seg2segout.pro, which in turn calls splice_out.pro seg2segout,fixed,fseg,adjust,aseg,type,flag,prop_err,nfa fixed,fseg - fixed data and segment number adjust,aseg - adjustable data and segment number type - determines wavelength ranges and degree of polynomial for normalization of segment 13, otherwise ignored flag - 0 for multiplicative normalization, 1 for additive prop_err - propagated error from previous normalizations not currently used nfa - correction factor and uncertainty in correction splice_out.pro is used to normalize Band 4 (seg 13) to Band 3D it performs normalization between non-overlapping segments by fitting a polynomial to each segment and extrapolating them to a reference position splice_out,la,fa,la0,la1,lb,fb,lb1,fb1,lc,DEGREE,MODE la,fa - wavelength and flux of reference segment lb,fb - wavelength and flux of segment to be normalized la0,la1 - range of segment a for polynomial fit lb0,lb1 - ditto for segment b degree - degree of the polynomial =0 is a special case linear for reference segment average for normalized segment (i.e. line with slope fixed to 0) =1,2,3 for linear, quadratic, cubic mode=0 for multiplicative normalization mode=1 for additive normalization the normalization of Band 3E to its neighbors requires special attention using seg2segtwo.pro seg2segtwo,fixed1,fseg1,adjust,aseg,fixed2,fseg2,flag,prop_err,nfa fixed1,fseg1 - fixed data and segment number on one side fixed2,fseg2 - fixed data and segment number on other side adjust,aseg - adjustable data and segment number flag - 0 for multiplicative normalization, 1 for additive prop_err - propagated error from previous normalizations not currently used nfa - correction factor and uncertainty in correction splice_two.pro normalizes one band to its two neighbors by finding the average over the ranges specified in the call calls segmentavg.pro to determine the average splice_two,la,fa,la0,la1,lb,fb,lb1,fb1,lc,fc,lc0,lc1,MODE la,fa - wavelength and flux of first reference segment lb,fb - wavelength and flux of segment to be normalized lc,fc - wavelength and flux of second reference segment la0,la1 - range of segment a for polynomial fit lb0,lb1 - ditto for segment b lc0,lc1 - ditto for segment c mode=0 for multiplicative normalization mode=1 for additive normalization the type parameter in swsmake2 sets the wavelength ranges and the polynomial to normalize Band 4 type=0 (default) - standard for most spectra standard wavelength ranges, quadratic fits type=1 - for sources with 26 um feature shortened wavelength range for Band 3D linear fit type=2 - for some bright red sources shortened wavelength ranges for both bands linear fit type=3 - for some fainter red sources shortened wavelength ranges for both bands degree=0 (linear fit to Band 3D, avg for Band 4) other types may be added as needed NOTE - splice_in, splice_in2, splice_out, and splice_two do not normalize they simply return the normalization constant seg2segin, seg2segout, and seg2segtwo actually apply the corrections (either additively or multiplicatively) and propagate the errors swsmake2 then calls cut.pro for each spectral segment cut.pro truncates bad data at each end of the segment cut,l,f,l0,l1,l_out,f_out,len l,f - input wavelength and flux l0,l1 - range of valid wavelengths l_out,f_out - returned wavelength and flux len - length of returned spectrum 0 if cut range didn't overlap with spectrum swsmake2 calls cut three times for each segment, once for flux, once for spectroscopic errors, and once for photometric errors finally, swsmake2 prints out the final spectrum to file "all.sws" sws format is identical to pws format swsmake2.pro also appends the log file: "all.log" gives details of all calls to splice_in and splice_out resulting normalization factors and wavelength ranges for cut Step 5. Clean up copy the files "all.pws", "all.sws", "all.lg1", and "all.lg2" to someplace safe so that they don't get overwritten when the next TDT is processed standard UNIX commands: mv all.Xws NNNNNNNN.Xws mv all.log NNNNNNNN.log X = p or s, NNNNNNNN = TDT no. discard all.dat and all of the fNNa.dat files Review to process the spectrum in the file swaa07900152.fits in IDL: 1 $flslws -o all.dat swaa07900152.fits 2 $runall 3 swsmake1 4 swsmake2 5a $mv all.pws 07900152.pws 5b $mv all.sws 07900152.sws 5c $\bin\rm *.dat note that lines 4 and 5b can be done later at the moment, though, swsmake2 expects to see file "all.pws" so one would have to $mv 07900152.pws all.pws swsmake2 $mv all.sws 07900152.sws Tools swsread.pro reads in a data file produced by swsmake1 or 2 swsread,'NNNNNNNN.sws',l,f,e,p it does not distinguish between segments it just produces 4 long 1-D arrays if you use it to read a pws file it works, but there will be some overlaps and discontinuities mk_sws_bat.c produces a series of commands to run in batch mode use: mk_sws_bat input_file > idl_batch_file input_file: a list of file names, one per line each of form swaaNNNNNNNN.fits or NNNNNNNN idl_batch_file: a series of commands for each fits file similar to those seen in Review section above this facilitates automated processing