aips2-request@nrao.edu User Test for R1.1 development cycle Tester: Debra Shepherd Date completed: 3nov04 Regression testing of scripts for all TST1 datasets. Goals: 1. Ensure that regression tests are being performed on a regular basis for ALMA test datasets. 2. Ensure that the test scripts exercise the entire process of reduction and imaging. Evaluation - top level report: ---------------------------------------------------------------------- ---------------------------------------------------------------------- 1. Ensure that regression tests are being performed on a regular basis for ALMA test datasets. ---------------------------------------------------------------------- There is a web site which summarizes the regression testing of all datasets that have been evaluated for ALMA testing: https://wiki.nrao.edu/bin/view/ISD/RegressionTests Web page shows that all datasets have successfully be processed with regression test scripts for the last development cycle. Problems have been noted and addressed. Goal 1. evaluation: Adequate. ---------------------------------------------------------------------- ---------------------------------------------------------------------- 2. Ensure that the test scripts exercise the entire process of reduction and imaging. ---------------------------------------------------------------------- Goal 2. evaluation: Some minor fixes and additions needed. Test scripts exercised most of the reduction and imaging steps with no masks or interactive flagging. This is reasonable - user tests will exercise interactive software. Scripts are comprehensive, well designed and the process of running them and recording timing results is effective. There were minor errors in the scripts as noted below. These should be fixed. Missing component that should be added: - At least one script should include uv continuum subtraction rather than image plane continuum subtraction. Perhaps the TST1.1 NGC 1333 data set could do this, or G192? - Create mom0 and/or mom1 maps of line emission and perhaps create a spectrum from one cube with the imageprofilefitter (can this be done automatically?). This will both exercise some analsysis routines and provide a few simple images and spectra that can be easily compared (e.g. this would have identified the problem with the autoflag.setdata(); issue in the L02D dataset). ============================================ ============================================ ============================================ Detailed script testing: Copied over test Scripts from: /home/ballista2/jmcmulli/ALMATST1/Regression/ G192/ H121/ L02D/ NGC7538/ TestAll/ Scripts also exist for GGTAU/ & NGC5921/ - excellent - I will not evaluate these in this test. Copied data from: /home/ballista2/jmcmulli/ALMATST1/Data/ ============================================= == G192 regression testing: == ============================================= Copied the automated script to map file. Took out the automatic components so I could evaluate each step. Also uncommented cal.plotcal steps to see calibration results. Compared this script with results and script that I developed during TST1. Continuum subtraction is in the image plane with only 2 channels selected for weak continuum emission. Not the best way to do it but I suppose that doesn't matter. The steps are evaluated. Reasonable results are obtained. Made moment 0 maps and compared them - all looks as it should. All steps completed successfully. There is a very nice section at the end of the script on regression test rules that give an initial evaluation of the flux density of the sources between latest and last test. If they are less than 5% different, then a message is printed out saying so. I did not run this segment but it is clearly necessary to determine, not only if the process ran in a given amount of time, but if the end result was scientifically correct. Differences and Issues: ======================= - vlafillerfromdisk automatically created scratch columns. The code worked fine. One problem: Humm, there was no logger message saying something to this effect: Adding MODEL_DATA, CORRECTED_DATA and IMAGING_WEIGHT columns Initializing MODEL_DATA (to unity) and CORRECTED_DATA (to DATA) Initializing natural weights I checked the latest documentation at: http://aips2.aoc.nrao.edu/weekly/docs/user/NRAO/node87.html#vla:vlafiller and the vlafillerfromdisk docs - no mention that I can find of the creation and initialization of the scratch columns. There should be. NOTE: ms.split does not add these extra columns - so you have to be careful to open the split ms with imgr (or cal) to add the columns. - I forgot to close out the autoflag tool. Then when I created an image tool to make a moment map I got messages about autoflag. This did not affect the processing steps - just odd behavior. I would not have thought that image and autoflag would interfere with each other. Especially since autoflag was operating on a split ms and the image tool was working with an image. What goes? Here are the relavent steps in the script: - af:=autoflag('g192_src.split.ms'); - af.setdata(); - af.setselect(timerng="2003/04/26/02:45:00 2003/04/26/02:49:00") - af.run(); T # ran imager to create g192_a2.im from g192_src.split.ms # no problems. - im:=image(infile='g192_a2.im'); - im2:=im.moments(outfile='g192.nh3.mom0', moments=0, axis=4, mask='indexin(4,[23:32])', includepix=[0.005, 1.]); - dv.gui(); error, autoflag is not a function value T - im3:=im.moments(outfile='g192.nh3.mom0-2', moments=0, axis=4, mask='indexin(4,[23:32])', includepix=[0., 1.]); - im3.done(); T error, autoflag is not a function value - af.done(); T ============================================= == NGC7538 regression testing: == ============================================= cp /home/ballista2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp* data/. Copied the regression script to 'map'. Modified data path to fill data from my local disk. For this test, do an automated run and examine cal solutions, logger output, and images afterward. Evaluate automatic processing procedures. - include 'map'; # glish responses are written to ALMATST1_regTest.log (not much). # Logger responses are not written out - these have to be examined from the log window. # continuum subtraction is not performed. script says: bad data file, do not run... # AIPS++ closes it self down and prints the message: The pass criterion of this test for calibrater is the difference of the amplitute of the flux density with the standard is less than 5% The pass criterion of this test for source data is the difference of the maximum amplitute of the source data with the standard is less than 5% NGC7538 run time is: 1510.92898 # restart aips++: # examine solution tables: cal:=calibrater(filename='ngc7538.ms'); cal.plotcal(tablename='ap314.gcal', plottype='AMP', multiplot=F); cal.plotcal(tablename='ap314.fluxcal', plottype='AMP', multiplot=F); cal.plotcal(tablename='1328.bcal', plottype='AMP', multiplot=F); cal.done(); # OK, solution tables look reasonable. # examine resulting images: dv.gui(); good gcal: 2229.cont2.im good src 55 chans averaged in split, fld 4: ngc7538d.cont.im 0.5" pixels, 1024 image, single plane good src with line+cont emission: ngc7538d.cube.im 4" pixels, 128 image cube w/ 48 chans # Manually run continuum subtraction # change line-free chan selection and fitorder = 0 (not 1): im:=image('ngc7538d.cube.im'); subim:=im.continuumsub(outline='ngc7538d_subed.line.im', outcont='ngc7538res.cont.im', channels=[1:5,44:48], fitorder=0); im.done(); subim.done(); # both images look fine. This should be put back in the script. Bottom line: automatic scripting runs fine but need to be careful that script exercises all steps - do not just remove segments that don't run rather than fixing those segments. Differences and Issues: ======================= - Put continuum subtraction back in the script. You will need to change the channel numbers selected in im.continuumsub to what is above (instead of channels=[2:53] which is in the script now and does not select correct channels). ============================================= == L02D regression testing: == ============================================= cp /home/ballista2/jmcmulli/ALMATST1/Data/L02D/*.fits data/. Copied the regression scripts: l02d_26jan_regTest.g l02d_29jan_regTest.g l02d_regTest.g Ran script for 26 and 29jan separately, examined results, then ran imaging using final master script l02d_regTest.g Modified data path to fill data from my local disk. Automated run: examine cal solutions, logger output, and images afterward. 26jan data reduction: ---------------------------------------------- - include 'l02d_26jan_regTest.g'; SEVERE error after autoflag selection: Please run setdata with/without arguments before any setmethod Affected segment: af:=autoflag('l.ms'); af.setselect(timerng=['2002/01/27/05:45:47.0', '2002/01/27/07:00:00.0']); af.run(plotscr=F); af.done(); There was no af.setdata(); step so this segment did not run or ran improperly. Glish warnings: SPLINE_CHECKToo many knots + lots of numbers... SPLINE_CHECKToo many knots + lots of numbers... SPLINE_CHECKToo many knots + lots of numbers... SPLINE_CHECKToo many knots + lots of numbers... SPLINE_CHECKToo many knots + lots of numbers... - it appears that the spline fit parameters are not appropriate now. gv l.3mmUSB.gcal0.PHASE.ps & - 2 widely separated points with wide phase variance, solution plots just show zero phase across these 2 points... Not right. gv l.3mmC34S.bpoly.ps & - OK gv l.3mmch3oh.bpoly.ps & - OK gv l.3mmcont.bpoly.ps & - OK gv l.3mmcont.gcal.PHASE.ps & - 2 widely separated points with wide phase variance, solution plots just show zero phase across these 2 points... Not right. cal:=calibrater(filename='l.ms'); cal.plotcal(tablename='l.3mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l.3mmcont.temp', plottype='AMP'); - yes, shows data that are widely separated - 6 hrs apart. gv l.3mmcont.pa.gcal.AMP.ps & - shows amp fit of 1 gv l.1mmUSB.gcal0.PHASE.ps & - phase fit = 0 gv l.1mmcont.bpoly.ps & - OK gv l.1mmcont.bpoly.ps & - OK gv l.1mmc34s5_4.bpoly.ps & - there is some wierd phase wrapping in here. gv l.1mmcont.gcal.PHASE.ps & cal.plotcal(tablename='l.1mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l.1mmcont.temp', plottype='AMP'); cal.done(); gv l.1mmcont.pa.gcal.AMP.ps & # This is not right. I looked in the orginal directory to see what the plots looked like there but the data and plots were not kept like they were for g192 (although NGC 7538 wasn't saved either). Rechecked the regressions web page, L02D was said to have run in 1124s on SS8. Re-run the script, add autoflag.setdata(); command to see if some data was incorrectly flagged, making the whole script look bonkers. - include 'l02d_26jan_regTest.g'; gv l.3mmUSB.gcal0.PHASE.ps & - OK gv l.3mmC34S.bpoly.ps & - OK gv l.3mmch3oh.bpoly.ps & - OK gv l.3mmcont.bpoly.ps & - OK gv l.3mmcont.gcal.PHASE.ps & - OK cal:=calibrater(filename='l.ms'); cal.plotcal(tablename='l.3mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l.3mmcont.temp', plottype='AMP'); gv l.3mmcont.pa.gcal.AMP.ps & - OK gv l.1mmUSB.gcal0.PHASE.ps & - OK gv l.1mmcont.bpoly.ps & - OK gv l.1mmcont.bpoly.ps & - OK gv l.1mmc34s5_4.bpoly.ps & - OK gv l.1mmcont.gcal.PHASE.ps & cal.plotcal(tablename='l.1mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l.1mmcont.temp', plottype='AMP'); cal.done(); gv l.1mmcont.pa.gcal.AMP.ps & - that was obviously the problem. The script was not updated to include af.setdata(); before flagging, too much got flagged and the rest of the script failed miserably. 29jan data reduction: ---------------------------------------------- # same problem exists with autoflag - add in setdata(); command # modify to fill data from my local directory. - include 'l02d_29jan_regTest.g'; gv l2.3mmUSB.gcal0.PHASE.ps & - OK gv l2.3mmC34S.bpoly.ps & - OK gv l2.3mmch3oh.bpoly.ps & - OK gv l2.3mmcont.bpoly.ps & - OK gv l2.3mmcont.gcal.PHASE.ps & - OK cal:=calibrater(filename='l2.ms'); cal.plotcal(tablename='l2.3mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l2.3mmcont.temp', plottype='AMP'); gv l2.3mmcont.pa.gcal.AMP.ps & - OK gv l2.1mmUSB.gcal0.PHASE.ps & - OK gv l2.1mmcont.bpoly.ps & - OK gv l2.1mmc34s5_4.bpoly.ps & - OK gv l2.1mmcont.gcal.PHASE.ps & cal.plotcal(tablename='l2.1mmcont.temp', plottype='PHASE'); cal.plotcal(tablename='l2.1mmcont.temp', plottype='AMP'); cal.done(); gv l2.1mmcont.pa.gcal.AMP.ps & Combined imaging of 29 and 26jan using l02d_regTest.g: ------------------------------------------------------ # comment out include statements that run above scripts. # comment says: # IMAGING_1MM_CONT := T; ## this step has error: illegal step pixel = ? for ... - include 'l02d_regTest.g' dv.gui(); all.1mmcont.cm - OK all.3mmcont.cm - OK all.c34s.cm - OK all.ch3oh.cm - OK l.1mm.cm - OK - some striping, bad data not flagged l2.1mm.cm - OK l2.3mmcont.cm - OK All images look OK, the test ran and passed the criteria. The problem noted above on IMAGING_1MM_CONT was not seen. Perhaps this is an old note? I know this used to happen but I thought it was fixed. Differences and Issues: ======================= - l02d_26jan_regTest.g af.setdata(); must be specified before flagging - modify script. I have no idea how this script could be run and give reasonable results. ============================================= == H121 regression testing: == ============================================= cp /home/ballista2/jmcmulli/ALMATST1/Data/H121/*.fits data/. Copied the regression script to 'map'. Modified data path to fill data from my local disk. Totally automated run of all days: examine cal solutions, logger output, and images afterward. copy over: h121_5may_1mm_regTest.g h121_5may_3mm_regTest.g h121_9may_1mm_regTest.g h121_9may_3mm_regTest.g h121_regTest.g # OK, for this one, try running everything, all days and image from the # master script. Modify script locations in master file and data # locations in individual day scripts. - include 'h121_regTest.g'; gv 2200.3mm.ph.gcal0.PHASE.ps & - OK gv 2200.1mm.ph.gcal0.PHASE.ps & - OK gv 2200.3mm.bpoly.ps & - OK gv 2200.1mm.bpoly.ps & - OK gv 2200b.3mm.ph.gcal0.PHASE.ps & - OK gv 2200b.3mm.bpoly.ps & - OK gv h121.3mm.ph.gcal0.PHASE.ps & - OK gv h121.3mm.bpoly.ps & - OK gv h121.3mm.ph.gcal.PHASE.ps & - OK gv h121.3mm.phamp.gcal.AMP.ps & - OK gv h121.1mm.ph.gcal0.PHASE.ps & - OK gv h121.1mm.bpoly.ps & - there are multiple chunks of data no problem - 3 spwids done at the same time. gv h121.1mm.ph.gcal.PHASE.ps & - OK gv h121.1mm.phamp.gcal.AMP.ps & - OK gv h121b.3mm.ph.gcal0.PHASE.ps& - OK gv h121b.3mm.bpoly.ps & - OK gv h121b.3mm.ph.gcal.PHASE.ps& - OK gv h121b.3mm.phamp.gcal.AMP.ps & - OK gv h121b.1mm.ph.gcal0.PHASE.ps & - OK gv h121b.1mm.bpoly.ps & - OK gv h121b.1mm.ph.gcal.PHASE.ps & - OK gv h121b.1mm.phamp.gcal.AMP.ps & - OK There is clearly some bad data left in here - OK, no problem. The solutions though look reasonable. dv.gui(); 0224b.1mm.im1: - OK 0224b.3mm.im1: - OK h121ab.co21.smallcube.im1: - beautiful h121all.3mm.im1: - yuck, looks like bad data in here, image OK h121b.co10.im1: - beautiful h121b.co21.im1: - OK h121c.co10.im1: - OK h121.co10.im1: - OK h121.co21.im1: - OK nrao150.1mm.im1: - OK nrao150.3mm.im1: - OK Glish response: ------------------------------------------------------------ The criterion passing the amplitude test is the difference of the maximum amplitute of correpondening quantity of the test run with that of the standard is less than 5% ---------------------------------------------------------- H121 run time is: 1991.43897 Differences and Issues: ======================= - none - script ran without a hitch and results look good.