October/November 2004 ALMA Offline software test NAME: Debra Shepherd Questionnaire on testing experience 1. Please list briefly your background in the following areas: A. Radio Interferometry ((sub)millimeter or centimeter) I've reduced OVRO 1 & 3mm data, VLA mm & cm data, PdBI millimeter data. I have worked extensively with OVRO data, mostly CO or SiO mosaics \& continuum. I've also worked with BIMA data (combining 12m+BIMA single field). B. Experience with VLA and/or PdBI data I've been reducing VLA data for about 6 years (at first using AIPS, now using AIPS++ exclusively). I have reduced 3 PdBI datasets in the past year, all in the AIPS++ package. C. Astronomical Data Reduction packages: - AIPS - Used occasionally for the past 6 years - MIRIAD - Used extensively for imaging OVRO data - MMA - Used to reduced OVRO data - Gildas/Clic- Attended a Clic tutorial in early 2003, have never reduced data in Gildas/Clic - AIPS++ - Testing AIPS++ for about 4.5 years, for testing preparations, I reduced and imaged the other datasets (2 VLA and one other PdBI) but did not look at the dataset I was assigned. D. How much experience have you had with the AIPS++ software package before this test? Significant. I've been reducing all my VLA science data with AIPS++ for the past 2 years. 2. Please identify which dataset you processed during this test: A. VLA SiO(1-0) observations of NGC 1333 B. BIMA CO(1-0) observations of NGC 4928 Reduced both datasets during pre-testing. Also evaluated TST1.0 datasets in regression tests: VLA NH3(1,1) observations of G192.16 VLA NH3(1,1) & (2,2) observations of NGC 7538 PdBI 1 & 3mm line observations of U Cam PdBI 1 & 3mm line observations of IRAS 20126 3. Were you able to complete the fill, editing, and calibration of the data? If not, why? Please comment on specific steps if desired (comments can be positive or negative): A. Filling the data into AIPS++ format yes B. Editing and visualizing your data yes C. Atmospheric phase corrections (PdBI data only) yes D. Gain (phase and amplitude) calibration yes E. Absolute flux calibration yes F. Bandpass calibration yes 4. Were you able to subtract continuum in the uv-plane (if applicable)? If not, why? No, no continuum in these datasets so I didn't try. 5. Were you able to split out the calibrated data (if desired)? If not, why? Yes, no problems. 6. Were you able to image the data? If not, why? Please identify any problems you had during imaging. Yes, no problems at the end. 7. Were you able to analyze the images adequately to determine if the results you obtained were scientifically reasonable (e.g. display the image, calculate RMS and peak, make a moment map or take a spectrum)? If not, why? Yes. 8. Please summarize the final results of your image(s): - RMS: NGC 1333: Both days concatenated together: # RMS = 1.7-2.5 mJy/beam (blue emission fields are noiser) 2may data only: # rms = 1.6-2.2 mJy/bm - better... really? 8may data only: # rms = 3.9-5.4 mJy/bm # so adding in the 2nd days worth of data doesn't really improve image NGC 4826 # RMS = 40 mJy/bm # peak = 1.913 Jy in channel 25 - Peak Flux Density: 9. Did you have adequate support during your test? If you contacted the AIPS++ groups for questions or to fix a bug, please comment on the interaction and whether it was helpful. Yes, Joe McMullin was especially receptive to helping to ensure that the test preparations were adequate, despite some delays. Actual support for other testers looks excellent (I monitored e-mails). There were some problems with the regression tests - I discussed these with Joe and he said he agreed with what I had found and said that he would fix the problems. 10.Was AIPS++ easy to install? If not, why? I did not install AIPS++ 11.The Synthesis Reduction Cookbook you used for this test is the first version of a comprehensive cookbook for ALMA users. Please evaluate the organization, content, and presentation of the cookbook. It is meant to be the first documentation users will see when they want to reduce ALMA data, it provides background on the code capabilities, and extensive examples. The on-line documentation provides more details and code descriptions. With this in mind, please answer the questions below. If you have detailed comments, please attach them to the end of this questionnaire. - Was the documentation adequate for you to complete your test? - Was the cookbook good? - Do you have any suggestions for how to improve the cookbook? I will not answer these questions about the cookbook since I did extensive editing of the first 2 versions. I'll let external testers decide on the cookbook quality. Note, there are still some sections that need to be filled in so the cookbook is still incomplete. - Was the on-line documentation good: * User Reference Manual? * Supporting documentation? No - User Ref Man is still not adequate. Needs a major overhaul - this should be done when the code moves to a task-based system with python scripting. Not useful to do it before. 12.Roughly how much time did you take to perform the following steps: - Installing aips++: N/A for pre-testing - Fill, editing, & calibration: N/A for pre-testing - Continuum imaging: N/A for pre-testing - Spectral line imaging: N/A for pre-testing - Analysis: N/A for pre-testing - Regression testing: about 20 hrs - Filling out this questionnaire: 1hr - Evaluating and grading the scientific requirements: 1hr - Total time: 13.Please rate your overall testing experience: - excellent x good - fair - poor - horrid 14.Was the test well designed and executed by those in the ALMA offline subsystem (e.g. the subsystem scientist and the Offline subsystem group). If not, can you provide any suggestions for improving the next test? Yes, basic regression test page is well designed. Some of the actual regression scripts need minor fixes but these can be done easily. 15.Do you have any additional comments that may help improve test of the offline software in the future? Here is a top level summary of my regression testing results: ---------------------------------------------------------------------- ---------------------------------------------------------------------- Goals: 1. Ensure that regression tests are being performed on a regular basis for ALMA test datasets. 2. Ensure that the test scripts exercise the entire process of reduction and imaging. Evaluation - top level report: ---------------------------------------------------------------------- ---------------------------------------------------------------------- 1. Ensure that regression tests are being performed on a regular basis for ALMA test datasets. ---------------------------------------------------------------------- There is a web site which summarizes the regression testing of all datasets that have been evaluated for ALMA testing: https://wiki.nrao.edu/bin/view/ISD/RegressionTests Web page shows that all datasets have successfully be processed with regression test scripts for the last development cycle. Problems have been noted and addressed. Goal 1. evaluation: Adequate. ---------------------------------------------------------------------- ---------------------------------------------------------------------- 2. Ensure that the test scripts exercise the entire process of reduction and imaging. ---------------------------------------------------------------------- Goal 2. evaluation: Some minor fixes and additions needed. Test scripts exercised most of the reduction and imaging steps with no masks or interactive flagging. This is reasonable - user tests will exercise interactive software. Scripts are comprehensive, well designed and the process of running them and recording timing results is effective. There were minor errors in the scripts as noted below. These should be fixed. Missing component that should be added: - At least one script should include uv continuum subtraction rather than image plane continuum subtraction. Perhaps the TST1.1 NGC 1333 data set could do this, or G192? - Create mom0 and/or mom1 maps of line emission and perhaps create a spectrum from one cube with the imageprofilefitter (can this be done automatically?). This will both exercise some analsysis routines and provide a few simple images and spectra that can be easily compared (e.g. this would have identified the problem with the autoflag.setdata(); issue in the L02D dataset). Detailed notes on regression testing provided in a separate file.