October/November 2004 ALMA Offline software test NAME: Crystal Brogan Questionnaire on testing experience 1. Please list briefly your background in the following areas: A. Radio Interferometry ((sub)millimeter or centimeter) I've used VLA, VLBA, ATCA, GMRT, BIMA, SMA B. Experience with VLA and/or PdBI data Lots of experience with VLA data, no experience with PdBI. C. Astronomical Data Reduction packages: - AIPS Extensive - MIRIAD Some - MMA None - Gildas/Clic None - AIPS++ Some D. How much experience have you had with the AIPS++ software package before this test? I was part of the NRAO aips++ testing group in 2001-2003. Much of this testing involved the gui's so I am not an expert in glish. I also took part in the first external aips++ ALMA test. 2. Please identify which dataset you processed during this test: A. VLA SiO(1-0) observations of NGC 1333 3. Were you able to complete the fill, editing, and calibration of the data? If not, why? Please comment on specific steps if desired (comments can be positive or negative): A. Filling the data into AIPS++ format No problems B. Editing and visualizing your data Editing the data was very difficult. In my opinion, the data editing remains the greatest weakness in the aips++ package. This is mainly because aips++ is not efficient for flagging spectral line data. Indeed, very few data problems are easily discernible in a single channel. The signal to noise on all but the very strongest sources is just too low. The NGC1333 dataset is a good example -- a number of antennas had low amplitudes during different timeranges that were not at all obvious in a single channel. These problems were readily apparent in the averaged data. However, it was then extremely tedious to actually flag the spectral line data based on this insight. You can view the averaged data but not flag it interactively. Moreover, you cannot even get a listing (i.e. locate) from the averaged data to make an autoflagger type flagging operation. It should be noted that "clip" is VERY RARELY and adequate substitute for actually finding and removing bad data. This merely cuts off the tip of the iceberg rendering the rest of the bad data impossible to find. This problem will only get worse as the total number of channels and total bandwidth grows with EVLA and ALMA. It should be noted that in the future THERE WILL BE NO "CONTINUUM" DATA. It will all be spectral line and the efficient editing of such data should be a priority. It also seems likely that on-the-fly averaging of data from these instruments will take too long, even if "averaged flagging" becomes possible. The ability to copy tables -- both flags and calibration from one dataset to another is a critical feature missing from aips++. Interactive flagging of the calibration solutions is also desirable. C. Atmospheric phase corrections (PdBI data only) N/A D. Gain (phase and amplitude) calibration Using cookbook this was relatively straightforward E. Absolute flux calibration Using cookbook this was relatively straightforward F. Bandpass calibration Using cookbook this was relatively straightforward 4. Were you able to subtract continuum in the uv-plane (if applicable)? If not, why? N/A 5. Were you able to split out the calibrated data (if desired)? If not, why? Yes. Although I found the defaults of split to be undesirable. The default should select everything not explicitly selected against. i.e. if you want all spectral windows you should have to include that in the data selection. 6. Were you able to image the data? If not, why? Please identify any problems you had during imaging. The mosaicking example provided in the cookbook combined with the information given in the "detailed data description" was sufficient to make a reasonable image. I found the cleaning difficult to interpret. (1) it would be better to completely clean one channel before going on to the next one. The current method of cleaning one cycle of each channel and then looping back makes it very difficult to tell if the clean is converging. (2) The logger output from the cleaning is difficult to decipher. It would be nice to get something that more closely resembles the output from aips. i.e. the peak residual, flux cleaned this cycle, and total cumulative flux. The main piece of information that you can glean before you start cleaning, is the total flux on your shortest spacing from a UV plot. The current output doesn't help you asses how close you are getting to this number. I tried using the displayprogress=T option -- but the strange looping over channels made this difficult to understand. 7. Were you able to analyze the images adequately to determine if the results you obtained were scientifically reasonable (e.g. display the image, calculate RMS and peak, make a moment map or take a spectrum)? If not, why? Yes. 8. Please summarize the final results of your image(s): - RMS: - Peak Flux Density: I made naturally weighted images with a beam of 1.94" x 1.71" The peak of the moment 0 map is 0.66 Jy*km/s and the rms noise is ~25 mJy*km/s (near center). The peak intensity in the image cube is 70 mJy/beam. The rms noise in a single pixel is about ~2 mJy/beam. The rms noise in the channel images is ~20 mJy/beam near the center. 9. Did you have adequate support during your test? If you contacted the AIPS++ groups for questions or to fix a bug, please comment on the interaction and whether it was helpful. Yes the support was very good. All of my questions were answered very quickly. 10.Was AIPS++ easy to install? If not, why? I had trouble with the install. This seemed to be due to the fact that I had a previous version of aips++ that was not completely overwritten so that the new version was trying to look at some of the old libraries (using the casa_install script). In future, specific instructions need to be given for the case of an update. The information provided on the web (outside of ALMA test) is also extremely vague on what you should do to update an existing installation -- it tells you where to get the software but nothing on how to install it. After I reported having difficulties the support was very good. 11.The Synthesis Reduction Cookbook you used for this test is the first version of a comprehensive cookbook for ALMA users. Please evaluate the organization, content, and presentation of the cookbook. It is meant to be the first documentation users will see when they want to reduce ALMA data, it provides background on the code capabilities, and extensive examples. The on-line documentation provides more details and code descriptions. With this in mind, please answer the questions below. If you have detailed comments, please attach them to the end of this questionnaire. - Was the documentation adequate for you to complete your test? Yes. - Was the cookbook good? The cookbook has dramatically improved since I last saw it in February. - Do you have any suggestions for how to improve the cookbook? The cookbook has begun to give advice on how and why to do things rather than just explaining how to use a tool and should continue this trend. For example: Quack. Currently the cookbook says that Quack is a "special option for VLA datasets" but is a bit mysterious. An explanation that describes that: VLA data often has low amplitudes for the first integration of each scan (this happens because the system has not yet completely "settled down" after a change in source or observing parameters). A good way to determine if this is affecting your data is to look at an average of all the channels in the viewer, looking for low amplitudes at the beginning of each scan. If you see this problem you can use quack to remove data from the beginning of each scan -- typically the first integration of each scan. Note that the current instructions "Flag the first 3 seconds of each 15 s integration" doesn't make sense. An integration is not divisible into smaller chunks -- its been integrated. I guess this means scan. BUT quack needs to detect scans for itself -- rarely are all the scans the same length so you can't specify this. - Was the on-line documentation good: * User Reference Manual? * Supporting documentation? 12.Roughly how much time did you take to perform the following steps: - Installing aips++: 2 hours - Fill, editing, & calibration: 5 hours (mostly flagging) - Continuum imaging: N/A - Spectral line imaging: 2 hour (trying different clean options) - Analysis: 1 hour - Filling out this questionnaire: 1 hour - Evaluating and grading the scientific requirements: 1 hour - Total time: 12 hours 13.Please rate your overall testing experience: - good 14.Was the test well designed and executed by those in the ALMA offline subsystem (e.g. the subsystem scientist and the Offline subsystem group). If not, can you provide any suggestions for improving the next test? It is clear that a great deal of effort went into designing and creating the documentation for the test. Everyone did a great job. 15.Do you have any additional comments that may help improve test of the offline software in the future? It would be beneficial if the testers were to get together for a couple of days of intensive testing/feedback with the developers.