April 2005 ALMA Offline software test

NAME: Andrew Baker

Questionnaire on testing experience

1. Please list briefly your background in the following areas:

  A. Radio Interferometry ((sub)millimeter or centimeter)

Extensive experience with millimeter interferometry (OVRO and PdBI) of 
extragalactic sources, although relatively little of this has involved 
mosaicing and none has involved merging with single-dish data.

  B. Experience with VLA and/or PdBI data

See above.  No experience with VLA data.

  C. Astronomical Data Reduction packages:
     - AIPS

I use AIPS (Classic) for nearly all mapping.

     - MIRIAD

I use MIRIAD for my small number of mapping projects that involve mosaics.

     - MMA

I have extensive experience calibrating OVRO data with MMA.

     - Gildas/Clic

I have extensive experience calibrating PdBI data with CLIC, although I 
generally map these datasets with AIPS rather than GILDAS.

     - AIPS++

My only use of AIPS++ was for the AIPS++ Reuse Analysis Test in 2003.

  D. How much experience have you had with the AIPS++ software
     package before this test?

See above.

2. Please identify which dataset you processed during this test:

  B. BIMA CO(1-0) observations of NGC 4826

3. Were you able to combine the single dish and interferometric data
  using feather and deconvolution techniques?  If not, why?   Please
  comment on specific steps if desired (comments can be positive or
  negative, you may not have tried all steps):

  A. Feather images or image cubes provided by Offline subsystem

I was able to combine 12m and BIMA data using this approach.  The core set 
of commands I used was

#################################################################
imb := imager('n4826_both.ms');  # dummy MS to get imager to work
imb.setvp(dovp=T);             
imb.feather(image='feathered.image',highres='n4826_mom0.im',
   lowres='n4826_12mmom0.im');
imb.done();
#################################################################
which produced the following output in the logger window:

-----------------------------------------------------------------
Starting imager::feather
Feathering together high and low resolution images
Cannot regrid axis 4 because it is of unit length - removing from list
Using primary beam to determine weighting
Determining scaling from  SD restoring Beam
Applying additional scaling for ratio of the volumes of the high to the low 
  resolution images : 0.0153742
Finished imager::feather
       0.77 real        0.44 user        0.04 system
-----------------------------------------------------------------

To evaluate the validity of these results, I inspected and determined 
statistics for both "n4826_mom0.im" (BIMA-only) and "feathered.image"
(combined) moment maps.  Qualitatively, the addition of the 12m data 
appeared to have reduced the depth of the negative bowl around the 
galaxy that can be seen in the BIMA-only map.  Quantitatively, I had to 
integrate up the emission myself in order to assess flux recovery for the
galaxy.  Starting from the "sum" reported for a polygonal region by the 
image statistics routines, I divided by the number of pixels per beam 

  (pi/4ln2)(bmaj)(bmin)/(pix)^2

to obtain Jy km/s.  For the 12m-only image, this meant dividing by 10.579; 
for the BIMA-only and feathered images, this meant dividing by 52.688.  The 
results were as follows:

For the 12m-only image:
 - Galaxy gives "sum" between 1.581e4 - 1.760e4 (depending on exact choice 
   of polygonal region) <--> 1494 - 1664 Jy km/s total line flux.  Helfer et 
   al. (2003) quote 1845 +/- 217 Jy km/s for their reduction of the 12m data,
   i.e., marginally consistent with my number.

For the BIMA-only image:
 - Galaxy gives "sum" of 1.064e5 <--> 2019 Jy km/s total line flux.  This is 
   puzzlingly higher than the global flux from the 12m data, even using the 
   higher value of Helfer et al. (2003) for the latter and allowing for the 
   fact that the BIMA-only (moment) map includes no negative pixels.  I 
   suspect the problem lies in the calibration of the 12m data rather than in 
   the mapping of the BIMA data or anything AIPS++ is doing: Figure 54 of 
   Helfer et al. (2003) shows that the implied flux recovery in regions of 
   high S/N (i.e., dominating the total integrated flux) is > 100%.  It is also
   possible that I have mistakenly included too much extended "emission", thus 
   inflating my total in proportion to the larger number of (positive) pixels.
 - Rough off-source annulus (excluding bowl) has "mean" = 11.64 counts and 
   "rms" = 12.91 counts.

For the feathered image:
 - Galaxy gives "sum" of 7.554e4 <--> 1433 Jy km/s total line flux.
 - Rough off-source annulus has "mean" = 2.125 counts and "rms" = 5.836 counts.

It would seem that feathering has (a) reduced the total galaxy flux (b) 
reduced the mean off-source flux (c) reduced the off-source rms.  (a) is 
consistent in sense with BIMA-only map's having a larger line flux than the 
12m-only map, although it is troubling that the 12m data should have pulled 
the total flux in the feathered image *below* the total flux seen in the 
12m-only map.  (b) and (c) seem sensible if short-spacing correction has 
essentially smoothed the overall "background" towards a more constant and 
(thanks to the negative pixels in the 12m data) lower value.

  B. Feather single dish image provided and interferometer image that
     you created from the dataset provided.

I did not attempt this version of feathering, although in hindsight it would 
have been a good way to check how seriously use of an apparently blanked BIMA 
moment map in (A) might have thrown off the evaluation of global line fluxes.

  C. Deconvolve the single dish and interferometer data using the
     single dish image to create an input model.

I was able to combine 12m and BIMA data using this approach.  The core set 
of commands I used was 

##########################################################################
imff := imagefromfits(outfile='n4826_12mchan.im',
infile='NGC4826.12motf.chan.fits');
imff.done();
im := imager(filename='n4826_both.ms');
im.setdata(mode='channel',fieldid=[1:7],spwid=[1:3],nchan=[64,64,64],
   start=[1,1,1],step=[1,1,1]);
im.setimage(nx=256,ny=256,cellx='1.0arcsec',celly='1.0arcsec',stokes='I',
   mode='channel',nchan=40,start=27,step=4,fieldid=1,spwid=[1:3]);
im.setoptions(ftmachine='mosaic');
im.setvp(dovp=T);
im.makemodelfromsd(sdimage='n4826_12mchan.im',modelimage='n4826_joint1',
   maskimage='n4826.mask');
im.clean(algorithm='mfclark',model='n4826_joint1',gain=0.2,niter=500);
im.done();
imm := image(infile='n4826_joint1.restored');
imm_mom0 := imm.moments(outfile='n4826_joint1.mom0',moments=0,axis=4,
   mask='indexin(4,[10:32])',includepix=[0.0,100]);
imm.done();
imm_mom0.done();
##########################################################################

I used a wider range of channels than recommended by the instructions in 
order to make sure that there were sufficiently many line-free channels that
I could evaluate the noise (as a result, channel 40 in my output contained 
no data, but this did not cause problems with the imager or the viewer).  The 
final logger window output stated that "Clean did not reach threshold", but 
since the channel maps looked good visually I was not motivated to explore 
parameter space for mfclark.

There were two aspects of the test about which I was uncertain at the outset.  
First, the single-dish data cube did not have units, so I was not sure whether 
it would be necessary to invoke "im.setsdoptions" before cleaning.  I found 
that the original 12m data cube on NED has BUNIT = Jy/beam, so no conversion 
was necessary (I suspect that BUNIT was somehow dropped when the fourth 
axis for Stokes parameter was introduced to the FITS file for the purposes 
of this test).  The second point that confused me was history information 
in n4826_both.ms indicating an unexpected previous round of cleaning.  I 
simply proceeded on the assumption that any previous cleaning had not 
affected the measurement set itself.

In the output data cube, line-free channels 1-5 and 35-39 gave an average 
rms of 87.0 mJy/beam, somewhat lower than the 90.5 mJy/beam reported in 
the instructions for the testers.  The mean intensity in these channels is 
slightly negative.  In my channel 12 (corresponding to channel 7 in the 
instructions), the statistics are
 - peak = 1083 mJy/beam (vs. 956 mJy/beam in instructions)
 - flux = 6.04 Jy (vs. 6.91 Jy in instructions)
 - rms off-source = 107.5 mJy/beam (vs. 90.5 mJy/beam in instructions)
 - mean off-source level = -0.7 mJy/beam (vs. 9.2 mJy/beam in instructions)
The slightly lower total flux and higher rms in my map relative to the 
project's map would be expected for a clean that was not constrained with a 
clean box and did not go as deep as possible (consistent with the message in 
the logger).  I cannot explain why the mean off-source level is positive in 
the project's map and negative in mine, although my negative level is 
consistent with I see in the line-free channels.

In the moment map I derived from this cube, the galaxy gives a "sum" of 
1.177e5 (7.792e4 for only the most credible, highest surface brightness 
regions), corresponding to a line flux of 2234 (1479) Jy km/s.  These values 
compare favorably with the 2019 Jy km/s and 1433 Jy km/s estimated (above) for 
the BIMA-only moment map and the feathered map.  (Both of the latter maps have 
the advantage that-- because their BIMA data were much more carefully cleaned 
by previous testers-- it is easier to evaluate which "emission" is real.)  
The moment map I derived from this cube has off-source mean = 16.22 counts 
and rms = 17.31 counts, in better agreement with the project's BIMA-only 
moment map than with the feathered image I produced above.

4. Were you able to analyze the images adequately to determine if the
  results you obtained were scientifically reasonable (e.g. display
  the image, calculate RMS and peak, make a moment map or take a
  spectrum)?  If not, why?

Due to a typo in Section 7.9 of the cookbook, my initial attempts to make 
a moment map for test (C)-- for the purposes of comparison with the result 
of test (A)-- produced a slew of error messages.  I was able to figure out
the correct syntax using the Reference Manual (although note that this is 
not the first web page that comes up when one Googles "aips++" and "moments"!).

I would also say that a major source of inconvenience was the requirement 
that I integrate up the flux in a given region myself (the image statistics 
routines appear not to return the total flux unless the native units are 
Jy/beam; the units of these moment maps are Jy/beam km/s, which in principle
should be just as tractable).

5. Please summarize the final results of your image(s):
  - RMS: 
  - Peak and Total Flux Density: 

See above.

6. Did you have adequate support during your test?  If you contacted
  the AIPS++ groups for questions or to fix a bug, please comment on
  the interaction and whether it was helpful.

Somewhat embarrassingly, I found it difficult to manipulate the permissions 
on the VLA Orion dataset so that I could delete it from my hard disk.  Joe 
McMullin kindly sorted this out.

7. Was AIPS++ easy to install?  If not, why?

Yes.  I used the load-casa script; once I found a local computer running 
Mandrake 10, I only needed to change line 75 from 
	} elsif ( $rhver =~ m/^Mandrake Linux/  ) {
to
	} elsif ( $rhver =~ m/^Mandrakelinux/  ) {
to get the installation to proceed smoothly.

In relative terms, this was an order of magnitude easier than the installation
process I undertook/underwent in 2003.

8. The Synthesis Reduction Cookbook you used for this test is the
  second version of a comprehensive cookbook for ALMA users.  Please
  evaluate the organization, content, and presentation of the
  cookbook.  It is meant to be the first documentation users will see
  when they want to reduce ALMA data, it provides background on the
  code capabilities, and extensive examples.  The on-line
  documentation provides more details and code descriptions.  With
  this in mind, please answer the questions below.  If you have
  detailed comments, please attach them to the end of this
  questionnaire.

  - Was the documentation adequate for you to complete your test?

Yes.  The Cookbook was generally sufficient by itself, although the User 
Reference Manual was crucial for fixing the "moments" typo noted above.

  - Do you have any suggestions for how to improve the cookbook?

See extended list of comments below.

  - Was the on-line documentation helpful:
     * User Reference Manual?

Yes (see above).  I did waste a certain about of time in finding my way to 
the right page, however, because my first attempt to track down the correct
syntax for "moments" involved Googling for "aips++" and "moments".  This led
me to a page that appears to be outdated, and to some fruitless experimentation
with momentsgui (which I suspect has now been deprecated).  At a very low 
priority, it might be worthwhile retiring outdated web pages of this sort.

     * Supporting documentation?

I did not use any supporting documentation.

9. Roughly how much time did you take to perform the following steps:

  - Installing aips++:

1 hour spread over two days (mainly a matter of identifying a local
computer with the right version of Mandrake Linux).

I also spent ca. 5 hours to read (admittedly slowly) through documentation, 
notably the Cookbook.  I suspect this was a time-efficient approach.

  - Imaging:
  - Analysis:

Together, I would estimate I spent about 15 hours on both of these (it is 
difficult to break them out as separate items, because image analysis in 
some cases motivated another iteration of imaging to fix problems).

  - Filling out this questionnaire:

4 hours by the time the list of suggestions for the Cookbook has been 
typed in.

  - Evaluating and grading the scientific requirements:

1 hours.

  - Total time: 26 hours

I was somewhat time-limited for this experiment due to a heavy travel schedule
during the first part of April.  If I had had additional time to spend, my 
priorities would have been 
 (1) investigate the origin of the variation in total flux
 (2) investigate sensitivity of method (C) to changes in mapping parameters
 (3) test method (B)

10.Please rate your overall testing experience:

  - good

11.Was the test well designed and executed by those in the ALMA
  offline subsystem (e.g. the subsystem scientist and the Offline
  subsystem group).  If not, can you provide any suggestions for
  improving the next test?

This is not really a question I can answer-- it falls to the subsystem 
scientist to evaluate how useful the input from the external testers is in 
addressing the needs of the project.  From my point of view, at least, the 
focus on the algorithms forced by use of Glish scripting (rather than the 
extensive de facto testing of the GUI that caused numerous problems back in 
2003) seemed sensible.  Also, with the benefit of hindsight, the choice of 
galaxy was not optimal for this particular test.  While I realize NGC4826 was 
a natural selection based on its use in a previous test, the comparison of 
single-dish and interferometric data would probably be easier to evaluate for 
a target where the interferometer clearly recovers < 100% of the flux (in 
contrast to the perhaps > 100% that was the case here).

12.Do you have any additional comments that may help improve test of
  the offline software in the future?

I would suggest that a rough outline of the overall strategy for testing be 
provided to future testers, both for background and for motivation.  The 
material provided with this test, for example, makes references to "regression 
testing", "pipeline heuristics", "framework conversion", and "TST1, 1.1, and 
2.0" that I personally find rather opaque.  It would be nice to know how my 
efforts here fit into the larger picture.

Cookbook minutiae
=================

The following comments pertain to the parts of the cookbook that I read-- 
chapters 1, 3, 5, 6, and 7, and sections 2.3 and 4.6-- are are listed by 
page number.

(15)

It might be good to specify somewhere in here for the complete novice that 
executing commands-- even as simple as opening a viewer-- will produce T or F 
in the Glish command line window.

"is described in Section 4." -> "are described in Section 4."

(19)

I would suggest consistent use of nomenclature on whether the items listed 
in Table 1.2 are "components" or "columns" of the MAIN table, here and in the 
text.

(22)

It would be appropriate to mention where PdBI data fall in terms of SNR (I 
would assume similar to BIMA at 3mm, at the very least).

(30)

"phase calibrater" -> "phase calibrator" four times on this page (I know 
it's just a typo, but it really grates)

(31)

In the "Target source example" towards the bottom of the page, the 
comment "In this case, for the calibrator, we need only..." is clearly 
inappropriate.

(32)

mset.concatenate: isn't a frequency tolerance of 10 MHz much too large?

(33)

Path name should be installed_directory/lib/casa/data/demo/NGC5921.fits
(i.e., without the extra "casa").

(41)

At bottom: "a pseudo-bad baseline solution is made..." is unclear to me.

(43)

The comment beside "dv.gui();" is impossible to parse.

In example 2 for clipping, it would be useful to explain in the text 
what XX refers to.

(70)

"concatinate" -> "concatenate" (five times on this page)

(82)

It would be useful to mention that the polygon region button (with an "R") is 
not the same as the polyline button (I used the latter at first and spent 
several minutes trying to figure out why clicking then produced no image 
statistics).

(83)

This script refers inconsistently to both "mask.image" and "mymask.image".

(84)

One question: is there a way to extend a *polygonal* mask over a range in 
channels?  This is something I would love to be able to do...

(86)

"See the example (5.2.": appears to be missing the word "Figure".

(95)

There are semicolons missing after several of the commands in this script--
not sure whether this is a typo that needs to be corrected, or is actually
permitted in Glish.

(101)

In comment for "im.setvp": "make image" -> "mask image".  Also, there are 
again many statements without terminal semicolons.

(102)

In "decon.makegaussian": why is bpa specified for a beam with bmaj = bmin?

(106)

"The DisplayData menu is extended in the figure...": this is not obvious to 
me.

(111)

It would be useful to explain what it meant by "emit an event" the first 
time it is mentioned (here, if it was not mentioned earlier in one of the 
chapters I didn't read).

(125)

"Marker type": I am a bit confused that this can be selected, given the 
comment on p123 that "this [square] is the only marker shape available".

(146)

"im_mom0:=moments" -> "im_mom0:=im.moments"