index.html

THE UMASS COADD IMAGE ARCHIVE

The UMASS image archive contains 10,745,841 compressed fits images stored on 3-750 GB Reiser file system SATA hard disks. A subset of these images, the All Sky Catalog images, provides photometrically calibrated coverage of the entire sky. This subset contains images of all of the sources in the 2MASS All Sky Catalogs.

Image files are distributed on the hard disks in order of date of original observation.

ia_1 Dates before February 17, 1998
ia_2 Dates from Febrary 17, 1998 through December 14, 1999
ia_3 Dates from December 15, 1999 to end of the survey on February 15, 2001

Image files on these disks may be accessed by clicking on the links above. As the disks may be in the idle state and the directories are large it may take some time for the disk directory to appear.

The complete path from the top directory of an archive disk and the file name for images in the archive is as follows (the example file is taken from the first row of the image_inventory table):

      /DDDDDDH/sSSS/BDDDDDDHSSSFCCC.fits.(bz2 or gz)

      /970607n/s003/h970607n003M009.fits.bz2

The fields in the above path/filename are date, hemis, scan, f_type, and coadd. An explanation of the fields is given in in format_image_inventory.html.

A complete set of image files has been modified by UMASS to greatly reduce the compressed .fits file size without significant loss of photometric and astrometric information. These files are identified by the presence of the letter "M" in the f_type field of the file name. Many images are also present in the archive in an H-compressed format which does not retain all of the image information but which is more highly compressed. These files are identified by the presence of the letter "H" in the file name. H-compressed images should not be used for quantatative examination of extended or faint sources. Finally there are a number of original IPAC files in the archive for comparison purposes. These files have the number ''0" in the f_type field of the file name. Images can be located through the use of SQL queries on downloadable image archive tables.

The size of this archive and selected subsets of the archive are as follows:

All images in the archive: 2,106.1 GB
All sky catalog H-Compressed images: 250.6 GB
All sky catalog 'M' images : 1,272.6 GB
All cal scan 'M' images: 261.5 GB

Known discrepancies in the fits file headers (but not in the tables) are as follows:

Ordate is incorrect in fits file headers where date='980309' and hemis='n' and f_type='M' and scan between 116 and 132.
Type is incorrect in fits file headers where date between '991101' and '991130' and hemis='s' and band='j' and s_type='lsc'. Should be 'tst'.

Other discrepancies:

Astrometric calibration is inaccurate (up to 10" error) in all coadd images of all bands from cal scans 990514s106 and 990518s102. Source extraction tables for these scans will also contain inaccurate astrometry.

PROPERTIES OF THE IMAGES

Image generation is discussed in sec_4_3 of the explanatory supplment. A description of image anomalies is given in sec_2_4b. A description of the image .fits header is given in sec_2_4a. The following caveats apply to the coadd images:

Coadd images are derived from read2-read1 data. Pixels which are saturated in both read2 and read1 have data value of zero. Hence the images of bright stars appear to have dark cores.
Plate scale and tilt differences between the NICMOS arrays may prevent coadds from being generated from data at the edge of the arrays. Null data are represented in the H-compressed images as 0's and in the other images as a floating point NAN's. These null pixels may confuse some fits viewers if they are used to normalize the color scale of the viewed images.
The coadd image pixels are 1" x 1" while the NICMOS pixels are 2" x 2" (approx). The value of adjacent pixels in the coadd images is strongly correlated because of the repixelation of 2" raw data pixels to 1". Hence the noise properties of the images are not those of independent pixel images. An autocorrelation of a dark sky image row demonstrates this property.
Small systematic array pixel position dependent astrometric corrections have been made to 2MASS catalog entries but have not been made to the coadd image files.
H-compressed coadd images are stored as 16 bit fixed point data. Other coadd images are stored as 32 bit floating point data.
All images are 512 pixels wide (e-w). Most images are 1024 pixels long (n-s) but some are shorter at the ends of scans. Use naxis2 from the images table for the exact dimension.
A study of image photometry and astrometry has been made by John Carpenter. A study of image photometric error estimation by Tom Chester and Tom Jarrett is given in sec_6_8a of the Explanatory Supplement. Note that the proper estimate of the image noise depends on the square root of how many raw data pixels contributed to a given coadd pixel. The number of contributing raw data pixels, which may be between 0 and 7, can not be determined from the .fits files.
At the north and south edges of the coadds there is a 54 coadd-pixel overlap between a coadd and its neighbour in the same scan. In overlap regions, the coadd column in the 2MASS catalogs lists the coadd in which the source image is farthest from the edge of the same-scan coadds in which it appears.


A saturated source. j_m=4.06	Null j-band edge pixels on left.	Noise row autocorrelation.

IMAGE PHOTOMETRY

The images are photometrically calibrated. Source magnitudes may be obtained by using the following expression:

MAGNITUDE = MAGZP - 2.5 * log10(DATA - SKYVAL)

MAGZP and SKYVAL are parameters in the fits header. As the background frequently varies from one location to another in an image the use of SKYVAL provides only an approximate correction. Note that it is in general not possible to estimate the statistics correctly as there is no way of knowing from the fits files alone how many coadds contributed to a given pixel. Source code for a utility that subtracts SKYVAL from a 2MASS image may be downloaded here.

UMASS MODIFICATION OF THE IPAC FLOATING POINT IMAGE FILES

Bzip2 or gzip compression is not very effective with the 32 bit floating point image files created at IPAC. The cause of this compression problem is the presence of low order bits in the 32 bit floating point image data that are insignificant in comparison with the real noise in the image. These low order bits were generated during the process of repixelization from 2" raw data pixels to 1" coadd pixels.

The graph to the right shows how the size of a .bz2 compressed image file depends on the minimum step size in the image data. The rms photon+sky noise in this image is about 0.7 units of data. Martin Weinberg, who suggested limiting the step size (minstep) of the pixel data, chose a conservative value of 0.102062 for minstep. The UMASS modified pixel values have been obtained from the IPAC pixel values using the following code:

umass_pix = minstep * round( ipac_pix / minstep )


This graph compares data values from one row of a coadd image. The UMASS modified image is nearly identical to the ipac image (shown as a black line). The limitations of the H-compressed image are obvious.	The horizontal white line marks the row plotted in the graph to the left. It passes through a 16th magnitude 2MASS all sky catalog sky star with pts_key=886696243.

Four different versions of the coadd displayed above may be downloaded for comparison by clicking on the filename.

Original IPAC coadd: j000108n1460080.fits
UMASS modified coadd: j000108n146M080.fits
Original IPAC - UMASS modified: difference.fits
H-Compressed coadd: j000108n146H080.fits

Click here to view a similar plot made with a 6X image and here to view the image.

THE IMAGE ARCHIVE INVENTORY TABLES

The summary table, image_inventory, lists parameters of all available image files. The first row of this table is:

date | hemis | scan | coadd | band | f_name | f_size | ra | decl | s_type | naxis2 | disk | f_type | cat | qual
-----------+-------+------+-------+------+--------------------------+--------+-------------+-------------+--------+--------+------+--------+-----+------
1997-06-07 | n | 3 | 9 | h | h970607n003M009.fits.bz2 | 311521 | 183.6096124 | 35.97025637 | cal | 1024 | ia_1 | M | n | 10

The format description for the image_inventory table is given in the file format_image_inventory.html. The image_inventory table can be downloaded from the following links to compressed ascii files:

A postgres dump including schema (130 MB)

The schema alone

The table data ( | delimited with null as \N )

A smaller table, images, is derived from the image_inventory table making use of the fact that all of the column entries for the three bands of an image have identical entries (except for band, f_name, and f_size) and that the file name can be derived from the other column entries. This table is ordered and indexed for rapid positional searching. It was created from the image_inventory table using the bash script make_images_table.

date | hemis | scan | coadd | ra | decl | s_type | naxis2 | disk | f_type | cat | qual
-----------+-------+------+-------+-------------+-------------+--------+--------+------+--------+-----+------
1997-06-07 | n | 3 | 9 | 183.6096124 | 35.97025637 | cal | 1024 | ia_1 | M | n | 10

The format description for the images table is given in the file format_images.html. The images table can be downloaded from the following links to compressed ascii files:

A postgres dump including schema (58 MB)

The schema alone

The table data ( | delimited with null as \N )