[fitsbits] Potential new compression method for FITS tables

One additional point is that many vendor mass storage technologies can be configured with general purpose compression "built-in". Such a system can't take advantage of algorithms tailored for the particular data in question.

Rob
--

On Dec 17, 2010, at 9:27 AM, William Thompson wrote:

Preben Grosbol wrote:

As a side comments, I participated in a meeting for general archives a few
years ago. There they did not recommend to save compress data since
the effect of single bit errors is more serious than for raw data, not
justifying the gain of disk space.

Data compression addresses three issues:

1. The cost of archiving the data.
2. The rate at which data can be delivered to users
3. The ability of users to locally store data for scientific research

I mentioned before that the Solar Dynamics Observatory (SDO) mission is using
the Tiled Image Compression convention to help them manage their ~2 TB/day of
data volume. This is actually to address points 2 and 3 above than the issue of
archiving the data. The data are not archived in FITS format. Instead, the
data are stored in a database management system, with the images and associated
metadata stored separately. The FITS files are generated on-the-fly as
requested. This is how the same data can be delivered in both uncompressed and
Rice compressed formats.

I'm not sure how the images are stored within the database management system,
whether they're compressed or uncompressed, but I suspect that they're
compressed. I do know that the raw data from the telescopes undergo compression
on board the spacecraft before being telemetered down. At least for the AIA
telescope, this on-board compression is lossy--I'm not sure about HMI.

Bill Thompson