Re: 3D data model

3D spectral data model

(Some of this repeats comments made by Doug, Brian and Francois, but I thought I'd give my take.)

Igor's started an interesting discussion on 3D data in the VO. I note that all of Igor's examples are essentially spatial-spectral cubes; the question we need to address is to what extent this needs a different treatment from other 3D data:
  -spatial-temporal cubes (movies) are an obvious case;   -spatial-polarization cubes are a little further away. Then there are

• redshift surveys, which to the extent that redshift is a spectral coordinate fit in the discussion, but also are not so far away from:
• 3D spatial cubes in which two axes are RA, Dec and the third coordinate is distance, and then to:
• 3D cartesian cubes of X,Y,Z which at the moment are mostly confined to theoretical simulations (or planetary probe data?).

Even within the list of Igor's examples we can distinguish an important difference between continuous and discrete axes in the spatial and spectral directions.

• integral field spectroscopy, radio line imaging, X-ray CCD, etc are true continuous spatial-spectral cubes.
• In multi-object spectroscopy (eg fibers), the spatial axis is replaced by a 1-D list of objects - it's a 2D dataset of object-number versus spectral coordinate, even if you arrange it differently. You can also think of it as a discontinuous spatial coordinate.
• In multi-line imaging, it's the spectral coordinate that is discontinuous.

The Quantity model addresses the basic problem of multidimensional data and coordinates. The STC model and the pieces of the 1D SED model address most of the rest of the metadata we'll need and the Observation model gives the context.

So Igor and others, I'd ask you to take these existing pieces of work into account as you construct a model. You should decide which of the above cases are going to be in scope for your particular model, and how it fits in the the existing framework.

> From: Francois Bonnarel <bonnarel-at-alinda.u-strasbg.fr>
> Message-Id: <200502150959.KAA13600-at-alinda.u-strasbg.fr>
> To: dal-at-ivoa.net, dm-at-ivoa.net
> Subject: 3D data model
> Doug, Igor, Philippe, Giovanni,

I agree with the comments of Francois, and welcome the proposals that Igor gather the consensus of various 3D data users, and that he participate in the Characterization discussion.

> I am just afraid that adding new subgroups will make the whole
> Working group unmanageable and I would really like to have Jonathan's
> advice about that.

I think that it's great that the 3D folks should discuss their metadata issues specifically, but they should look at the work done in Observation, Quantity, STC and Characterization and see if they can fit within that framework. If not, that needs to feed back to those proto-standards, they will have to be changed. But you're right we shouldn't have separate standards for very similar problems, the whole point of DM is to avoid that.

Regan noted:
> The Data Format Description Language group of the
> Global Grid Forum is making progress on building
> XML-based representations of the structure of
> arbitrary binary data sets.

Yes, and we should look at this when we discuss serializing the Observation model and its 3D spatial-spectral instance. However, DFDL won't address what astronomical metadata needs to be added to the datasets to make them interoperate scientifically, which is the first thing we need to agree on. The data structure of the binary data array is pretty trivial really.

• Jonathan