Dear Claudio
>From the VOTable spec, in particular section 2.2, I gather that they indeed
already included
support for multi-dimensional arrays. This seems then indeed the natural way
to support at least
uniform grids coming from simulations as well. Some comments:
>From their example I gather that arraysize="41x41x41x3" means "three data
cubes of dimensions 41x41x41",
not "one 3D-vector valued datacube of dimensions 41x41x41".
"41x41x41" would mean "41 2D datafields of dimension 41x41". I think that
therefore a 3D vector field
could/has to be encoded as (for example)
<?xml version="1.0"?>
<VOTABLE xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd">
<RESOURCE name=myVectorField>
<TABLE name="VelocityField" ID="Vel">
<FIELD name="vx" ID="vx1" ucd="phys.veloc;pos.cartesian.x"
datatype="float"
arraysize="41x41x41x1" unit="km/s" />
<FIELD name="vy" ID="vy1" ucd="phys.veloc;pos.cartesian.y"
datatype="float"
arraysize="41x41x41x1" unit="km/s" />
<FIELD name="vz" ID="vz1" ucd="phys.veloc;pos.cartesian.z"
datatype="float"
arraysize="41x41x41x1" unit="km/s" />
<DATA>
<BINARY>
<STREAM href="file:///scratch/myhome/test.bin"/>
</BINARY>
</DATA>
This makes the content of the individual field components more explicit.
Each gets it own UCD for example.
I have removed the rank attribute for the moment.
There is no way yet to specify the spatial coordinates of the grid cells.
For a grid one can specify
the spatial coordinates in general in a shorthand way, for example using a
set of standard parameters
as in the FITS array keywords (see
http://fits.gsfc.nasa.gov/standard21b/fits_standard.pdf 5.4.2.5),
CRPIXn, CDELTn etc. I think we need to specify something like that here as
well, it is definitly more
efficient than having separate cubes with the coordinates.
Luckily in general our coordinate system will not require the full WCS like
formalism in general.
Still I also liked your original approach, which, as I commented in my earlier reply, seemed to lead to a kind of equivalence in XML of the FITS image specification. I wonder whether the VOTable group has considered to put image data in an XML form of FITS just as they did for the FITS binary table. I'll pose the question on their mailing list. Tough we can use the multi-dimensional array, it seems not as natural.
Then, though it is possible to use this same formalism for particle data as
well, I think there the tabular approach is more natural in many
circumstances. In particular in the work that I have been doing with
databases,
the natural representation of more complex individual objects is as a table,
with all the properties, including
now the positions, in a row. The way to store such tabular datasets in
binary form is specified exactly in the
the existing VOTable spec, in section 5.3. An equivalent C-struct oriented
format in binary files is what I have encountered consistently for more
complex objects coming for example from the postprocessing of cosmological
simulations at the MPA in Garching.
But you're right that many people also store particle data in individual
arrays for each particle property.
That is more naturally mapped in the sense of your rank 1/2 examples. Making
the same adjustment as above for
the datacubes I would propose to allow also something as in the following
example:
<?xml version="1.0"?>
<VOTABLE xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd">
<RESOURCE name=myParticles>
<TABLE name="Particles" ID="NBody">
<FIELD name="x" ID="x1" ucd="pos.cartesian;pos.cartesian.x"
datatype="float"
arraysize="100000x1" unit="Mpc" />
<FIELD name="y" ID="y1" ucd="pos.cartesian;pos.cartesian.y"
datatype="float"
arraysize="100000x1" unit="Mpc" />
<FIELD name="z" ID="z1" ucd="pos.cartesian;pos.cartesian.z"
datatype="float"
arraysize="100000x1" unit="Mpc" />
<FIELD name="vx" ID="vx1" ucd="phys.veloc;pos.cartesian.x"
datatype="float"
arraysize="100000x1" unit="km/s" />
<FIELD name="vy" ID="vy1" ucd="phys.veloc;pos.cartesian.y"
datatype="float"
arraysize="100000x1" unit="km/s" />
<FIELD name="vz" ID="vz1" ucd="phys.veloc;pos.cartesian.z"
datatype="float"
arraysize="100000x1" unit="km/s" /> <DATA>
<BINARY>
<STREAM href="file:///scratch/myhome/test.bin"/>
</BINARY>
</DATA>
I would advocate supporting both representations for particle data, tabular
and (1D) array.
In the latter case we still need something to distinguish between particle
data and image data.
Your rank basically does that, just the name might be unfortunate. We might
want to be more explicit
about the kind of data that is stored, an attribute with values MESH, N_BODY
maybe ?
In your example you use an HDF5 binary file. VOTable does not support that,
though it does support FITS,
I suppose as BINARY table (see VOTable spec section 5.2). Is there a natural
mapping from VOTable key words
to HDF metadata structures ? Or shall we first concetrate on the binary
serialisations specified in VOTable ?
Cheers
Gerard
Ciao Gerard,
in the meantime I had thought a litlle about possible formats for the
VOTable. In fact I come to the conclusion that there is little new to add to
the already existing VOTable specification, both for grids and for
particles.
The only parameters that I think we have to add is the "rank" parameter (it
may already exist, but I could have missed it).
Rank is the only parameter that makes grids different from particles,
scalars from vectors. For the rest, particles are completely the same as
grids. NO different approaches are needed.
In practice:
Rank = 1 --> scalar on particles (a sequence of scalar values associated to
the N particles, one info per particle, N values)
Rank = 2 --> vector on particles (sets of three values per particles, Nx3)
Rank = 3 --> scalar on grids (one value per grid point, NxNxN - assuming a
cubic grid for simplicity)
Rank = 4 --> vector on grids (set of three values per grid point, NxNxNx3)
At the moment let's consider only 3D simulations.
>From the example belowe you can notice that "rank" is a ridondant info that
can be obtained also directly from the "arraysize" parameter. But you must
go through a parsing and therefore it could be useful to keep it highlighted
in a specific parameter.
In this version, more variables, of different sizes, can be stored in the SAME file. The file could have different formats (fits, hdf... that must be specified properly). I assume, for the moment, a raw binary file, where variables are written one after the other (the standard table structure in row and colums is not efficient or even possible). The entry point for each variable can be easily calculated using the "arraysize" and "datatype" parameters. Furthermore, the order in which they are sotred must be specified. And this could be the order in which the FIELDs are stored in the VOTable.
Example: our data file contains a scalar field on a mesh, a vector field on a mesh, a scalar and a vector fields on particles:
<?xml version="1.0"?>
<VOTABLE xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd
">
<RESOURCE name=myTestResource> <TABLE name="BmTemperature" ID="MyTestTable" > <FIELD name="BmTemperature" ID="myTestObject1" ucd="" datatype="float" rank="3" arraysize="41x41x41" unit="K" /> <FIELD name="BmVelocity" ID="myTestObject2" ucd="" datatype="float" rank="4" arraysize="41x41x41x3" unit="km/sec" /> <FIELD name="ParticlPos" ID="myTestObject3" ucd="" datatype="float" rank="2" arraysize="10000x3" unit="Mpc" /> <FIELD name="PartDens" ID="myTestObject4" ucd=""datatype="float" rank="1" arraysize="10000" unit="g/cm3" />
<DATA><BINARY> <STREAM href="file:///scratch/myhome/test.h5"/> </BINARY></DATA>
Let me know your opinion.
Claudio
Gerard wrote:
Hi Claudio
Sorry for the late reply to this email. I'm Cc-ing the theory group as well
I gather you are thinking of grid simulation data here, so this mail does not apply to N-body. Anyway, for that I think we can use the VOTable spec as it stands, in particular section 5.3 dealing with binary serialisation (see http://www.ivoa.net/Documents/REC/VOTable/VOTable-20040811.pdf ).
In the case you address, would it make sense to try to mimick FITS in the naming of key words, so use NAXIS for rank, and NAXIS1 for size0, NAXIS2 for size1 etc for the dimensions ? If I am not mistaken VOTable itself is based on the FITS binary table spec, so your proposal might be seen as a translation of a FITS datacube (IMAGE). Did we actually not think about using FITS as is for (uniform) grid simulations ? In that case your proposal could also be used I guess, where iso STREAM we'd have FITS as in standard VOTable usage (though I don't know whether votable presumes that the FITS file contains a table).
I am not sure whether FITS images/datacubes allow multiple values per cell
(i.,e. have an array size), but don't think so. Otherwise we could probbaly
generalise in that direction.
Do you propose to follow the VOTable/FITS directions on little-vs big-endian
?
Cheers
Gerard
-----Original Message-----
From: Claudio Gheller [mailto:c.gheller-at-cineca.it]
Sent: Thursday, July 20, 2006 12:37 PM
To: Gerard Lemson; Ugo Becciani; Alessandro Costa; Marco Comparato; R.
Smareglia
Subject: VOTable for simulations
Dear friends,
I have tried to figure out the structure of a VOTable for simulated
data. In the following the result.
I made the following assumptions:
At this point I made the Snap program create binary files (at present
still HDF5, but just for backward compatibility) and associated XMLs.
For example:
test.h5 ----> snapped data
test.h5.xml ----> associated VOTable:
<?xml version="1.0"?>
<VOTABLE xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd
">
<RESOURCE name=myTestResource>
<TABLE name="BmTemperature" ID="MyTestTable" >
<FIELD name="BmTemperature" ID="myTestObject"
ucd="" datatype="float" arraysize="41x41x41" unit="Kelvin" />
<PARAM name="rank" datatype="int" value="3"/>
<PARAM name="size0" datatype="long" value="41"/>
<PARAM name="size1" datatype="long" value="41"/>
<PARAM name="size2" datatype="long" value="41"/>
<DATA><BINARY>
<STREAM href="file:///scratch/myhome/test.h5"/>
</BINARY></DATA>
</TABLE>
</RESOURCE>
Notice that the rank and size of the dataset is expressed in the arraysize keyword of FIELD. It is also written in the 4 PARAM fields. This is just to avoid the parsing of the string to get the basic info of rank and size and to have them directly as numbers (with their precise type). At present there are no UCD and no reference to the SNAP protocol, since both are not yet defined. I'm working on the latter...
This is the very first attempt!!! Let me know all your comments. Claudio
--
------------------------------------
Dr. Claudio Gheller, Ph.D.
High Performance System Division
CINECA - Bologna - Italy
Tel. +39-051-6171560
Fax. +39-051-6137273
------------------------------------
--
------------------------------------
Dr. Claudio Gheller, Ph.D.
High Performance System Division
CINECA - Bologna - Italy
Tel. +39-051-6171560
Fax. +39-051-6137273
------------------------------------
Received on 2006-08-31Z18:59:51