Mark Taylor wrote:
> On Wed, 14 Apr 2004, martin hill wrote:
>>Nor do we save any effort in using VOTable to represent SEDs than in writing a
>>new SED schema (except maybe for a few people to learn schemas - but this is
no
>>harder than learning VOTable and it's a good skill to have!).
I suspect I haven't understood this right. You can use schemas to describe tabular data very easily - The VOTable schema is an example of this. And I don't see too much wrong with using VOTable (or even, <spit>, CSV!) for representing tables. My issue is that using a generic tabular form (of any sort) is not the right way to communicate between web services; we should be explicitly 'typing' the data and its natural structure, which is *not* purely tabular. A fair amount of astronomy data is in *relational* tables because that has been the only suitable storage mechanism available, and VOTable does *not* represent relational tables - at least, not without an extra VO-specific layer to resolve the relationships. It's just as easy to write a new schema as it is to design and agree a new VOTable flavour, and you get many advantages to doing so.
One of the advantages of using tables is, as you say, you can do statistical analysis/plots/etc. Good stuff. Let's use VOTable for that - as well as CSV and any other commonly used formats used - as inputs to these tools. Transformation sheets that take, eg, SEDs and create CSVs to be imported into Excel (or your favourite non-microsoft spreadsheet) would be very useful.
> An important related point is that VOTable provides ways of storing and
> transmitting very large data sets for which raw XML is not well suited
> in terms of bandwidth and/or processing efficiency.
Of course, VOTable/XML *is* XML... Any difference in size between VOTable/XML and XML with longer element names is irrelevent; if size is a problem we should not be using Votable/XML in the first place! And there are many unresolved issues with using VOTable/FITS; mostly based around ensuring that the VOTable is correct for the wrapped FITS file. Using this mechanism you can submit FITS catalogues to SExtractor... We are blinding our web services to each others' capabilities, something we want to avoid as part of our 'metadata-rich' VO. There are other possible solutions to using binary XML-like structures without having to go through ASCII.
> If a SED is passed around as a VOTable then the application programmer
> can use an existing VOTable processing library to turn it into something
> which looks like a column of numbers for further processing without
> further ado, or the astronomer can use a VOTable-aware tool to do
> something tabularly-generic with it such as plot.
But we don't want to have to build special VOTable processing libraries in all the various languages that astronomers use. We should be using standard XML tools (below) to move data from one existing tool to another.
> If it's passed
> around using a SED-specific schema this functionality has to be
> rewritten from scratch for SEDs (and the same applies for any other
> specific formats that we want to define new schemas for).
>
With VOTable the same thing is going to have be done 'from scratch' for each flavour of VOTable - ie converting from an SED VOTable to SED visualisation tools. This is 'hidden' because we can lump it all under 'developing a VOTable tool'.
Our data formats should only be storage/transportation mechanisms; we should not be trying to build our own toolsets around them. There are plenty of existing tools in the astronomical world for visualising and analysing (and producing) data. We need to be able to transform the outputs of one into the inputs of another. VOTable provides an illusionary 'many to one to many' interface format, but the 'one' in the middle is of many flavours, and yet the flavours are not explicitly specified in any standard way. Further, we are finding we have to build a huge *code*set around it, which needs to be installed and run somewhere (ie a new toolmaker is going to have to do so).
Instead we should have XML message formats that correctly represent the data, and transformation sheets that understands the source data type so it can produce the right 'native' format for the target tool. XML transformation sheets are an industry-standard conversion mechanism available on many different platforms, so people who want to introduce new tools to the VO can use them rather than find and learn some VO-specific tools.
Building specialised schemas and associated sheets is *not that hard*! With an SED-specific schema, we add a transformation sheet to create (say) Votables and CSVs for things like spreadsheets, and other transformers as required for existing SED visualisation tools. This means that if you run an SED visualisation tool *it will only accept SED data*. It means you can construct the right input document, with help from XML-building tools. You can't throw a sky catalogue at it, and you will know before you try.
>>I have been assuming that VOTable is for representing 2d tables (I realise it >>can now hold tables that include tables).
In summary: we need VOTable (for many of the reasons discussed). But we should not be trying to use it as the sole representation of all astronomical data for either transport or store. It is not designed for it, we don't have the tools for it, we shouldn't be spending effort making tools for it, and we will be shooting ourselves in the foot trying to do so.
MC
-- Martin Hill Software Engineer AstroGrid @ ROE Tel: +44 7901 55 24 66 www.astrogrid.orgReceived on 2004-04-16Z13:16:02