On Wed, 21 Dec 2005, Jim Gray wrote:
> There is also the cross match that Tanu & Maria implemented in SQL.
> That may be the easiest way to do the implementation if the nodes all
> have a SQL backend.
I'd like to point out that if the data are already in a relational DBMS then by far the simplest way to do the cross-match, and in many cases also the fastest, is to use R-tree indexing and a spatial join. I think the first astronomical use of this was by Andrea Baruffolo (see http://monet.ncsa.uiuc.edu/adass98/Proceedings/baruffoloa1/ ) but it has also been extensively tested here and documented on the AstroGrid wiki, see: http://wiki.astrogrid.org/bin/view/Astrogrid/DataDocs
Support for spatial indexing is now included in or readily available for DB2, Oracle, Informix, Sybase, MySQL, and Postgres, i.e. just about all the DBMS widely used in astronomy (with perhaps just one exception, which Jim can tell you about :-).
> But, getting objects into a node dominates all other costs (moving stuff
> thru xml is expensive).
Indeed that is a very serious problem. I wonder if we can't solve this by using, instead of XML, some more efficient data format, e.g. one which holds tabular data in binary form with just the metadata in plain text. There's something called the "FITS table" with exactly these properties which perhaps astronomers should investigate :-)
Merry Christmas (translation for those in countries where Christianity is not the established religion: Happy Holidays).
-- Clive Page Dept of Physics & Astronomy, University of Leicester, Leicester, LE1 7RH, U.K.Received on 2005-12-21Z17:04:58