RE: Collaboration on Source Catalogue DM, ADQL and SkyNodes

There is also the cross match that Tanu & Maria implemented in SQL. That may be the easiest way to do the implementation if the nodes all have a SQL backend.
When things are "in" the database it runs fast ~100k object/sec for small radius.
But, getting objects into a node dominates all other costs (moving stuff thru xml is expensive).
We recently found some logic errors in our "zones" cross match (near the meridian and poles) but seem to have a fix for them now. Maria can tell you all about this (it is the subject of her PhD).

-----Original Message-----
From: owner-voql-at-eso.org [mailto:owner-voql-at-eso.org] On Behalf Of Pedro Osuna
Sent: Wednesday, December 21, 2005 4:26 AM To: Mark Taylor
Cc: Pedro.Osuna-at-esa.int; dm-at-ivoa.net; Maria A. Nieto-Santisteban; Francois Ochsenbein; Inaki.Ortiz-at-sciops.esa.int; Aurelien.Stebe-at-sciops.esa.int; voql-at-ivoa.net Subject: Re: Collaboration on Source Catalogue DM, ADQL and SkyNodes

Hi Mark,

wonderful news. I do want to chat about it with you, and I suggest that we do so from now on out of the general list to not annoy people unnecessarily...

This is the type of collaboration I was seeking when sending a very early -maybe even premature- mail, and at first glance your libraries might fit easily in our scheme.

I'll send you a separate mail and look forward to a wide collaboration.

Thanks.
Cheers,
P.

On Wed, 2005-12-21 at 12:02 +0000, Mark Taylor wrote:
> On Wed, 21 Dec 2005, Pedro Osuna wrote:
>
> > Hi Maria,
> >
> >
> > > - How does Catalogue Data Model used look like, especially what is
the
> > > common set of attributes and the associated metadata.
> >
> > The point is in the (Source) Catalogue Data Model, with emphasis in
> > the "Source" part. This one is the one I showed on behalf of the
> > Catalogue DM subgroup at our last interop meeting here at ESAC. I
> > attach a pdf with the initial proposal, but please use it only for
> > temporal reference, as the whole document will be changed (according

> > to requirements from Jonathan after the interop meeting).
> >
> > > - What are the plans about registration? Will these nodes (Basic?)
be
> > > registered and therefore accessible through Open SkyQuery? How
many?
> >
> > yes, they will. How many, I don't know. In Strasbourg, Inaki and
> > Aurelien worked on a couple of them, Tycho-2 and USNOB, but the CDS
> > colleagues will work on more.... Francois will answer to this
> > question at some point I presume.
> >
> > > - The cross-match functionallity. Are you planning on a 2 catalog
> > >cross-match or n-catalog cross-match? How does the cross-match
> > >function look like?
> >
> > n-catalogue cross-match is what we are trying to get at; it will be
> > a client based cross-match, and therefore the cross-match function
> > will be designed and run at the client side (i.e., servers do not
> > need to worry about implementing one specific cross-match or the
other).
> > At the current status, the client sends an ADQL to the server to
> > discern which type of cross-match it can do with it (whether only
> > positional, positional with errors, etc.), and takes the
corresponding action.
>
> Pedro,
>
> my STILTS package contains a standalone crossmatcher suitable for
> client-side operation which might be of use in implementing the
> crossmatching functionality you need. It is highly flexible (can
> perform matches in N-dimensional space, e.g. sky position only or sky
> position plus proximity in one or more flux values), suitable for
> matches up to around a million rows or so (could be extended with some

> work), and fairly efficient (a recent test matching 600,000 vs 700,000

> row tables resulting in 2000 matches took about 90 sec). It does not
> understand ADQL so it's not going to be a complete solution to your
> problem, but it may be of use as a back end matching engine. The tool

> currently only matches two tables, but I'm planning to provide N-table

> matching fairly soon. STILTS provides this functionality from a
> command-line tool (tmatch2), but a public java API is also available
> for programs that want access to it within a JVM.
>
> I'm not sure how you're planning to implement your matching, so this
> may not suit your purposes, but I'm happy to discuss it further if you

> think it might be of interest.
>
> For more details see:
>
> http://www.starlink.ac.uk/stilts/sun256/tmatch2.html
>
> Mark
>

--
Pedro Osuna Alcalaya

ESA
Science Archives System Engineer
Science Archive Team
European Space Astronomy Centre
(ESAC/ESA)
e-mail: Pedro.Osuna-at-esa.int
Tel + 34 91 8131314
---------------------------------

European Space Astronomy Centre
European Space Agency
P.O. Box 50727
E-28080 Villafranca del Castillo
MADRID - SPAIN