RE: VOTable Questions

A small add-on info: with SDSS, people are routinely requesting output tables containing over 10 million rows. This is the typical size that we should be worried about, not tables with 100-1000 rows. Yes, they usually buffer these on the client end, but not always -- we have written now several statistical applications, which process the incoming data stream without buffering the whole.

I like Guy's suggestion, that would avoid any conflicts with streaming.

Cheers, Alex

-----Original Message-----
From: Patricio F. Ortiz [mailto:pfo-at-star.le.ac.uk] Sent: Thursday, October 23, 2003 8:20 AM To: Alex Szalay
Cc: Alasdair Allan; Randall Thompson; votable-at-ivoa.net Subject: RE: VOTable Questions

Hi all,

I was trying to avoid jumping on this issue, but OK, I feel it's important.

The number of rows a VOTable contains can be quite relevant as a piece of meta-information, as it is the number of rows of the original resource.

I'd advocate that the number of rows should be presented before the table (eg, if I'm only interested in the meta-data content and I want to know the percentage of a resource a given VOTable represents).

On Thu, 23 Oct 2003, Alex Szalay wrote:
> It should ONLY be an optional keyword, since many of the
> current VOTable generators work off of a query, where
> you do not know in the beginning, how many rows the
> table has in the end.
>
> Thus, one cannot write a VOTable on a stream, if we make this
> part of the mandatory spec, it has to be buffered first on
> the server side, a serious impediment.

Then buffering occurs on the reading end. It seems to be unavoidable :-(

In any case, wherever it is decided to be located, I find this an important piece of information to be added to a VOTable. I have not seen the presence of TotalTableRows in the current implementations yet. How do people feel about adding this as an optional information? What I have in mind are applications where a user may not know the details of a table beforehand, like in queries involving "content selection" (ie, using UCDs).

Cheers,

Patricio

---
Patricio F. Ortiz			pfo-at-star.le.ac.uk
AstroGrid project
Department of Physics and Astronomy
University of Leicester			Tel: +44 (0)116 252 2015
LE1 7RH, UK