Dave Morris wrote:
> It is not beyond imagining that a valid science query to a large data
> set could return 21G bytes of data.
>
> Whatever we replace/update VOTable with it should be easy process the
> service response as a stream of rows,
Dave
VOTable is well suited to handling bulk data. The answer, in my opinion, is NOT to encode everything in XML and SOAP, but rather use either the binary or FITS data streams in the VOTable specification. The metadata stays in XML, the big data streams in fixed-length records. It is easy to split into parallel streams, allow parallel and asynchronous evaluation, etc. This approach scales well beyond your gigabyte dataset -- by factors of a 1000 or more.
Roy Received on 2008-07-30Z20:47:07