Re: Asynchronous querying and tabular data

From: Patrick Dowler <patrick.dowler-at-nrc-cnrc.gc.ca>
Date: Tue, 1 May 2007 10:08:26 -0800

This looks like a pretty good analysis of the situation. The claim is that in typical DAL the queryData is always fast and thus fine as a sync method while in TAP the primary/necessary operation is the async query for a query result because the query is expensive/time consuming. I agree with this in principle and think we should design for async. Since async has not seen much support so far, I feel we have a rather clean slate to start from.

However, there are still going to be many cases were a TAP service has modest content, a sync query would work fine, and a service supporting sync query only is much easier to implement robustly. Here I am mainly thinking that when you develop anything async, you bring into the implementation nice things like server-side resource contention, cleanup after some amount of time, controlled access to staged results (via obscurity or security), and all that kind of stuff. For the sync response you just dump it to the output stream and you are done.

So, priority has to be to make the async operation very functional and natural, but I think allow a sync-only TAP service to be compliant. Practically, one can write some wrapper code to make an async call synchronous, but I think this is better handled as a client side option. Thus it looks to me like syncQuery and asyncQuery both need to be optional, but you obviously have to support one of them :)

Pat

PS - Examples of small TAP services I can envision:

These would all be relatively small and highly useful TAP services. A simple sync implementation is quite feasible. OTOH, we are also building a source catalog to capture results from the entire CADC data collection; I estimated roughly 5-6 billion sources (~5TB)...

On Tuesday 01 May 2007 08:18, Kona Andrews wrote:
> Dear all,
>
> Copied below is a useful discussion from a colleague of why access
> protocols like SIAP and SSAP don't extend so gracefully to large
> tabular data queries, and why therefore we shouldn't try to make
> TAP exactly conform to the model assumed by these protocols.

-- 

Patrick Dowler
Tel/Tél: (250) 363-6914                  | fax/télécopieur: (250) 363-0045
Canadian Astronomy Data Centre   | Centre canadien de donnees astronomiques
National Research Council Canada | Conseil national de recherches Canada
Government of Canada                  | Gouvernement du Canada
5071 West Saanich Road               | 5071, chemin West Saanich
Victoria, BC                                  | Victoria (C.-B.)
Received on 2007-05-01Z20:02:57