Re: table metadata and the registry

From: Tony Linde <Tony.Linde-at-leicester.ac.uk>
Date: Tue, 08 May 2007 17:50:30 +0100


Hi again, Ray.

Let's break this down. Firstly, I hope we agree that no-one has any problem with getting metadata from the service: we'd agreed to allow this a long time ago and that was why it made it into VOSI (AFAICR).

The issues seem to be about:

  1. getting all the metadata (VOResource) or a limited subset;
  2. getting it via a URL or via a VOSI:getWhatever method;
  3. the structure of registries.

Discussion:

  1. You've proposed returning a limited subset but in the same format as VOResource table schema. Why do you stop at that level? Surely everything within Capability describes the service. If we're going to pick an arbitrary level, why not Capability or, better, VOResource? What is the issue with the service returning *all* its metadata?
  2. I have no problem how the metadata is returned but only want the most efficient method. I would have thought it was more efficient to have all services implement a single standard interface aligned to the way the service is implemented: soap method if a soap service; url if a cgi service.
  3. we had already agreed to have fine- and coarse-grained registries back in Victoria: I'm not sure why this has come up again (this in response to your comments below about forcing you to do registries our way or getting detailed metadata from your services!). As long as these registries are identifiable and applications can tie themselves to the one they need, what is the problem?

Am I missing anything else? Finally... 

 > It is the only way today.  NVO is working on other techniques for
 > drilling down into the information for discovery outside the context
 > of the Registry.

That's fine, but let's make sure the applications and services that rely on registry information *now* can continue to work while these techniques are being developed.

T.

Ray Plante wrote:
> Hi Tony,
>
> On Tue, 8 May 2007, Tony Linde wrote:

>> One last query, if the URL only returns the 'extra' metadata, where 
>> does the core service metadata come from? The registry only? Does this 
>> mean the service provider has to maintain metadata in two locations? 
>> Surely one additional benefit of the getWhatever method is that a 
>> service provider can update their registry record simply by changing 
>> the VOResource record served up by getWhatever?

>
> Let's look at this in two evolutionary stages. In the near future, the
> user still goes to a publishing registry and fills out a form. Consider...
> ...an SIA service: the user need not enter URL for the table metadata;
> this can be set automatically based on the base URL and the
> translater service.
> ...a theoretical TAP service: some near-future standard protocol could
> provide this URL-based access as part of its definition; in this
> case the URL is set automatically in the record. If the standard
> does not provide this, it can still be handled like an SIA through
> a translator.
> ...a Data Collection: since there is not an associated service to
> provide this information, the publisher can optionally enter this
> URL explicitly. A registry may provide tools for formatting this
> information.
> When the form is submitted, the registry decides whether it wants to
> retrieve the table metadata.
>
> In the next phase, when VOSI is more widely supported, essentially the
> same process happens at service machine when the publisher sets up the
> VOResource description getRegistration(). The VOSI standard provides a
> straightforward way to recognize service resource descriptions that
> support getRegistration() (the VOSI capability extension). The
> publisher provides the VOSI URL to the registry. The registry retrieves
> the VOResource description. At that time the registry decides whether
> it wants to retrieve the table metadata.
>
> In either case, the registry has access to the table metadata if it
> wants it, but if it doesn't want it, it doesn't get it anyway.
> 
>>> to see is that all of the table information be retrievable in one URL.
>>
>> That is certainly more reasonable. But what I don't understand is why, 
>> if we're to have a single method of getting the metadata, it cannot be 
>> returned in the format recognised by the registry: VOResource? It 
>> would take no more work to format the metadata as VOResource than some 
>> other format so why not stick with what we have, why come up with 
>> another format?

>
> I hope you noticed my suggestion that the VOResource table schema could
> provide the format because we are already using it. This is my preference.
>
> The issue is not about another format. It's about providing access to
> the information that is not exclusively through the registry.
> 
>> But, as I said previously and above, this will lead to registries 
>> serving information in ways no other does and so, applications which 
>> rely on that information will only be able to work against specific 
>> registries.

>
> Are you going to force us to do registries your way? We need
> interoperability, but this does not mean our registries must behave in
> identical ways. We need--you need--to be able to explore new ways to
> support discovery and access that best suits the needs of our respective
> communities without sabotaging each others efforts. That's what I'm
> looking for. Do you agree with this?
> 
>>> To put it in concrete terms, the AstroGrid registry effectively
>>> pressures the NVO registries into supporting fine-grained table
>>> ...  First, you
>>> encourage your publishers to provide table metadata.  We harvest these
>>> records which in turn go out to our users as a result of queries.  We
>>> have to then help users make sense of this information.
>>
>> I don't understand this - the information is the same whether you get 
>> it from the registry alone or from the registry plus the service. What 
>> is the difference?

>
> If the fine-grained information is in our registry to begin with, then
> we do not have to deal with the metadata problems within the context of
> the registry.
> 
>>> When there are
>>> problems with the information, it reflects poorly on us, not you.
>>
>> Poor metadata information is a problem we all have to tackle, not just 
>> AstroGrid: it is exacerbated by poor registry population applications 
>> and has nothing to do with the type of registry.

>
> Do you see that we have to bear some responsibility for the metadata you
> export from your registries? And vice-versa?
> 
>>> Second, your application in effect encourages our publishers to provide
>>> table metadata to our registry if they are to be used in your
>>> application, because your application only gets this information from
>>> the registry.
>> This is not going to change, however the metadata is collected, 
>> whether by entry into the registry, by VOSI:getWhatever from the 
>> service or by the metadata URLs you propose. We will continue to 
>> provide full registry information and applications that rely on it: it 
>> is the only effective way to build responsive applications.

>
> I want you to be able to do this. I want you to have a way to get the
> information you need from our services without requiring that you get it
> directly from our registries (via the harvesting interface).
> 
>> It is also, I might add, the only way to discover resources based on 
>> the additional metadata. If someone wants to discover x-ray catalogs 
>> with a given type of informaiton (specified by a ucd), how does it do 
>> this from a pointer-only registry, apart from getting every possible 
>> x-ray source and querying every one of them?

>
> It is the only way today. NVO is working on other techniques for
> drilling down into the information for discovery outside the context of
> the Registry.
> 
>> I guess if we do follow your proposal it will, over the next couple of 
>> years, show what the application developers really want by the number 
>> that tie themselves to AG-style registries vs NVO-style registries.

>
> That is fine. We only ask for the freedom to discover this ourselves.
> 
>> I'd like to expose more to the list since I won't be at the meeting.

>
> (I'm hoping to capture it in a Note.)
>
> cheers,
> Ray
> 
-- 
Tony Linde
Phone:  +44 (0)116 223 1292    Mobile: +44 (0)785 298 8840
Fax:    +44 (0)116 252 3311    Email:  Tony.Linde-at-leicester.ac.uk
Post:   Department of Physics & Astronomy,
         University of Leicester
         Leicester, UK   LE1 7RH
Web:    http://www.star.le.ac.uk/~ael

Project Manager, EuroVO VOTech   http://eurovotech.org
Programme Manager, AstroGrid     http://www.astrogrid.org
Received on 2007-05-08Z18:51:07