table metadata and the registry

Hi WGers,

I've been conversing with many of you regarding the general issue referred to as fine-grained/rich vs. coarse-grained registries. One current manifestation of it is the issue of whether descriptions of table columns should appear in the registry. We see use cases emerging that want to use this information for discovery and planning, but handling this information in the registry raises costly curation issues. I would like to propose a solution to this issue that I believe can serve as a model for handling other reputed fine-grained information. This solution will ultimately call for a standard format for describing a set of tables.

First, I recognize that before we can agree on whether to put fine-grained information into the registry, we need a common understanding of what qualifies as "fine-grained". I have some ideas on this that I will be presenting next week in the RWG session.

The use cases that are driving table metadata into the registry are:

1. Finding tables based on the columns. A specific use case is to build an SED from existing catalog data by searching for tables that have columns described by certain UCDs.
2. Automating the construction of specific queries to catalog services for use within a workflow.

One major reason that placing the column metadata in the registry is attractive is that the registry is an existing system for collecting the information and provides a common way to access it. One current problem with our existing catalog services (Cone Search, SIA, OpenSkyNode, and SSA) is each has a slightly different of presenting this information. Thus, in practice, it is difficult to mine this information--you need 4 different methods. For data collections that are described independently of any service that accesses them, there is no standard way of getting this information other than having it in the registry.

I would like to propose we define a standard format for describing a set of tables and all their columns that can be served by a single, static URL. With this, we can:

1. Include this URL in the resource description of any table service or data collection that includes tables.
2. Define a simple GET method that can be part of a standard service protocol to return this document.

Implementation considerations:

   o More than one URL could be associated with a resource. Thus, if

      a service or collection serves many tables, their descriptions
      could be distributed over several documents of manageable size.

   o  While the information is packaged into individual documents,
      a service can generate this information on the fly as necessary.
      (For example, if TAP were to define separate "getTables" and
      "getColumns" methods, the information could be aggregated via
      internal calls to these methods.)

   o  For existing "standards"--Cone Search, SIA, OpenSkyNode, (and if
      necessary, SSA)--we could devise trivial HTTP GET services that
      convert on-the-fly calls to their respective metadata methods
      into the standard format.  These services could be provided by
      registries.

The advantages are:

• The information originates at the service and is maintained by the publisher.
• There is a common way to get at the column information. It is not restricted to standard services but can be associated with any data collection or custom service that handles catalogs.
• The information can be obtained through the registry without it being stored in the registry.
• A registry (or other data discovery service) may harvest and warehouse this information for the purposes of fine-grained discovery; at the same time... + it does not require other registries to do the same, and + it does not require/encourage publishers to put this information into the registry explicitly.

The pressure for supporting the above use cases is large, so we need something quickly. I would strongly recommend a v1.0 that is simple and based on existing formats. I think either of two such options would work fine:

   o a profile on VOTable
   o the Catalog description model currently in the VODataService

      extension schema used in the registry
      (http://www.ivoa.net/xml/VODataService/v1.0).

I also want to point out the Source Catalog Data Model, which some of you may be familiar with. Because of its emphasis on the astronomical semantics more than table & catalog structure, it's probably not a good candidate for the format itself. However, it would be a good model for annotating a table description via utypes.

The point is to just support what people are already doing with the registry. If we want to add more to the format (or even totally replace it), I recommend we save it for a save it for a subsequent version.

So the general pattern for "fine-grained" information would be to have VOResource records point to this information that is primarily managed at the providers site. Another area I would like to explore this idea is in using detail coverage information to aid in discovery. We currently have a place in the VODataService schema (an extension of VOResource) a place to point to a detailed footprint service. We would need to add a place to point to a table description. Thus, there is a critical time issue for putting this into place.

I invite your comments, and I will raise this in Beijing during RegWG2.

cheers,
Ray Received on 2007-05-08Z08:02:23