Re: VOStore/VOSpace use cases

Matthew:
SDSC is supporting similar data access use-cases to those defined by Doug, plus additional data sharing and preservation use cases. They include:

• build a private collection. The private collection may be distributed across multiple physical storage systems. We use a logical resource name to represent the set of physical resources where the data may reside. The user writes a file to the logical resource name. The data grid (VOspace) then picks a physical resource based on available space, or disk cache at the same IP address as the user, or disk cache at a remote IP address, or a tape archive.

The VOstore interface would provide the standard access mechanism to the storage repository. In our use of the data grid, the user would never talk directly to the VOStore interface in front of the storage system. Instead the VOSpace data grid would interact with VOStore on behalf of the user. In practice, the VOSpace data grid could provide an abstract VOStore interface to the user as a standard access method.

The other difference is that the user can associate metadata with each stored file. The metadata could be the parameters in a FITS header, or any other descriptive or provenance attributes of interest.

The files may be referenced by a logical file name managed by VOSpace. This logical file name can be mapped to a URL for access through a web browser or WSDL service. The user typically would not know the physical file name at the actual storage resource where the data ends up.

• build a local disk cache. This case is the same as the first, with the specification that the storage resource is the local system. Again a VOStore interface is provided to the local storage, but access is through VOSpace. This allows the authentication and authorization to be managed by VOSpace. VOStore only checks authentication related to VOSpace identification. VOSpace checks the user authentication.

Multiple types of APIs can be put in front of VOSpace including object, java, perl, python, web, WSDL, ... The storage system does not have to support these interfaces. Instead VOSpace maps from these protocols to the protocol used by VOStore, which in turn maps to the local storage repository protocol.

• support for third party transfer is managed by VOSpace. VOSpace has to authenticate itself to each VOStore instance. This means that trust has to be established between all storage providers through the VOStore interface to the VOSpace data grid. Third party transfer is then simply the copying of data between two storage systems, both of which trust the VOSpace data grid. In effect, the VOSpace data grid is a trust virtualization mechanism that enables controlled interactions between storage repositories on their behalf.
• replication environments. To support large scale analyses, we have replicated image surveys onto the Teragrid storage systems. This adds the requirement for support of replicas, synchronization of replicas, aggregation of files into containers for interactions with tape archives, parallel I/O support for large transfers, packing of small files before transmission, support for firewalls, and support for GSI authentication.
• digital libraries. For small collections, the DSpace and Fedora digital libraries provide mechanisms for assembling, organizing and managing files. Both DSpace and Fedora have now been ported on top of the SRB data grid, making it possible for these digital libraries to manage data that is distributed across multiple storage systems.

Reagan

>Hi Matthew -
>
>On Thu, 11 Aug 2005, Matthew Graham wrote:
>> Hi Doug,
>> Guy has mentioned to me that you had some specific VOStore/VOSpace use cases
>> - are these already documented somewhere or could you provide a short
>> description of them?
>
>Probably this refers to my data access related use-cases. The are three
>primary cases:
>
> 1) Service generates datasets and puts them in a local VOStore.
> Client picks up the images with a getData request via URL (this
> is the default and is what we do currently).
>
> 2) Client operates a local VOStore. Client issues a staging request
> to a remote service to generate datasets and deliver them to
> the Client's VOStore. Service sends a message to the client as
> each dataset is delivered (or client gets the messages when it
> polls etc.). In a typical scenario a client may issue many such
> asynchronous requests which execute concurrently. Support for
> authorization is needed since the local VOStore is externally
> writeable, and to support access to proprietary data.
>
> This is the scenario I see for VO-Client functionality integrated
> into a desktop data analysis system. The VO-Client implements
> the client side of the data access services including support for
> queries, asynchronous services, authentication/autorization, etc.
> Data is fetched and placed in a local VOStore in the local VO-Client
> daemon. Data analysis software can then directly access the files
> as they would a local file, or we can put an object API in front
> of the data and the analysis code will see an object API and can
> completely ignore all the plumbing.
>
> In this use-case the client may also function as a service,
> generating new data products which it exposes to the VO via the
> local VOStore.
>
> 3) As in 2), but the staging request specifies that data is to be
> delivered to a third VOStore, not co-located with either the
> controlling client or the service. This permits general grid
> workflows. The client "application" is generally controlled from
> one location, but may have components which execute at multiple
> locations.
>
>Note that asynchronous services, VOStore, and security all need to be
>interoperable to do these things. Ideally we need some mechanism for
>asynchronous messaging. A polling-based mechanism would be adequate for
>simple cases but does not scale well.
>
>All of these use-cases emphasize dataflow and data caching rather than
>permanent storage of data. The only exception might be 2) where we might
>want to keep the data around indefinitely. However, it is still primarily
>a network data cache mechanism.
>
>Another major use case is a user filestore where data is maintained
>indefinitely, possibly distributed to multiple locations. This is more
>the VOSpace type of scenario.
>
> - Doug
Received on 2005-09-01Z16:44:03