Roy said:
> I was thinking that the VOEvent standard is finished now
The protocol is "finished" (meaning we should continue to expect the need for relatively infrequent maintenance updates), but we're just getting started with deployment and commissioning. I wouldn't be surprised at our fielding several generations of broker and transport prototypes before the distributed system nears maturity.
> All we have to do is make sure they don't change STC under our
> noses ....
I've completed my first read-through of STC v1.30. I think that the last thing "they" (meaning Arnold) wants to do is change STC :–)
> For the SEAP, the first question is are we ready to standardize a
> protocol?
Easy. No. We're ready to be ready, however.
> Perhaps we should start by asking where are the current VOEvent
> repositories, why do people want to extract past events, and which
> events do they want.
Ok, we've started this conversation under a separate thread.
There is a second thread, whether it also falls under the heading of "SEAP", or not. This is the issue of the backbone topology and the overall network architecture. It isn't just the word "packet" that makes me think that there are parallels between VOEventNet and Internet technologies. Authors connect directly to publishers. Subscribers connect directly to brokers. How do the packets get from the original publisher to the final broker? Or do subscribers need to connect to multiple publishers? What are the logistics of negotiations between brokers? Do all brokers talk to all other brokers? Do packets need a time-to-live field and other gizmos familiar to Cisco? In short, how are VOEvent packets routed?
> One of the key difficulties here is the <What> section with its
> free-form parameters. Queries will have predicates like "All events
> which have a Param called BANANA which have a Float value that is
> between 2 and 5". If we want to build forms that allow subscription
> and query, we will need to allow people to insert their own BANANA-
> type predicates -- because we do not know in advance what Params
> might be there.
I'm confident that application specific schema will evolve in the fullness of time (for some applications, anyway). Params are an option to reduce the cost of VOEvent buy-in to a level that projects are willing to pay. I think they have been completely successful for this purpose.
> This has been defined as a Registry extension. Publishers and
> Repositories of VOEvents should have a valid Registry record.
> Perhaps we can have a joint Registy/VOEvent session to figure out
> how to build these schemas.
Ok. Let's start listing focus/session topics:
SEAP
Broker arbitration
Registry use for VOEvent
Packet authentication
> My take would be to make it work and get people involved in the
> simplest way first, before generalizing. Subscription to specific
> Publisher. Later, they can subscribe to an Aggregator if they want
> that. This provides both of the models you mention.
Yes. Each publisher may support direct connections indefinitely. Assuming we remain as successful as we've been at continuing to build the VOEvent community, however, I believe a more scalable architecture will be needed soon.
> You're building your own Registry? Why? Surely we can utilize the
> VO Registry structure? Perhaps we can revive Matthew's Carnivore
> registry and make sure it is happy with registering VOEvent nodes.
I didn't know his predator was hibernating. Time for the prey to play. I've appended the diagram from my NVO summer school slides. Let's see if I can resurrect my thought process from 1.5 years ago. "SEAP" in this slide is some superset of the current transport protocol - that is, VOEvent packets being moved from place to place. The roles have changed. Portal here is our current Publisher role, while what used to be Publisher is now a Broker (or Aggregator as some insist on calling it).
Authors generate content on the left in different formats. It is published and injected into what we now call the backbone. It bounces around the backbone like pachinko, and in some fashion the content wends its way to a Subscribing client on the right which passes it on to users in application dependent ways. Just like the current backbone, there are three nodes. The reason for depicting it this way is that with three nodes you don't have to worry about the topology. Is it a ring? Or a star? We haven't even had to fully face fundamental client server/P2P questions yet. On the other hand, we do have a good notion of what publishing means and what subscribing means.
So the original breadth of the SEAP notion was something like transport (or preset query, if you will) + query (or delayed transport, if you won't). The red arrows, on the other hand, were an attempt at depicting whatever service negotiations occur internal to the backbone/network of brokers. OAI is the Open Archives Initiative which "promotes interoperability standards that aim to facilitate the efficient dissemination of content". Other people know infinitely more about this than I do, but while OAI has been used interchangeably with "registry" in the VO, it isn't obvious to this naive observer that current registry technology exhausts the notion of metadata harvesting that underlies the efficient dissemination of content.
In particular, VOEvent packet streams have some pretty interesting metadata - not just so-called science metadata, but actual content and routing metadata. I see the meta-conversation between brokers that is superimposed on the actual packet transport as an attempt to characterize the streaming of events in an adaptive, robust, efficient (+ other good adjectives) fashion. So it would be simplistic to call this an internal registry, but I wouldn't be surprised to reuse some of that technology.
On the other hand, VOEvent brokers (Publishers, Repositories, Filters) should certainly appear in the normal VO registry for all the obvious reasons.
Rob