Hi Roy -
On Mon, 3 Jan 2005, Roy Williams wrote:
> How easy would it be to interoperate your proposed system with the
> Astrogrid CEA system?
Paul Harrison and I discussed this briefly following the OPTICON meeting in December. We had limited time and did not resolve the issue, but made some progress. Clearly there is some overlap. Interoperability at some level is possible. The degree to which implementation could be common is less clear as the two projects are not addressing quite the same problems.
Interoperability at the level of components is possible. Ideally one would like to be able to produce components without knowing how they will be run, and run components from multiple execution frameworks. It is hard to build a single framework which addresses all problem domains well. CEA addresses mostly distributed grid workflows, whereas our proposed system mainly targets tightly integrated systems (desktop systems or HPC clusters), but could be used to build services to participate in a grid workflow.
Although a grid workflow and a distributed application running on a single system (desktop or cluster) have similar distributed computing architectures, I am dubious that the same approach could be used successfully for both. The problem is that the components in a HPC system tend to be finer grained than in a grid workflow, and the dataflow tends to be very different. Furthermore if we try to go the desktop there are severe requirements on system size, performance, and complexity, or astronomers will not use the software or contribute their own code.
This leads us to believe that a different approach is required in the two areas. The HPC approach requires finer grained components capable of parallel execution, with efficient access to shared data. Such a system can however be used to implement services (scripted meta-components) which are designed to be used in a grid workflow, and which support the VO grid communications infrastructure.
Hence, one approach to interoperability is to say that a system like Astrogrid does distributed grid workflows which link services and other VO resources together. A system such as we describe addresses the problem of building the computational services which participate in such a grid workflow. In general these are not conventional programs (like SExtractor), but complex distributed applications composed of multiple components linked by an execution framework and driven from some sort of scripting layer.