Hi Masahiro,
during our development for the ALMA dataflow to the ALMA archive we encountered the same problems and we found the following solution, which gives us a performance of up to 96 MB/s on a Gigabit Ethernet connection.
We are using both HTTP and Corba based AV streaming where the payload is in both cases a message with mime-type multipart/related. One of the parts (the first in fact) is a VOTable, which points to binary attachments within the same message. You can find a more detailed description along with an example application and message file under http://almasw.hq.eso.org/almasw/bin/view/Archive/VOTableDataTransfer. I actually made a proposal for some very small modifications in VOTable, which would make the integration of such message files even simpler, but unfortunately that was rejected at the time, maybe we should try again.
The other point is that Apache Axis is not implemented to support streaming as this would probably solve the main issues you found.
Andreas
> Dear all,
>
> Have you experienced large data transfer (e.g. >100MB) between
> distributed VO machine? Would you tell your practical solution?
>
> Our JVO team has built a SkyNode server which receives a query and
> returns a result VOTable through Web Services, using Apache Axis. Then
> we encounter a performance problem in large data transfer using SOAP
> messages; Large amount of memory is allocated to hold the entire data
> as Java objects. It slows processing speed, too. Sending large data
> through SOAP messages seems unrealistic.
>
> So we are considering SkyNodes pass result VOTable using:
>
> 1. Staging (Result data are separately transferred through HTTP or
> FTP, like SIAP)
> 2. Attachment Both will need some extension to SkyNode protocol.
>
> Masahiro Tanaka
Received on 2005-07-21Z09:21:12