Parallel Write Tests Clause Samples

Parallel Write Tests. Figure 12 shows a comparison of the results from parallel write tests using the sites native protocol and the iRODS protocols. This information is relevant for the data staging service case outlined in WP4, since they indicate for each site how write performance drops off with load, which will govern how fast data can be written back to an EUDAT data centre after completion analysis on, say, a PRACE site. The values are clearly only indicators since the test storage used in this work package are small compared to production systems where bandwidth can be distributed across many physical machines. In a production set up, the performance will drop off less rapidly than with the test set-ups. In this case we have used what we believe to be a representative file size if 2GB. In some communities this will actually be a large file while in others it will be average or even small. 30 and NFSv3 import from GPFS - GPFS server configuration abov 1400 1200 DKRZ (IPUT) DKRZ (IRSYNC) JSC (Native) 1000 JSC (IPUT) (Native) PNSC-GPFS (IPUT) PNSC-NFS (Native) PNSC-NFS (IPUT) RZG (Native) STFC (Native) STFC (IPUT) STFC (IRSYNC) UiO (Native) UiO (IPUT) At some sites we have used two available iRODS protocols to write from local storage to the storage element, iput and irsync. For all sites except JSC, performance using iRODS iput method was worse than using the native protocol, with the difference increasing as the number of parallel writes was increased. The results for JSC can be explained by changes to their underlying storage configuration between runs. STFC shows the largest difference. To some extent this is because the storage system used there does not support multi-threaded writes in a manner iRODS can use. During testing, two sites (STFC and DKRZ) reported errors on transfers of the form ‘SYS_COPY_LEN_ERR, Broken pipe’. In some cases these resulted in partial file transfers, in others it seemed to occur after file transfer completed based on filesize and checksum. In the worst case at STFC, 15% of transfers resulted in these problems. After some reconfiguration of the underlying Oracle database used here, the number of errors was significantly reduced. Two sites, STFC and DKRZ have also tested irsync as a protocol. At DKRZ, this protocol demonstrated significantly worse performance than iput. Tests at STFC showed rather than being slow, the protocol was quite unreliable, failing up to 75% of transfers in even single threaded transfer. These errors were ‘silent’ in ...