Parallel Read Tests Clause Samples

Parallel Read Tests. Figure 13 shows a comparison of the transfer times based on parallel read tests using native and iget protocols. This information is relevant for the data staging service case outlined in WP4. These tests indicate for each site how read performance drops off with load, which will govern how fast data can be staged from an EUDAT data centre before analysis on, say, a PRACE site. The values are clearly only indicators since the test storage used in this work package are small compared to production systems where bandwidth can be distributed across many physical machines. In a production set up, the performance will drop off less rapidly than with the test set-ups. In this case we have used what we believe to be a representative file size if 2GB. In some communities this will actually be a large file while in others it will be average or even small. Based on results from baseline testing observe that, as expected, read rates drop off as the number of concurrent reads increases. The RZG implementation shows the greatest variability between the maximum and minimum read rates, varying by up to 80% for a given number of threads. The storage implementations at PNSC, DKRZ and UiO show the most consistency in read rates for a given number of threads, varying by less than 10%. Also, it would seem that the Juelich storage implementation performs significantly worse than the others under load. However, this instance is under constant development and the results represented here are only a snapshot of the performance when the tests occurred. One slightly surprising result is that with the limited number of threads used here, performance seems to degrade linearly for most sites. This may be as a result of loading the client machine at each site; most sites read data from archival storage to local disk at only one client, so at least some of this performance will be caused by load being put on the client. 4000 DKRZ (Native) JSC (Native) PNSC-GPFS (Native) RZG (Native) STFC (Native) UiO (Native) As can readily be seen from this figure, for most sites the use of iRODS iget command does not introduce a significant overhead in reading out to the maximum number of concurrent reads performed in this test. STFC and DKRZ have also performed the test using irsync. While this appears to show very good performance for this protocol, the graph does mask significant problems. As with the write tests using this protocol, there were very high failure rates. These failures were silent in...