Data Staging Clause Samples

Data Staging. When scientific communities use data-intensive applications, their data inputs are typically staged onto computational resources prior to job execution. Following execution, output data sets are generally staged out of compute nodes and onto storage systems. Additional data transfer operations may be required to move intermediate data products from nodes where they are produced to nodes where they will be consumed by other jobs. If the storage space on a node is limited, these stage-out operations need to be performed before other jobs can run. In distributed computing, where resources are geographically dispersed, efficient data staging is essential. Starting with this assumption, the objective of the data staging service is to enable research communities to stage data from EUDAT repositories to external HPC facilities for further processing. The “Data Staging” service can be considered as the fundamental service for exchanging data between the EUDAT domain of globally registered data and the domain of data consumers and producers (e.g. scientific instruments, HPC or HTC facilities). Three EUDAT core communities (VPH, ENES and EPOS) explicitly requested this service as they need to perform statistical model analyses on stored data. This service area differs from the safe replication scenario, which was discussed in the previous section, on three main points.  The replicated data are discarded when the analysis application ends.  The PID references are not applied to data replicated into scratch workspaces.  The users initiate the process of replicating data, whereas in the safe replication scenario, data are replicated automatically on a policy basis. A typical utilization scenario is depicted in Figure 14. 10▇▇▇▇://▇▇▇▇.▇▇▇▇▇▇.▇▇▇/p/irodspython/issues/detail?id=11 1. The desired data are replicated from community storage to an EUDAT node using the “safe replication” solution. 2. Next the data are replicated between two EUDAT nodes, since the destination node is close to the target HPC facility. The corresponding PID record contains all the relevant URLs for the copies. 3. The data are then staged into a HPC facility that is either close to the EUDAT node or available elsewhere, for example, as part of the PRACE infrastructure11. 4. Finally the replicated data are staged into the local HPC facility. Afterwards, the analysis results will be staged out to the original source, and the results can be copied back to the community storage.

Data Staging. The question that remains is how to integrate the architecture with the outside world. We have already seen that we make use of external infrastructures for AAI. For the moment, we assume data is being transferred with GridFTP. In this case, three entities must have X.509 certificates (and private keys, and credentials): the source, the destination, and the controller (namely the user client). One or more of these could be the same. But, more importantly, the certificates must be signed by CAs that is trusted by each entity. This means that EUDAT cannot necessarily run its own CA and expect PRACE (say) to accept it44. The proposal of the AAI TF is to secure the infrastructure itself with known and trusted certificates from NRENs, IGTF, and commercial CAs (particularly for browser-facing services) – these are all (usually) trusted by e- and cyber-infrastructures across the world. However, earlier we proposed to generate federation credentials on behalf of users, to be used internally in EUDAT. Such a service, if X.509 based, would most likely not be trusted outside of EUDAT (see footnote 44 on page 43). Consequently, data staging would have to be done by an automated service with a trusted certificate. However, it would also need to preserve, or at least respect, permissions and rights associated with the user. 42 The case is equivalent in general for command line login. 43 See e.g. the IETF SCIM working group. Moonshot (ABFAB) will also be a potential alternative. 44 That is not to say this is impossible: it can be done but it requires a lot of work (defining a CP/CPS, see RFC3647), buying a Hardware Signing Module for protection of an online CA; checking and validating the CA against the IGTF SLCS profile, and seeking accreditation – the latter process can take 2-3 years – which is not compatible with the timescale of this project.

Data Staging. The Data Staging service is designed to facilitate transfers of large data sets to and from the EUDAT infrastructure. The building of this service has been continuing since the start of the project, and reached a high level of maturity recently when its core component was completely renovated to overcome performance limitations experienced with the previous technology. As reported in deliverable D5.2.1, the EUDAT Data Staging service was originally based on the ▇▇▇▇▇▇▇ technology, a GridFTP server – written entirely in JAVA  being able to interact with iRODS resources. After a long period of testing, and upgrading the underlying iRODS Java library (Jargon37) to the latest release, serious performance issues were revealed that significantly reduced the benefits of GridFTP (see Annex I for further technical information)38,39. To address this issue, CINECA has build a new component in order to simplify the software stack (the Java layer causing a big transfer overhead was removed), optimize the performance of the transfer, and thus make the software more stable and reliable. The new component was developed in accordance with the GridFTP Data Storage Interface (DSI) specifications which allow a GridFTP server to be a transfer interface to as many data storage systems as possible, like iRODS. Besides the work done on the server side of the service, the Data Staging script was also extended to support new functionalities such as the capability to handle data using its assigned PIDs. The Data Staging service is currently employed by the VPH community to ingest and stage their data sets onto the EPCC storage resources that are part of the EUDAT federation. 33 Home page iRODS: ▇▇▇▇://▇▇▇.▇▇▇▇▇.▇▇▇ 34 ▇▇▇▇▇://▇▇▇▇▇▇▇▇▇▇.▇▇▇.▇▇/download/attachments/28836527/EUDAT-DEL-WP2-D2.2.2-EUDAT_Services_Rolling_Plan_2012- 2015_%28update%29.pdf 35 ▇▇▇▇://▇▇▇▇.▇▇▇.▇▇▇▇▇▇.▇▇▇/AmazonS3/latest/API/Welcome.html 36 ▇▇▇▇://▇▇▇.▇▇▇▇.▇▇▇/tech_activities/standards/curr_standards/cdmi 37 ▇▇▇▇▇://▇▇▇.▇▇▇▇▇.▇▇▇/index.php/Jargon 38 ▇▇▇▇▇://▇▇▇▇.▇▇▇▇▇.▇▇▇/gf/project/jargon/tracker/?action=TrackerItemEdit&tracker_item_id=1243&start=0 39 ▇▇▇▇▇://▇▇▇▇▇▇.▇▇▇▇▇▇.▇▇▇/forum/#!topic/irod-chat/tgtsvVXDCNI The following subsections report further details about the new DSI GridFTP component and the extensions made to the Data Staging script respectively.

Data Staging. Creation of a database that contains all the data in the format required by the new system. This data is updated regularly and can be used by the legacy application as well as the new system, thus providing a two-fold benefit to the agency.

Data Staging. The Data Staging service works on top of the EUDAT storage services (e.g. iRODS) provided by the Safe Replication service. Due to the nature of the different communities requesting this type of service (namely VPH, EPOS and ENES), different solutions have been evaluated and demonstrated. These solutions ranged from basic iRODS tools and third party transfers via GridFTP, to higher level tools such as Globus On-Line and the XSEDE file manager. To make it possible to access GridFTP from iRODS, iRODS was extended with the ▇▇▇▇▇▇▇ GridFTP server. The Data Staging service is regarded as a user-level service that allows individual users to easily manage the staging of large amounts of data (that consists of many files) and that results in many file transfers. Consequently the Data Staging service depends on the services developed by the AAI task force to provide federated access to the EUDAT data services. The time plan of the AAI TF, does not predict that a federated AAI solution will be available in the short term. Because the data staging solutions are all based on standard authentication methods (e.g. X509, username/password), it is possible to provide access in a less federated way (e.g. separate EUDAT or local credentials). As well as continuing to work on the selected technologies, the Data Staging TF will focus on developing a practical solution to enable user access during the coming months. During the service-building process, the Data Staging task force collaborated with the following organisations.  PRACE The communities actively participating in the Data Staging task force (VPH and EPOS) also use the PRACE HPC computing facilities. To provide an optimal solution, the EUDAT data service should be accessible from the PRACE supercomputers. At the moment, EUDAT and PRACE are holding ongoing strategic discussions for the purpose of collaborating. At time of writing, EUDAT and PRACE had agreed to conduct some pilots.  XSEDE XSEDE is the successor of the TeraGrid project in the US, and is funded by the National Science Foundation. The XSEDE organization develops and maintains the XSEDE file manager. The XSEDE file manager is the tool to use for moving large amounts of data between XSEDE supercomputers. EUDAT has collaborated with XSEDE to adopt the XSEDE file manager to make it work with the EUDAT data services.