Optimization of Metric Collection Sample Clauses

Optimization of Metric Collection. ‌ Xxxxxxxxxx is able to collect a wide variety of metrics and data about the host system and store it for further purposes such as logging and data analysis. The latter point is particularly interesting, as the stored data serves as the foundation for the training of the AI scheduler and provides the digital twin with constant flow of data. An excerpt of metric collection requirements can be found in Table 4 with a collection of metric names accumulated in Appendix A. Besides collecting these metrics there is also a risk of putting too much or unnecessary load on servers or edge devices. Collecting metrics can be a resource-intensive task and if too many metrics are collected, it can slow down devices where these metrics are collected from. To prevent a performance decrease or latency increase the following strategies for load reducing can be considered: • Appropriate Scraping Intervals: Instead of collecting metrics every second, metrics can be collected with larger intervals (e.g., every 15 seconds or 1 minute). For some metrics, a higher scraping interval may be sufficient without sacrificing the accuracy of monitoring. • Aggregation of Data: Aggregate metrics locally where possible (e.g., sum, average, max) before collecting them to reduce the overall data volume. • Using Compression: Enable compression for data transfer between the servers and the moni- toring system to reduce network usage. • Using Push-Based Metrics: Where applicable, use push-based metric collection systems (e.g., Pushgateway in Prometheus) to offload the scraping responsibility from the monitored servers. • Filtering Unnecessary Metrics: Only collect metrics that are essential for monitoring and alerting requirements. Exclude irrelevant or low-priority metrics. Table 4: Technical Details of Metric Collection (Digital Twin Input Data Requirements). Monitoring Item Metrics (Digital Twin Input Data Requirements) API/Plugin Monitoring Soft- ware Job and Queue Job Completion and Status and Queue Metrics • Specification of the jobs • Pending jobs • Running jobs with their mapping to resources • Priority • Restrictions • Run time • Statistics about job arrival • Time of day • Common burst times • Workload model on non-cluster resources required • Jobs scheduled on a node • Scheduler Performance • HPC – Slurm Plugin – SlurmDBD (Slurm Database Daemon) API – Slurm Daemon Monitoring • Cloud – Kubernetes API • Edge – KubeEdge (Kubernetes API) • HPC: HPC mon- itoring Systems (ExaMon, and Grafa...