Monitoring Engine Clause Samples
Monitoring Engine. (ME) The monitoring engine runs in each router and aims at sampling and capturing packets at the interfaces of the router. It then exports them in the form of NetFlow records to the central collector. As depicted in Figure 1, one can observe four main modules: • Packet capturing: this module listens to the network interface and sample data at a given sampling rate. This sampling rate is configured each time by the Cognitive Engine (CE) next to the optimization it carries out after correlating measurements from all routers. • Classifier: Once a packet is sampled by the Packet capturing module, the classifier identifies flows by a key (in our case this key corresponds to the 5-tuple consisting of source and destination addresses, source and destination port numbers, and protocol number). The Classifier then determines if a flow is active or if it is a new flow. If the flow is active, it updates real-time statistics on that flow such as the number of packets and bytes. If it is a newly observed flow, it inserts a new flow record for this new packet’s key. The ME maintains the keys of flows forwarded by the router together with the collected statistics on those flows. A flow is declared terminated by the Classifier in one of three cases: (i) when observing a FIN or a RST packet (TCP control), (ii) when a timeout expires after the record for that flow was created, and finally (iii) when the number of records exceeds a given threshold in order to release memory. • Reporting: Once collected, flow records are exported using UDP messages to the central Unit (Cognitive Engine) through the CM (Cognitive-Monitoring) interface. • Controller: Based on the collected data and machine learning methods, the cognitive component takes a decision on how to tune the sampling rates and sends the decision back to the ME. The controller in each router receives the decision and updates the sampling rate in the ME accordingly.
Monitoring Engine. The Monitoring Engine is the component of the Provider layer of the Monitoring Architecture responsible for filtering, aggregating and correlating events collected by the WISER Agents and for the generation of alarms based on a predefined set of correlation rules or security directives. Deliverable D4.1, Section 6 gives details about the internal operation of the Monitoring Engine. A summary can be found below:
1) The events are received by the Monitoring Engine in a standardized JSON format (see section 5.1.2) from the Rabbit MQ server.
2) The events received are pre-processed and filtered, in accordance to certain policies. A policy is a set of filtering conditions configured in the SIEM system included in the Monitoring Engine. This way, the number of correlation events arriving to the correlation engine is notably reduced. The data schema to use is also provided for this pre-processing step so as to know how the data are structured and need to be passed to the next correlation step. Since the architecture of the Monitoring Engine is designed to run in a distributed way using Apache Storm10, a different process will be in charge of each policy defined. In particular, a different policy associated to each organization defined in the Data Warehouse is initially configured. Each policy will filter the reports received from the monitored infrastructure based on the WISER agents defined in the Data Warehouse for a specific organization. In this way, the incoming traffic is split into separate data streams (one by organization) for its analysis and correlation. To these policies, included by default to provide multi-tenant support, the administrator can add more conditions such as filtering based on a set of the IP addresses in case only a subset of the events collected by the agent are relevant.
3) The events matching the filtering criteria are correlated by means of the Correlation Engine, which is the core of the Monitoring Engine. The Correlation Engine has preconfigured rules aiming at detecting patterns being fulfilled. These rules are defined in EPL (Event Processing Language), which is a SQL-like language. When the pattern is found, an alarm is generated accordingly. In the Storm topology defined for WISER, there is by default a different correlation process by each organization. As described in the previous step, each of these correlation processes only receives events coming from agents deployed in a specific organization, according to the policy an...
Monitoring Engine. The Monitoring Engine (already introduced in D2.2, section 3.4.1) is the component on the Provider layer of the monitoring architecture responsible for the filtering, aggregation, correlation of the events collected by the WISER Agents and generation of alarms based on a predefined set of correlation rules or security directives.