The Guidelines to Adopt an Applicable SIEM Solution

The need for SIEM (Security Information and even Management) systems increased in the last years. Many companies seek to reinforce their security capabilities to better safeguard against cybersecurity threats, so they adopt multi-layered security strategies that include using a SIEM solution. However, implementing a SIEM solution is not just an installation phase that fits any scenario within any organization; the best SIEM system for an organization may not be suitable at all for another one. An organization should consider other factors along with the technical side when evaluating a SIEM solution. This paper proposes an approach to aid enterprises, in selecting an applicable SIEM. It starts by suggesting the requirements that should be addressed in a SIEM using a systematic way, and then proposes a methodology for evaluating SIEM solutions that measures the compliance and applicability of any SIEM solution. This approach aims to support companies that are seeking to adopt SIEM systems into their environments, suggesting suitable answers to preferred requirements that are believed to be valuable prerequisites an SIEM system should have; and to suggest criteria to judge SIEM systems using an evaluation process composed of quantitative and qualitative methods. This approach, unlike others, is customer driven which means that customer needs are taken into account when following the whole approach, specifically when defining the requirements and then evaluating the suppliers’ solutions.


Introduction
Information and communication technology (ICT) has made a remarkable impact on the society. Companies nowadays rely on information and communication technology which puts their assets under certain risks especially cyber ones, hence they must be kept under control by means of security countermeasures that generate confidence in the use of these assets [1]. Companies all over the world need to ensure valuable assets, uninterrupted business operation (processes), reliable data and quality of service (QoS) to various groups of users [2] [3]. They need to protect their clients and employees both inside and outside the organization [4]. According to Gartner, by 2020, 30% of global enterprises will have been directly compromised by an independent group of cybercriminals or cyber activists. Moreover, in 60% of network breaches, hackers compromise the network within minutes [5]. On the other hand, companies' IT environment is getting more complex, involving many security appliances that may contribute to security strategy in business processes. Therefore, organizations started to invest in integrating SIEMs (Security Information and Event Management) to improve their security.
The term SIEM was introduced by Gartner in 2005. The SIEM system has replaced two types of systems before separated-Security Information Management (SIM) and Security Event Management (SEM) systems [6]. The former provided long-term storage, analysis and reporting, while the latter collected events in real time. Their combination yearned for near real-time analysis, to send notifications and represent information at an operator's console in charge of taking defensive actions. Overall, SIEM system combines SIM and SEM functionalities into one security management system, which collects and correlates relevant data from multiple sources, outputs reports, identifies deviations and takes appropriate actions. For example, when a potential issue is detected, SIEM might log it as new information, generate an alert and instruct other security controls to stop any activity progress. Gartner estimates the SIEM market will grow at a compound annual growth rate (CAGR) of 9.5% between 2016 and 2022, and the worldwide spending on SIEM will reach 3.72 billion dollars [7].
From an organization perspective, the challenge is not just about selecting any SIEM but implementing the right solution that fits better within company structure and is aligned with the existing threats landscape. In addition, it must be flexible enough to be easily adapted to meet any changes thereafter. Security and risk management (SRM) leaders evaluating SIEM solutions must understand their use cases and then define specific requirements in conjunction with applicable stakeholders and company strategy in general [8].
On the other hand, organizations must require a structured approach for managing their challenges. This will ensure that there are agreed objectives, good management controls in place and effective monitoring of performance to keep on track and avoid unexpected outcomes. Therefore, this paper proposes not just technological but pragmatic approach to support companies that are H. Mokalled  and to suggest criteria to judge SIEM systems using an evaluation process composed of quantitative and qualitative methods. This approach proposes the requirements that should be addressed in a SIEM solution in a systematic way, and then proposes a methodology for evaluating SIEM solutions that measures the compliance and applicability of any SIEM solution using quantitative and qualitative methods. The goal is to select the appropriate SIEM that fits best in company's environment. However, and because of the complexity, precision and thoroughness required to apply our whole approach, it is mainly dedicated to large enterprises, which include wide variety of broad and specific skills and several specialists to manage certain applications or parts of the IT infrastructure, and most of them comprise a dedicated department to manage information security. Therefore, this approach can be followed by those bigger enterprises that in general tend to manage their work in a very structured manner, and they need to assert successful management and performance, and this is our goal, to aid in following a thorough approach for the issue. This work represents the first and primary phase in the procedure of choosing a SIEM or a set of qualified SIEMs, however, it has to be followed by a testing phase that enable the customer to check the solution directly after the installation. The remainder of this paper is structured as follows: after this introduction, Section 2 will briefly report a background about main high-level features of SIEM systems and related works.
In Section 3, we present our proposed approach. Section 4 demonstrates how this approach is applied to select one solution among a set of SIEMs proposed and received by different suppliers, through defining both technological and organizational requirements that the customer seeks to have in the SIEM and evaluating the received SIEMs using the proposed quantitative and qualitative evaluation methods. After that, Section 5 compares this approach with similar existing work. Finally, Section 6 presents the conclusions. involved, such as a syslog server; while agent-based means that an agent is installed on a source-log to gather security events from the endpoint itself. Today, most SIEM systems work by deploying multiple collection agents (collectors) in a hierarchical manner. Log collectors forward events to a centralized management console, which performs inspections and flags anomalies [4]. Then after collection, the data should be normalized so it can be correlated and analysed.

SIEM System: Definitions
Another feature is the pre-filtering that is related to processing center, some systems use a pre-processing mechanism at the edge collectors, with only certain events being passed through to a centralized management node. In this way, the volume of information being communicated and stored can be reduced. SIEM technology provides near real-time correlation of events for security monitoring, query and analytics for historical analysis and other support for incident investigation, compliance reporting, and alerting [11]. According to Gartner, by 2020, 75% of all SIEMs will use big data technologies at their core, along with machine learning, to improve threat detection and response capabilities [8]. In short, there are so many SIEM systems in the market created by skilled and expert security vendors with their own features; however, selecting the suitable SIEM is not a trivial task anymore: it is not simply about installing the most powerful one, for instance, a very powerful SIEM may be too complex to apply in some cases.

Related Works
Several studies were conducted in the field of "How to select a SIEM system". Gartner reports are an excellent example where they present a detailed evaluation of the current SIEM products based on many characteristics such as sales execution, pricing, costumer experience, marketing message evaluation against the understanding of customer needs [8] [11] [12]. In [11], authors examined different SIEM products that are the leaders of SIEM technology, they focused on technical requirements and showed strengths and cautions for each vendor and at the end they defined evaluation criteria from an ability-to-execute point of view. According to authors in [8], a list is defined containing the critical capabilities that a SIEM should include, and they suggested three different and general use cases to use in the evaluation: Basic Security Monitoring, Complex Security Monitoring, Advanced Threat Defence. They used an evaluation criterion to evaluate the most powerful SIEM products available, which is based on the defined critical capabilities and the three suggested scenarios. [13] provided an environment-specific criteria to benchmark SIEM solutions, where organizations should consider factors like Events-Per-Second (EPS), considering the number of employees in each sub-net, number of databases, and the ability to store and analyse these events, in order to evaluate and even design a SIEM system. [14] proposed an after-installation evaluation approach: such an approach may be time-consuming for companies, and it should be preceded by a PRE-installation evaluation approach that qualifies and select the applicable SIEMs from the plethora of solutions available before installing them. The common point between the above-defined related-work is that all of them are product driven, where the evaluation of the solution is for the product as a technical solution. However, our approach is customer driven, where the selection phase is not only based on the technical features of the product but also subject to pre-defined customer's needs.

A SIEM Selection Approach: Requirements and Evaluation
What characterizes our work is that it proposes an overall approach for the problem of selecting the applicable SIEM, and searching the previous work will show how few, if no similar comprehensive approaches were proposed. Some of the work done focused just on the technical requirements without addressing the organizational ones, and other aspects. Others did not consider the problem of applicability or integration in the environment. This approach, unlike others, is customer driven which means that customer needs are taken into account when following the approach, specifically when defining the requirements and then evaluating the suppliers' solutions. Saving time, using a systematic-organized strategy for decision-making and balancing costs to needs are the main advantages of adopting such an approach. This approach starts by suggesting the requirements (technical and organizational) that should be addressed in a SIEM solution in a systematic way, and then proposes a methodology for evaluating SIEM solutions that measures the compliance and applicability of any SIEM solution using quantitative and qualitative methods. This evaluation methodology is divided into two phases: 1) Quantifying each requirement of the received SIEM solution using a quantitative based method; 2) Measuring the applicability of the solution using a qualitative based method after defining a list of indicators that enables the evaluation of this applicability.
The goal is to select the appropriate SIEM that fits best in company's environment and resources; however, we stress that a final installation-testing phase must be accomplished with the suppliers to make sure about the compliance of the selected solution to the needs addressed. The time complexity of such an approach depends on more than one factor: First, since this approach is customer driven, it depends on the specification that the customer includes while setting the requirements, as larger enterprises might require in-depth specification of requirements. In addition, the evaluation phase of the received SIEM solutions affects the time complexity of this approach also. It is recommended to adopt a cost-benefit analysis to select a set of the relevant and optimal (or even sub-optimal) solutions among the received solutions as some solutions might be absolutely complex and hard to implement in the customer's environment or they could cost more than the limit available, so there would be no need to waste time on the evaluation phase for those proposals. And this is taken into account in this approach when using both a qualitative and quantitative methodologies to evaluate.

Technical and Organizational SIEM Requirements:
Defining requirements is an important task; it helps the companies to define their needs, be aware of any shortage, and aids them in the evaluation phase. Information security and risk management leaders responsible for security operations should focus their evaluation on the critical capabilities that align with their use cases, requirements, and current and future IT environments, e.g. on-premises versus cloud-based services [8]. This section groups the requirements needed to adopt a SIEM platform covering in detail the "mandatory" and "nice-to-have" requirements. Those requirements represent the needs of the customer, they cover all the features and services that the supplier should include when proposing a SIEM, they are divided into 5 sections: platform, operations, integration, advanced features and licensing-support services, as listed in Table 1.

Measuring the Compliance and Applicability of a SIEM: An Evaluation Process
Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished [15]. It can assist an organization to assess and help in decision-making; or to ascertain the degree of achievement or value about the aim and objectives and results of any action. Evaluation is methodologically diverse; two types of methods may be qualitative or quantitative. Quantitative methods are distinguished by emphasis on numbers, measurement, experimental design, and statistical analysis [16], and hopes the numbers will yield an unbiased result that can be generalized to some larger population. However, qualitative methods evaluate other parameters such the success and the eligibility of a product in a specific environment using non-numerical (textual forms) data to assess the eligibility and reliability of adopting the solution, such as the use of internal discussions, interviews, comparisons to provide feedbacks, etc. Both quantitative and qualitative evaluation methods have their benefits, quantitative evaluation can help remove human bias, thus more accurate. However, qualitative evaluations may also involve truths, but these truths are harder to get at, and evaluators may not always agree. In our approach, an evaluation process is proposed; it is applied after receiving the description of the SIEM solution from suppliers. It is divided into two methods: quantitative and qualitative. The first method "Requirements-based Evaluation" is the quantitative side of the evaluation process; it evaluates the degree of compliance for each requirement of the received SIEM solution using numerical values and mathematical operations. This method is applied to the SIEM solutions that might be adopted and used to obtain a list of qualified ones as an output. After that, the output of this method is then provided as input to the second method ( Figure 1).
On the other hand, the second method "Applicability Evaluation" represents the qualitative side of this evaluation process, it focuses on the observations, H. Mokalled et al.   lutions and finally to select the most applicable one. Figure 1 shows how to apply the approach starting by defining the requirements that the customer seeks in their SIEM product, then evaluating the received ones using the proposed evaluation process described in the next section.

Compliance Measurement: A Quantitative Requirement-Based Evaluation Method
The first evaluation method measures the degree of compliance of each requirement in the SIEM solution that might be adopted, it is a first step evalua- nice-to-have ones. Then, a score (S) is calculated for each requirement (S = V * W), and after that a total for each requirements section is computed. Finally, the total score is obtained by adding all the totals corresponding the requirements family sections. Table 2 is an example used in applying this evaluation method.
For a better evaluation, a scale is suggested to represent the requirement value (V). This scale is used to differentiate between an insufficient, good, very good, and excellent requirement proposed by the supplier, by translating the level of compliance of the under-evaluation requirement into a numerical value. A non-linear growth scale is suggested to be used because of its ability to differentiate between the values using the high growth rate.
At the end, evaluators may define a passing grade to use to select the qualified "under-evaluation" SIEM solutions, so they can directly reject a solution with lower total score. The output of this method should be a set of accepted SIEM solutions which all complied the defined requirements, but at the end one solution should be adopted, and this is the role of the second evaluation method, which examines the applicability.

Applicability Evaluation: A Qualitative Method
SRM leaders increasingly seek SIEMs with capabilities that support early targeted attack detection and response. Users must balance advanced SIEM capabilities with the resources needed to run and tune the solution [11]. The best SIEM system for an organization may not be suitable at all for another. Other variations should be considered along with the technical side when evaluating a SIEM solution. Therefore, the qualitative side of the approach takes place; it is about examining the whole solution in terms of applicability rather than measuring mathematically the value of each requirement. The highest-grade solution is not always the choice, it may have powerful features, but too complex to install, or even too expensive. This method aims to evaluate the qualified SIEM    (Table 3).

Applying the Approach
The approach presented in this work was applied at Hitachi Rail STS Company.
Hitachi Rail STS is a leading Company operating in the sector of high technology for Railway and Urban Transport [17]. To select a SIEM solution, the Company believes that a structured and systematic procedure should be followed, in an organized manner, which is the way Hitachi Rail STS tends to manage its challenges, i.e. ensuring that there are agreed objectives, good management controls in place to keep on track and avoid unexpected outcomes. Another main reason for selecting and adopting such an approach is to provide enhanced compliance and requirements coverage in bids.

Creating a Request-for-Proposal (RFP): Specifying SIEM Requirements
Specifying requirements is the first step while applying this approach. Therefore, a request-for-proposal (RFP) document (containing all the requirements that are believed to be mandatory or nice-to-have in the SIEM quest) was prepared and o) Alerting capabilities: Capability of triggering alerts, e.g. sending a notification message or email, and so to respond to incidents. p) Technical documentation and online help: Availability of technical documentation, both with offline and online help. q) Monitoring: The SIEM platform should be monitored using any standard protocol (e.g.: SNMP) so it can be added in the company's monitoring platform. r) Secure Software: The supplier should state the SIEM platform operating system and version along with the "secure by design" techniques adopted. s) Context enrichment based on collected logs: Availability of correlation rules to gather and merge information from the different log sources, to be able to provide all the info of the array [Mac, IP, hostname, username] whenever available somewhere in the logs. t) Support for collection of real-time and deferred logs: The supplier should provide details about the support, normalization and indexing of logs retrieved in real-time and deferred way (e.g. sent by batch jobs). u) Multi-tenant capabilities (views): The supplier should provide details about the SIEM solution capability to display some views based on connector grouping (e.g., connectors associated to different geographical entities or based upon management responsibilities like systems, DBs, network or security devices 2) Operations: a) Role-based access control: The platform has to implement a role based access control mechanism suitable by the configuration of multiple user profiles owning different privileges to implement the accountability and separation of duties principles. b) Accounting: The SIEM platform must have an audit log facility in order to track the activity relevant from the security perspective performed by operators. c) Web interface for day-by-day operation: The SIEM platform interface used by users for daily analysis has to be web-based. d) Customizable time-zones for the GUI: The interface must allow the user to choose in which time zone all the data must be displayed.
3) Integration. This section groups the requirements needed to integrate the SIEM solution into the Company's information system. a) Active Directory integration for administrative management: The platform access must be granted only to qualified users authenticated and authorized via the Company's Active Directory database. b) Integration with asset management tools: The ability to integrate the solution with asset management standard tools (e.g. Configuration Management Data Base). c) Case Management and trouble-ticketing activities tracking: The ability to manage incident handling issues and the conditional support of standard IT trouble ticketing systems (workflows, prioritization, KB email exchange). d) Trouble ticketing module: The supplier should provide details in case of the availability of a trouble ticketing system with the SIEM. e) Integration with vulnerability management tools: A nice feature is the ability to integrate with vulnerability management tools. 4) Advanced features. This section describes the advanced features, they could be considered as nice-to-have requirements.
a) Threat Intelligence analysis tools support: Availability of threat analysis tools is a plus if already available by the vendor and by using standard formats for exchange such as: STIX, TAXII, IoC, other standard formats. b) Support for forensics analysis activities: Additional value if forensics analysis activities are available (file integrity monitoring, pcaps, NetFlow, evidence acquisition). c) Analytics support: Additional value if there is a support for anomaly detection, use and entity behavioral profiling. d) Automatic response capabilities: Additional value if there is a support for automatic response capabilities (e.g. SOAR: security orchestration automatic response). c) Project Roadmap: Describe the tasks involved with the project of the SIEM platform installation and configuration and the corresponding timeframes. d) Delayed license activation: The license activation should start only after the end of the acceptance tests in which all the project requirements will be met. The customer must ask for an acceptance-testing period to verify that the solution complies with the received description. e) Technical assistance support and professional services: The supplier should include a description of the including technical support and professional services provided/available by vendor/system integrator.

Evaluating the Received SIEM Solutions
Different solutions were proposed by a set of suppliers according to their expertise in the field and using the most powerful SIEM products available in the market nowadays. In this case study, only three solutions are selected to apply the proposed approach, which are believed to be enough to show and verify the efficiency and effectiveness of our approach. However, for the purpose of not being subjective or promoting a solution, the study will not mention any supplier or SIEM product name, and instead it will anonymize their names and use the following notation for them: 1) Supplier 1 using Product X 2) Supplier 2 using Product Y 3) Supplier 3 using Product Z Where the "supplier" is the one who provided the whole SIEM solution (product, architecture, installation, licenses, training, etc.) and the "product" is the name of the innovator who created the SIEM product.
After receiving the tenders from the suppliers, describing their overall solutions, the next step in the approach is to apply the evaluation phase. Evaluation is divided into two methods, requirements-based (quantitative) and applicability-based (qualitative), where the first aims to qualify a set of the best in terms of matching and complying the requirements specified by the customer, and the latter aims to select only one solution that best fits the company. And so, the evaluation done next is based on the whole solution offered, which includes the proposed architecture (topology for installation and deployment), kind of the platform, complexity, technical features, licensing, etc…, and it is not only an evaluation for the tool or the vendor that develops it. Again, the evaluation applied using this approach is from a customer point of view, ensuring the selec-Journal of Information Security tion of the most appropriate and applicable solution based on the needs and context.

Requirement-Based Evaluation: Using a Quantitative Method
According to the received tenders, only three solutions were selected to demonstrate the evaluation of our approach, which we believe are enough to show the effectiveness of such an approach, knowing that they used the most powerful SIEMs existing in the market nowadays that are developed by well-known, skilful and expert vendors.
The approach assigns each requirement two different parameters, which are the weight (W) and the requirement value (V):  The weight (W) represents the importance of current requirement in the solution from the user point of view, and is assigned initially when the customer defines his requirements, for example, a mandatory requirement has a high value compared to nice-to-have ones.  The requirement value (V) is the grade assigned to evaluate the answer of the supplier to the current requirement in the under-evaluation SIEM solution.
Then, a score (S) is calculated for each requirement (S = W * V), and after that a total for each requirement section is computed, and finally a total score is computed for the whole solution.
The approach suggested to use a non-linear growth scale because of its ability to differentiate between the values using the high growth rate, and so requirement value will get a non-linear value as suggested in the approach and will vary between 0, 1, 2 and 4 corresponding a zero, low, medium and high requirement evaluation. On the other hand, the weight will vary between zero and one, where one represents a critical requirement.
Therefore, to evaluate those solutions, a quantitative evaluation is used to show the requirement-based evaluation. Tables 4-6, present the quantitative evaluation of the received SIEM solutions.
 Supplier 1, product X: Supplier 1 offered a solution based on a product X, the solution composed of two layers: 1) Collection layer that uses collection nodes to collect the logs, and to parse and normalize.
2) Processing layer for data storage and correlations based on specific rules. In addition, the supplier offered a solution that can be deployed using two alternative architecture for deployment: 1) All-in-One deployment, where all layers are within a single node.
2) Distributed deployment, that consists of multiple nodes, composed of multiple nodes.
The solution covers most of the requirements, e.g. indexing, normalization, compression, and hashing for tamper proof, and it is able to parse a lot of sources and supports also custom parsing. Moreover, it is able to store events/logs locally for a long period. In addition, based on the proposed architecture of deployment, high availability is preserved.    The solution proposed by supplier two uses product Y. It has a modular and distributed architecture, and is composed of two layers: 1) A layer that groups collection and standardization, it can receive logs of any type of data source within the network without the need to install agent designed for each specific platform.
2) A layer for correlation on the main appliance. The solution uses a hardware-based appliance (on premise or in cloud) that is able to support continuous traffic.
However, the solution lacks a lot of the main requirements such as indexing, hashing, normalization, customizable dashboard per user, and for compliance reports, etc…  Supplier 3, product Z: This solution is divided into two layers: collection and correlation. 1) Collection layer: composed of software instances (agents) where each agent works on a source.
2) Correlation layer: a copy of the collected logs is sent to this layer to be processed; it consists of one hardware and is able to handle fair number of EPS.
The solution ensures high availability through deploying redundant appliances. Other main requirements that this solution covers are indexing, hashing and normalization. However the license is activated at the beginning of the deployment process and there is no delayed license.
Therefore, applying the quantitative evaluation on the received SIEMs will output an evaluation (total score) for each solution. At the end two solutions are selected as qualified SIEM solutions, where only one of them will be adopted finally. In the next section, applicability evaluation is applied to select one of the two qualified SIEM solutions.

Applicability Evaluation: Using a Qualitative Method
As said before, not always the solution that got the highest score in the requirement-based evaluation will be adopted; however, as explained, there are other factors and indicators that should be considered, and they are represented in the applicability evaluation.
After evaluating the SIEM solutions quantitatively, the next step is to apply the qualitative evaluation (applicability evaluation). In our case, the applicability evaluation is applied to choose between the two qualified solutions (by supplier 1 and supplier 3). Table 7 and Table 8 show the applied qualitative evaluation:

Comparison
To prove the eligibility of using our approach we select Gartner's work to compare with. Gartner Inc. is one of the leading information technology research and advisory company that deliver the technology-related insight necessary for clients to make the right decisions, research, analyze and interpret the business of IT within the context of their individual role. Gartner published its 2018 Critical Capabilities for Security Information and Event Management report [8] to evaluate and rank different SIEM vendors (products). First, they started by specifying the critical capabilities that a SIEM should have. Critical capabilities are architecture, deployment, operations and support, log and data management, real-time monitoring, analytics, data and application monitoring, threat and environmental context, user context and monitoring, incident management and threat detection tools.
After that, they defined three use cases: basic security monitoring, complex security monitoring, and advanced threat defense, where the SIEM vendors are evaluated in each use case. These use cases used represent three "operational" levels in which the SIEM will be deployed. Journal of Information Security Critical capabilities are then weighted, where each capability is given a weight in terms of its relative importance in each use case. In addition, for each SIEM product, the critical capabilities are rated on a scale of 1 to 5, where 1 represents a poor requirement, whereas 5 is an outstanding (significantly exceeds requirements). Finally, an overall score is carried out for each SIEM on the three different use cases. To determine an overall score for each product in the use cases, the ratings (evaluations) are multiplied by the weightings to come up with the product score in use cases. Different SIEM products developed by the most expertise and skilled vendors are used in the evaluation applied by [8], where each vendor's product or service is evaluated in terms of how well it delivers each of the capabilities defined. This kind of evaluation can be defined as a product driven evaluation, where the capabilities of the product are evaluated without differentiating the customer's needs factor.
However, our approach suggests the customer to create a more detailing list of requirements that reflects their needs, which is a subjective and detailed list representing the SIEM seeking for, therefore, to receive an overall solution and not just a product, and so the evaluation will be influenced by these specific requirements.
Moreover, we defined a second level of evaluation, which is a qualitative evaluation method (applicability based) that selects from the qualified solution evaluated previously by the quantitative evaluation method (requirement based) ( Table 9).

Conclusion
In conclusion, organizations tend to ensure good management controls are in place to avoid unexpected outcomes and to keep on track, so they require a structured approach for managing their tasks. This paper proposed a thorough approach to support companies that are seeking to adopt SIEM systems into their environments; it suggests suitable technological and business requirements that are believed to be valuable in a SIEM system and proposes a two-phase Table 9. A comparison between the proposed SIEM evaluation approach and Gartner's report.