A Fault-Based Testing Approach in Safety Critical Medical Systems

The advent of technology has opened unprecedented opportunities in health care delivery system as the demand for intelligent and knowledge-based systems has increased as modern medical practices become more knowledge-intensive. As a result of this, there is greater need to investigate the pervasiveness of software faults in Safety critical medical systems for proper diagnosis. The sheer volume of code in these systems creates significant concerns about the quality of the software. The rate of untimely deaths nowadays is alarming partly due to the medical device used to carry out the diagnosis process. A safety-critical medical (SCM) system is a complex system in which the malfunctioning of software could result in death, injury of the patient or damage to the environ-ment. The malfunctioning of the software could be as a result of the inadequacy in software testing due to test suit problem or oracle problem. Testing a SCM system poses great challenges to software testers. One of these challenges is the need to generate a limited number of test cases of a given regression test suite in a manner that does not compromise its defect detection ability. This paper presents a novel five-stage fault-based testing procedure for SCM, a model-based approach to generate test cases for differential diagnosis of Tuberculosis. We used Prime Path Coverage and Edge-Pair Coverage as coverage criteria to ensure maximum coverage to identify feasible paths. We analyzed the proposed testing procedure with the help of three metrics consisting of Fault Detection Density, Fault Detection Effectiveness and Mutation Adequacy Score. We evaluated the effectiveness of our testing procedure by running the suggested test cases on a sample historical data of tuberculosis patients. The experimental results show that our developed testing procedure has some advantages such as creating mutant graphs and Fuzzy Cognitive Map Engine while resolving the problem of eliminating infeasible test cases for effective decision making.


Introduction
To investigate the pervasiveness of software faults in Safety critical medical system, a fault-based testing approach is desired. The emergence of technology has opened unprecedented opportunities in health care delivery system as the demand for intelligent and knowledge-based systems has increased as modern medical practices become more knowledge-intensive [1]. As a result of this, there is a greater need to investigate the pervasiveness of software faults in safety critical medical system for proper diagnosis. A safety-critical medical system is a complex system in which the malfunctioning of software could result in death, injury of the patient or damage to the environment. The rate of untimely deaths nowadays is alarming partly due to medical device used to carry out the diagnosis process. Not surprisingly, software quality is a serious and growing problem. To scope with the limitation of software testing, a five-stage fault-based testing approach has been proposed in this paper that does not only generates test cases but equally a Fuzzy Cognitive Map (FCM) Engine for the system under test. In view of this, mutation analysis is used as a technique to assess the quality of a test suite.
In recognition of the fact that exhaustive testing is impossible in any software development project, the need to have a methodical means to improve on the available diagnostic and treatment modalities for the system under test in order to reduce the rate of mortality, a diagnostic system is desired. This paper presents an innovative five-stage testing procedure for a fault-based approach in Safety Critical Medical System and the system under test is a diagnostic system for the management of Tuberculosis (TB).
Computer based techniques that employ fuzzy Cognitive maps have been extensively examined for improving diagnosis and treatment of diseases [1] [2] [3] [4] and until today remains an active research area because the merging of mutation analysis and FCM to address the oracle problem to determine successful test cases is still an unresolved issue.
The remainder of this paper is organized as follows. Section 2 provides a background on mutation analysis and fuzzy cognitive map. Section 3 provides an overview of our approach, and Sections 4 and 5 present the experimental setup and evaluation of the research. Finally, the paper concludes with a discussion of future work.

Related Work
In this section, we present a categorized survey of the background concepts mutation analysis and Fuzzy Cognitive Maps FCM). These related works are classified according to the difference strategies for their test case generation.

Mutation Analysis
This paper proposes the use of mutation analysis to design effective test cases for safety critical medical systems for the management of TB. Mutation testing modifies a software artefact such as a program, requirements specification, or a configuration file, to create new versions called mutants [5]. The mutants are usually intended to be faulty versions and are created by applying rules for changing the syntax of the software artefact. These rules are called mutation operators [6]. The tester then creates tests that cause the original and each mutated version to exhibit different behaviours, called killing the mutant [9]. Several researches have been conducted on mutation testing to discover faults in some programming languages [7] [8]. To the best of our knowledge, this is the first attempt to define mutation operators for safety critical medical systems that attempt to merge the concepts of mutation analysis and fuzzy cognitive maps.

Fuzzy Cognitive Maps
The concept of a fuzzy set was originally proposed by [10] as an extension of the notion of a set by allowing partial membership, and the usage of fuzzy sets theory in medical applications can be traced back early to work by [3] [4] who advocated and put the foundations of the theory to model relationships of symptoms and diseases by using the compositional rule of inference (CRI) as an inference mechanism. Fuzzy Cognitive Maps FCMs, which were introduced by (Kosko, 1986 [10]) as an extension of Cognitive Maps, are powerful tools for modeling dynamic systems. A Fuzzy Cognitive Map (FCM), as a branch of fuzzy logic [11] [12], is a causal knowledge-driven methodology for modeling complex decision systems originally developed by [13].
Few frameworks based on fuzzy cognitive maps for medical reasoning have been proposed [1] [2] [3] [4]. Due to the fact that, the complexities of medical practice make traditional approaches of analysis inappropriate, this multi-expert approach presented by these researchers would have efficiently aid in medical decision but for the fact that they did not base on a model-based approach to generate test suits while maintaining accuracy and interpretability. "Improving accuracy while maintaining interpretability" is the main focus of a Clinician.
We proposed that the performance of a FCM Engine can extensively be increased by merging the FCM model with mutation analysis for effective decision making.
In a recent research, a fuzzy expert system for tuberculosis diagnosis was developed [1]. The main focus of the work was accuracy and no doubt, the researchers would have achieved "high peak" accuracy. But a physician involved in sensitive decision making about a patient's treatment, demands more than that.
Factors including interpretability, system's ability to adopt human reasoning to deal with uncertainties, identification of successful test cases, mutation operators for safety critical medical systems and performance consistency were ignored.
While [14] attempted and suggested a novel fuzzy system for TB management, Journal of Software Engineering and Applications no test cases were generated and no metric was used to evaluate the system. According to our approach, the performance of the FCM model was improved by developing new mutation operators to remove redundant test cases and the identification of suitable test cases for effective decision making.
The review of related literature shows that there is a painful lack of diagnostic predictive models for TB management and no explicit attempts are made to merge mutation analysis and Fuzzy Cognitive Maps to address the test suite problem for proper medical decision.

Materials and Methods
For data gathering, historical patients' medical records on tuberculosis were sourced from Urgences-CHE CNPS Yaounde, Cameroon from 2018 to 2019. Personal interview was conducted with domain experts (Physicians) in order to get the necessary data for this study. We sampled a total of 200 patients' medical records. The method of sampling strategy adopted was systematic sampling method.
We proposed a five stage fault-based testing procedure and analyzed the proposed testing procedure with the help of three metrics consisting of Fault Detection Density, Fault Detection Effectiveness and Mutation Adequacy Score.

Fault-Based Testing Procedure
Our proposed testing procedure consists of the following five stages:  (Table 1,   Table 2).

1) Procedure to Generate Linguistic Weights for FCM Engine
The procedure to generate linguistic weights that describe the cause-effect relationships among the concepts of the FCM engine was developed and is presented below: Step 1: For all N Domain Experts, set credibility weight e ij in the interval [−1, 1].
Step 2: For all ordered pair of concepts (C i and C j ), N Domain Experts describe the interrelationship between concepts using IT-THEN rules derived from Journal of Software Engineering and Applications Step 3: If causality occurs, it occurs to Maximum Positive or Maximum Negative.
Step 4: Using Fuzzy Ranking Values, the Fuzzy Maximum is applied and a linguistic weight is assigned between concept C i and concept C j .
Step 5: IF for two or more interconnections between concept C i and concept C j , more than 3N/4 Experts assigned difference linguistic weights, THEN request the Experts to GOTO step 2 and reassign weights for that particular interconnections.
Step 6: Aggregate all the linguistics weights proposed for every interconnection using the SUM method and construct the Weight Matrix.
, such that v ij is one (1) if t i identifies fault and Zero (0) if t i does not identifies fault.
Different mutants can be created to discover the behavior of each concept in every state of the program. In this approach model-based mutation testing is used to generate mutant graphs by applying mutation operators. In our testing procedure, we need mutation operators to add vectors, update vector, normalized vector to the FCM Engine. Since the operators, such as delete node, add node, delete edge, or add edge, cannot create a new vector/concept/node, add node, update node and normalized node are considered as the mutation operators. As a result, new edges can be produced by changing the input and output parameters of each state vector (C1-C12 represent input parameters) while eliminating the infeasible test cases and detecting all redundant test cases. According to the number of created edges, the output of this third stage can be one or more mutant graphs ( Figure 1) to be used for the FCM Engine.

Selection of Coverage Criteria and Production of Test Paths
The goal of the third stage is to produce test paths from the Mutants graphs. At this stage a proper coverage criterion (Table 3) and an automatic tool are required. As shown in Figure 2 Table 1. The 13 identified concepts with 13 fuzzy sets keep relation with each other, in order to characterize the process of diagnosing tuberculosis. After the determination of fuzzy sets, each expert was asked to define the degree of influence among the concepts and equally describe their interrelationships using IF-THEN rules derived from fuzzy decision tree and the inferred fuzzy weights were combined and normalized using sigmoid function and the result was a crisp value representing the weight of each interconnection. In this way, the weights of interconnections between the concepts were determined.
According to domain experts' judgments, input variables selected for this re-  Table 2.
From the FCM Engine (Figure 3), an Overall Weight Diagnostics Matrix (M OverallWeight ) with 13 rows and 13 columns was computed as shown in Table 4 below.

Deriving and Running Test Suite
All the test paths that are extracted from mutant graphs and FCM Engine are inputs to the final stage to generate test cases. The test cases are executed, the actual results are compared with the expected results and test oracle is established as shown in Table 5. The output of the final stage is represented as a list of faults arising from running test cases.

Dividing the Application Based on Its Structure
For this stage of our testing procedure for the safety critical medical system, by analyzing the concepts (input variables) the main diagnostic structures in the system are presented in Table 1.

Creating a Mutant Graph from Application Structure
Considering the add node, update node and normalized node as the mutation operators for the safety critical medical system, 76 mutants were created (for both diagnosis and treatment) and 14 mutants were killed as shown in Table 6 and Figure 1 shows a sample of the mutant graphs. The mutants were killed by EPC and PPC test sets. Equivalent mutants were identified by hand analysis. Table 7 shows the results for each mutation operator.

Selection of Coverage Criteria and Production of Test Paths
A sample path can be seen in Table 5 with the corresponding test case. We used Prime Path Coverage and Edge-Pair Coverage as coverage criteria to ensure maximum coverage to identify feasible paths. Journal of Software Engineering and Applications

Create a Fuzzy Cognitive Map Engine
By simulating a safety critical medical diagnostic system, our FCM Engine performance is illustrated by means of simulating two scenarios of tuberculosis infection as a means to diagnose and manage a TB patient. In each of these scenarios, Thus the initial state vector (Input Vector) V Initial is: Thus the initial state vector (Input Vector) V Initial is: After the fuzzy inference process (simulation process), at the eleventh itera-

Deriving and Running Test Cases
By applying our developed testing procedure on safety critical medical system, 1778 test cases were produced. An example of a test case document is shown in Table 5.

Evaluation
In this study, we used the following three performance metrics consisting of The mutation score for the case study is represented in Table 7 and Table 8 presents the values of FDD, FDE and the unique faults for the stages of the developed testing procedures.

Conclusions and Future Work
This paper presents an innovative approach to fault-based testing in Safety Critical Medical Systems. We introduced a new five-stage testing procedure that has some advantages: 1) creating testing cases while resolving the problem of removing infeasible test cases, 2) creating FCM Engine for our case study because without sound diagnosis and accurate treatment, medical practice is as good as guess work.
While the research is promising, several research questions remain unans- wered. An important evaluation, currently being planned, is to do a full fault study. We will generate tests to kill all non-equivalent mutants, then evaluate those tests to determine how many faults the tests detect, and compare with tests generated for other criteria (possibly statement or branch coverage).

Threads to Validity
Our empirical evaluation has some threats to validity. Firstly, equivalent mutants were identified manually by one person. Secondly, our implementation of the three proposed mutation operators may include faults. To ensure they work as expected, we tested our suggested testing procedure constantly, and checked mutants generated by hand very carefully. Thirdly, like most software engineering experiments, it is not possible to guarantee the representativeness of selected subjects,