Study and Analysis of Defect Amplification Index in Technology Variant Business Application Development through Fault Injection Patterns

Software reliability for business applications is becoming a topic of interest in the IT community. An effective method to validate and understand defect behaviour in a software application is Fault Injection. Fault injection involves the deliberate insertion of faults or errors into software in order to determine its response and to study its behaviour. Fault Injection Modeling has demonstrated to be an effective method for study and analysis of defect response, validating fault-tolerant systems, and understanding systems behaviour in the presence of injected faults. The objectives of this study are to measure and analyze defect leakage; Amplification Index (AI) of errors and examine “Domino” effect of defects leaked into subsequent Software Development Life Cycle phases in a business application. The approach endeavour to demonstrate the phasewise impact of leaked defects, through causal analysis and quantitative analysis of defects leakage and amplification index patterns in system built using technology variants (C#, VB 6.0, Java).


Introduction
Formulating reliable and fault tolerant software is difficult and requires discipline both in specifying system functionality and in implementing systems correctly.Approaches for developing highly reliable software include the use of formal methods [1][2][3], and rigorous testing methods [4].
Testing cannot guarantee that commercial and business software is correct [5], and verification requires enormous human effort and is subject to errors [6].Automated support is necessary to help ensure software correctness and fault tolerance.
Fault injection modelling involves the deliberate insertion of faults or errors into a computer system in order to determine its response.It has proven to be an effective method for measuring and studying response of defects, validating fault-tolerant systems, and observing how systems behave in the presence of faults.In this study, faults are injected in key phases of software development of business application following a typical water fall software life cycle viz., SRS, Design and Source code.

Literature Review
The literature review consolidates the understanding on fault injection, associated topics and subsequent studies to emphasis the need to fault injections in business software application.It also crystallizes the need for awareness, tools and analyzes defect leakage/amplification.
Even after 20 years of existence the awareness of fault injection and associated modelling with tools are very rarely used and understood in the commercial software industry and used.The usefulness in the defect modelling and building fault tolerant software systems are not properly preached and/or practiced.Added, the availability of appropriate literature and software tools is very few and not used in commercial and business application design and testing.
After a detailed review of literature by the researcher it was concluded that there is an industrious interest software fault injection in the software industry to develop commercially reliable software.

Approach
In recent years there has been much interest in the field of software reliability and fault tolerance of systems and commercial software.This in turn has resulted in a wealth of literature being published around the topic, such as the Development through Fault Injection Patterns Fault Injection in the form of the 'Marrying Software Fault Injection Technology Results with Software Reliability' by Jeffrey Voas, Cigital Norman Schneidewind.
Many critical business computer applications require "fault tolerance," the ability to recover from errors or exceptional conditions.Error free software is very difficult to create and creating fault tolerant software is an even greater challenge.Fault tolerant software must successfully recover from a multitude of error conditions to prevent harmful system failures.
Software testing cannot demonstrate that a software system is completely correct.An enormous number of possible executions that must be examined in any real-world sized software system.Fault tolerance expands the number of states (and thus execution histories) that must be tested, because inconsistent or erroneous states must also be tested.
Mailing lists, websites, research and forums have been created in which all aspects of this fresh new niche software engineering area are discussed.People are interested, partly because it is a new area but also because the whole field of commercial software reliability is in itself so interesting; as it holds so many wide ranging disciplines, perspectives and logic at its core.Software reliability engineering is uniting professionals in disciplines that previously had little to do with one another, it is creating more opportunities for employment in the online environment, and it is changing the face and structure of all information that we seek out on the web.In the era of economic recession, customer demands reliable, certified and fault tolerant commercial and business software applications.
In this research, the focus is on software testing techniques that use fault injection.Several potentially powerful existing systems have drawbacks for practical application in business application development environment.We first examine existing fault injection techniques and evaluate their potential for practical application for commercial and business software applications.Available and accessible literature infrastructure including premium subscribed IEEE and ACM resources were studied and summarized for literature review from 1986 (20 years).

Fault Injection Modelling
Fault Injection Modelling (FIM) involves the deliberate insertion of faults or errors into a computer system in order to determine its response.It has proven to be an effective method for measuring and studying response of defects, validating fault-tolerant systems, and observing how systems behave in the presence of faults.In this study, faults are injected in all phases of software development life cycle viz., Requirements, Design and Source code.

Objectives
The objectives of conducting these experiments are to measure process efficiencies, statistically study, analysis and report defect amplification of defects (Domino's effect) across software development phases with a similar system constructed with technological variation.
The goal of this research is to understand the behaviour of faults and defects pattern in commercial and business software application and defect leakage in each phase of application development.
Throughout the literature certain questions reoccur, which one would anticipate when a new field emerges in commercial software fault tolerance?People are interested, and want to understand and define commercial software reliability and fault tolerance since the work on most fault injections and software reliability is found in life critical and mission critical application, so we try to answer the following questions;  What work was performed, current status and work proposed?These questions will be expanded upon throughout the research, and seek to bring clarity to those who want to find the answers to the above, or to see if there truly are any answers!

Background Concepts
A fault is a hardware or software defect, inconsistency, transient electrical field, or other abnormal circumstance.An error is an invalid internal state, which may or may not be detected by the system.
A failure is an invalid output.Thus a fault or error becomes a failure when it propagates to the output.There is a natural progression from fault to error to failure.Recovery code is the part of a program that is designed to respond to error states.Recovery code executes after the program recognizes that some erroneous or abnormal state has been entered.This code should gracefully restore the system to a valid state before a failure occurs.
Figure 1 shows the progression from faults to errors and finally to failures.The recovery code should serve as a safety net to prevent the progression from error to failure.A fault tolerant system should never fail, even if it has faults.
Testing recovery code requires the modeling of bad states that accurately simulate exceptional situations.As much as 50% of a fault tolerant program can consist of recovery code.Although testing might include invalid data that executes some of the recovery code, often much of this code is never executed during normal testing.
Any recovery code testing technique must be based upon an assumed fault model [7].We assume that all faults will behave according to some specific rules.Any fault model can only consider a subset of all possible faults.
For example, a common debugging practice is to insert a series of print statements in key positions.This debugging practice assumes a particular fault model.
Faults will cause the program to execute in the incorrect order and will be demonstrated Figure 2 illustrates the taxonomy of Fault Injection Techniques in the printed output.Clearly, not all faults will adhere to this model.
No one fault model will fit all faults.However, a fault model can be very effective in detecting faults that fit the model.
Fault Injection technique of fault injection dates back to the 1970s when it was first used to induce faults at a hardware level.This type of fault injection is called Hardware Implemented Fault Injection (HWIFI) and attempts to simulate hardware failures within a system.The first experiments in hardware fault injection involved nothing more than shorting connections on circuit boards and observing the effect on the system (bridging faults).It was used primarily as a test of the dependability of the hardware system.Later specialised hardware was developed to extend this technique, such as devices to bombard specific areas of a circuit board with heavy radiation.It was soon found that faults could be induced by software techniques and that aspects of this technique could be useful for assessing software systems.Collectively these techniques are known as Software Implemented Fault Injection (SWIFI) [8].
Martin defines software fault injections as faults which are injected at the software level by corrupting code or data.So faults are applicable at the implementation phase when the code of the system is available, and it can be applied on an application to simulate either internal or external faults.
Internal faults represent design and implementation faults, such as variables/parameters that are wrong or not initialized, incorrect assignments or condition checks.External faults represent all external factors that are not related to faults in the code itself but that alter the system's state.
The injection of failures can discover errors that normal procedures cannot.First, it tests the mechanisms of exception and treatment of failures that in normal circumstances are not sufficiently proven and, helps to evaluate the risk, verifying how much defective can be the system behavior in presence of errors.All of the injection failures methods are based on concrete hardware or software characteristics associated to systems which are applied, then, to realize generalizations is a very complicated task.

Prior Work on Fault Injection
Fault injection can be used to modify either a program's source code text or the machine state of an executing program.

Domino's Effect
Domino's effect is the cascading effect of defects from the initial stages of the project to all the subsequent stages of the software life cycle.Errors undetected in one work product are 'leaked' to the child work product and amplifies defects in the child work product.This chain reaction causes an exponential defect leakage.E.g.: undetected errors in requirements leak and cause a significant number of defects in design which, in turn, causes more defects in the source code.The result of this study is to arrive at an "Amplification Index" which will characterize the extent of impact or damage of phase-wise defects in subsequent Software Development Life Cycle (SDLC) phases.
The defect components in a work product and leakage into subsequent phases are illustrated in Figure 3 below:

Trimentus Approach for Fault Injection Experiments
Defects were deliberately injected into each phase (work product) in the software development life cycle of a typical application development project and the effect of the defects injected was studied subsequently.The injected defects are typical defects that are characteristic of the software systems of a commercial application on Library Management System (LMS) and were chosen from the organizational defect database.
An approach was adopted towards studying the impact of defect amplification in a software system was causal analysis of the defects occurring in subsequent phases caused due to injected defects.
Fault injection can occur in several ways:  Additional code can be linked to the target program and executed synchronously with the program flow.


A separate process can perform the injection asynchronously with the flow of the target process.


Separate hardware can directly access the memory to modify the state, thus not affecting the timing characteristics of the target process.Overlay faults occur when a program writes into an incorrect location due to a faulty destination operand.Chillarege and Bowen claim that overlay faults account for 34% of the errors in systems programs.The experiment involved the use of failure acceleration, decreasing fault and error latency and increasing the probability that a fault will cause an error.The experiment applied failure acceleration by corrupting a large region of memory in a single injection.To inject an overlay fault, all bits in an entire page of physical memory are set to one.Because the page is in physical memory, the probability that the latency will be short is further increased.About 16% of the faults immediately crashed the system; about 14% caused a partial loss of service, which was usually recovered from soon after.
Half of the faults did not cause failures.These potential hazards are failures waiting to occur.The injection  Function faults: Invalid functions.The assembly code for a function is dynamically replaced with the assembly code from a manually rewritten alternate version.Initialization faults can be caught statically with a good compiler.The assignment and condition check faults are clearly relevant to the testing of recovery code, since an incorrect assignment or condition can be a condition that should force the execution of recovery code.Function faults are also relevant, especially if they could be automatically generated.Unfortunately, manual rewriting of sections of code is prohibitive in a large system.

Why Study Fault Injection Modelling?
Fault Injection Modelling has gradually crept into prominence over the last decade as one of the new buzz words in software design.However, as Martin observes: "The main characteristic of fault injection software is that it is capable of injecting failures into any functional addressing unit by means of software, such as memory, registers, and peripherals.The goal of the fault injection software is to reproduce, in the logical scope, the errors that are reproduced after failures in the hardware.A good characterization of failure model should be allowed that this one was as versatile as possible, allowing a major number of combinations among the location, trigger conditions, kind of fault and duration, so that the coverage was maximum.Recent days, the Fault Injection technique has been considered as a very useful tool to monitor and evaluate the behavior of computing systems in the presence of faults.It's because the tool tries to produce or simulate faults during an execution of the system under test, and then the behavior of the system is detected."[9] Figure 4

Description of Software System
A Library Management System (LMS) help in automating functions of the library.It helps in reducing the time spent in record keeping and management effectively.The management information system application was used to conduct the fault injection experiments.The same application was developed in the following technologies in 3G languages as listed in Table 1 below.
LMS was simultaneously developed by independent project team and were made mutually exclusive.The application development for the projects followed the same process as described in the quality management system for software development of Trimentus.LMS was chosen to FIM because common MIS Domain knowledge for the application was high; it can be independently managed  In this paper, the system contains injected defects common across all projects.The same count of defects (5 numbers) were introduced in each phase of SDLC.

Results of the Experiments
The results from the independent experiments are derived at each stage of the Software Life Development Cycle Phase.The following section describes the detailed activities and step-by-step process followed in the introduction of defects in each software work product with results output.

Requirement Review
SRS (Software Requirement Specification) document was prepared and used as the basis for development of for all the experiments.SRS is identified as requirements documents.However, after the review of SRS, defects were injected into the same document.The SRS containing the defects were baselined by independent project team respectively to be used as basis for the Design.
The defects injected into the Requirement document are given in Table 2.The requirements defects are analyzed through causal analysis techniques to be classified and categorized.

Design Phase Analysis
Design document is prepared with (fault injected) SRS as basis.There were several defects observed with "source" as requirements.The Injected defects were major cause for design defects.3 lists the number of defects injected independently in Requirements and inherent defects detected after Requirements document review.Further, it lists the defect leakage to the child work product (Design) with inherent defects detected after Design review for each experiment.

Design Defect Amplification: Technology
Variant Figure 5 represents the comparison of Amplification Index between the LMS developed on different technologies.The amplification of design defects caused due to the injected requirement defects in LMS is evidenced in all technologies and more prominent in VB Microsoft technology.

Amplification Index (AI) for Requirements
Table 3 and Table 4 represent the methodology that was used to calculate Requirement Amplification Index (i.e.impact of requirements defects on Design).

Defects in Design
Table 5 lists the various types of known design defects that were introduced after design review.The defects are classified and categorized after causal analysis.

Statistical Analysis and Validation
Based on the AI derived from the above requirement data analysis, a statistical study was carried out to understand and analyze the statistical significance and relationship of AI across phases.
A hypothesis was formulated based on the conditions of analysis as follows; H0 : Requirements Amplification Index is same across technologies H1 : Requirements Amplification Index is different between technologies Minitab tool was used to analyze the data set of Requirement Amplification Index.A simple T-test was run to validate the statistical significance of the requirement AI data across technologies.
Minitab output on the Hypothesis Testing is listed in Table 6 below.
The statistical rule of elimination is: 1) If the P-Value > 0.05, Then H0 is true and there is no difference in the groups.= Accept H0 2) If the Value < 0.05, Then H0 is false and there is a statistically significant difference.= Reject H0 and Accept H1  This results in: 0.065 > 0.05; so by the rule, Accept the Ho.
To conclude that, "Requirements Amplification Index is same across technologies and there is no statistical significant difference on AI across technologies in the Library Management System (LMS) developed in different technologies".

Coding Phase Analysis
Coding was performed with (fault injected) design as basis.There were several defects observed with "source" as Design and Requirements.The Injected defects were the major cause for Code defects detected in Code review.

Code Review
Table 7 appends to Table 3 with the number of defects injected independently with leaked defected in design document.Further, it lists the defect leakage from Design to Code with inherent defects detected after Code review for each experiment.

Code Defect Amplification: Technology Variant
Figure 6 represents the comparison of Amplification Index between the LMS developed on different technologies.
The amplification of coding defects caused due to the injected design defects in LMS is evidenced in all technologies and more prominent in VB Microsoft technology.

Amplification Index for Design
Table 8 illustrates the methodology and computation details used to calculate Design Amplification Index (i.e.impact of Design defects on Code).

Statistical Analysis and Validation
Similarly, based on the AI derived from the above design data analysis, a statistical study was carried out to understand and analyze the statistical significance and relationship of AI across design phases.
A hypothesis was formulated based on the conditions of analysis as follows; H0 : Design Amplification Index is same across technologies H1 : Design Amplification Index is different between technologies Minitab tool was used to analyze the data set of design Amplification Index.A simple T-test was run to validate the statistical significance of the design AI data across technologies.
Minitab output on the Hypothesis Testing is listed in Table 9 below.The statistical rule of elimination is: 1) If the P-Value > 0.05, Then H0 is true and there is no difference in the groups.= Accept H0 2) If the Value < 0.05, Then Ho is false and there is a statistically significant difference.= Reject H0 and Accept H1 This results in: 0.038 < 0.05; so by the rule, Reject H0 and Accept H1.
To conclude that, "Design Amplification Index is different across technologies and there is a statistical significant difference on design AI across technologies in

Testing Phase Analysis
Testing was performed with (fault injected) code as basis.There were several defects observed with "source" as Coding, Design and Requirements.The injected defects were the major cause for Code defects detected in Testing.

Testing
Table 10 appends to Table 7 with the number of defects injected independently with leaked defected in Code.Further, it lists the defect leakage from Code to Test Cases with inherent defects detected after Test Case review for each experiment.

Test Defect Amplification: Technology Variant
Figure 7 represents the comparison of Amplification Index between the LMS developed on different technologies.The amplification of test defects caused due to the injected code defects in LMS is evidenced in all technologies and more prominent in VB Microsoft technology.

Amplification Index for Code
Table 11 illustrates the methodology and computation details used to calculate Test Amplification Index (i.e.impact of Code defects on Test results).

Statistical Analysis and Validation
Similarly, based on the AI derived from the above code data analysis, a statistical study was carried out to understand and analyze the statistical significance and relationship of AI across test phase.Minitab output on the Hypothesis Testing is listed in Table 12 below.
The statistical rule of elimination is: 1) If the P-Value > 0.05, Then Ho is true and there is no difference in the groups.= Accept Ho 2) If the Value < 0.05, Then Ho is false and there is a statistically significant difference.= Reject Ho and Accept H1 This results in: 0.009 < 0.05; so by the rule, Reject Ho and Accept H1.
To conclude that, "Code Amplification Index is different across technologies and there is a statistical significant difference on design AI across technologies in the LiBrary Management System (LMS) developed in different technologies".

AI Trend Analysis
The Amplification Index indicates the extent of damage caused by a defect in various phases of the project.The index increases with every step in the life cycle of the project.This is evident in the case of Microsoft technologies (VB and C#.net) but AI in the case of open source technologies such Java, the AI increases in requirements and design but in code, it is found have marginal decrease compared to other technologies.It is also seen that defects amplification in the VB Technology show substantial increase in the amplification index across phases compared to other selected technologies.
The relative growth of AI across phases in Java technology is less compared to Microsoft technology.This indicates a better fault tolerance for Java technology.
It was concluded and validated statistically that;  The defect leakage analysis emphasizes the importance of thorough and systematic reviews in the early stages of a software project with an emphasis on defect prevention.The analysis indicates a high increase of cost and effort to remove the defects at later stages.The number of defects increases exponentially as a direct result of defects leaked from previous stages.
Figure 9 consolidated the defect leakage pattern across technologies distributed each SDLC phase.

Future Experiments
Currently, the study is being extended to analyze the effect of the defects and amplification index in the development phases of the different domain based projects developed with same technology.
Guidelines for review time and effort estimation are being computed by analyzing and defining the review and test stop criteria.Error seeding during testing can be

Limitations of Experiments
The following are the limitations of the experiments: Causal analysis is relatively subjective to understand the cause of amplified defect.This required detailed review and discussion with project team and technical/technology experts.


Defect removal efficiency percentage used for experiments in different technologies are based on a test in a sample requirement, design and code with known defects provided to project members and review efficiency percentage derived from the defects detected.


It is verified that the skill set of the analysts and programmers working in the projects are same and/or similar across technologies.The experiments do not consider specialized automated tools and techniques used in the development of software work products which could have impact of the work product output quality.


Why study Fault Injection Modelling? Why study business software fault tolerance requirements?Why are they called 'Fault Injection & Error Seeding'? Why review Software Implemented Fault Injection (SWIFI)?

Figure 2
shows taxonomy of the key methods of fault injection.Fault injection techniques based on static analysis-program source modification-are modeled by the left subtree.The most common static fault injection is mutation testing.The right subtree in Figure 2 models dynamic fault injection techniques where changes are made to an actively running program's state.Much of the recent fault injection research is concerned with dynamic injection.

Figure 3 .
Figure 3. Fault injection pattern illustrates the relative cost factor in the defect resolution as the work product elaborates in the Software Development Life Cycle phases;  The Carnegie Mellon Software Engineering Institute 1 reports that at least 42-50 percent of software defects originate in the requirements phase. The Defense Acquisition University Program Manager Magazine 2 reports that a Department of Defense study that over 50 percent of all software errors originate in the requirements phase.1) MSDN (November, 2005) "Leveraging the Role of Testing and Quality across the Lifecycle to Cut Costs and Drive IT/Business Responsiveness".2) Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology Symposium, 5 June 2003.

Figure 4 .
Figure 4. Relative cost to fix defects vs. development phases


Requirement defects amplification index across on identified technologies remains are same. Design and Code defects amplification index across on technologies vary based on technologies for the common application developed in the same domain.

Figure 8
illustrates the consolidated Amplification Index trend across technologies classified under each phase of SDLC;

Table 2 . Definition of defect types -requirements Action taken Defect Injected Defect severity Defect Type
Curing the Software Requirements and Cost Estimating Blues Development through Fault Injection Patterns

Table 4 . Defects amplification index computation-require- ments to design
 One requirement defect leaked causes 0.8 defect in design in Java technology

Table 11 . Defects amplification index computation-code to test
: Code Amplification Index is same across technologies H1 : Code Amplification Index is different between technologiesMinitab tool was used to analyze the data set of Code Amplification Index.A simple T-test was run to validate the statistical significance of the code AI data across technologies. Ho