Reliability Assessment Tool Based on Deep Learning and Data Preprocessing for OSS ()
1. Introduction
Open source software (OSS) systems have been developed by various OSS project. Along with this, the number of users become increasing regardless of individuals or companies. On OSS, the number of reported faults is also increasing. In particular, several serious faults are detected in the easier phase of operation. In addition, since the source code of OSS is opened to the public, it is easier for crackers to find vulnerabilities than commercial software. In particular, the zero-day attacks are targeted to vulnerabilities. The vulnerability will be fixed when the serious problems are recognized by users. This is one of the major risks by operating OSS on the internet.
Under these circumstances, it is necessary to assess the reliability of the OSS in order to operate OSS safely and stably. Many methods for software reliability assessment have been proposed by software reliability growth models [1] [2] [3]. However, it is difficult to assess the reliability of OSS. OSS user is increasing year by year. However, these users are not familiar with OSS. Therefore, the users may be exposed by the potential security and operational risks, if the OSS is in an unstable state. Various researchers have discussed the reliability of OSS to solve this problem [4] [5]. Software reliability model is one of the methods to measure OSS reliability. This method has the advantage for the reliability evaluation dynamically. Historically, the methods of software reliability assessment based on the software reliability growth models have been proposed by many researchers [1] [2] [3]. Also there are several methods of reliability assessment based on deep learning for OSS fault data in the past [6] [7].
On the other hand, a bug tracking system (BTS) is often used in OSS development to manage project progress by correcting faults. The BTS records many information in terms of the faults reported after releasing the OSS. There are the data such as the date, time, the nickname of reporter, assignee, OS, severity etc. on the BTS. This recorded data set is different according to OSS project. Also, there is the data collected in the specified OSS project only. Several researches have used BTS fault data to estimate the OSS reliability [6] [7]. However, there are no research papers in terms of the collected specific data to the OSS project.
We focus on specific data recorded on the BTS of OSS. We discuss the difference between the general data collected by BTS and the specified data. Furthermore, we develop GUI-based software based on deep learning by using the proposed method.
The organization of this paper is as follows:
Section 2: describes the several methods that have been proposed in previous papers.
Section 3: proposes the estimation method of OSS software reliability assessment based on deep learning. Then, the proposed method applies data recorded only in a specific OSS to deep learning.
Section 4: shows the optimizing method of neural network.
Section 5: describes the development of tools to perform deep learning.
Section 6: shows the several numerical examples based on proposed method and previous method.
Section 7: discusses the proposed method.
2. Data Preprocessing for OSS Fault Data and MTBF
There are various reliability models and neural networks for improving the software reliability. Many of the methods for estimating reliability with neural networks based on time-series analysis are using the past data of failure occurrence [8] [9]. In particular, many reliability estimation methods based on machine learning use only the software failure data in order to analyze the reliability. On the other hand, a reliability evaluation method using the BTS has been proposed in the past in terms of OSS software reliability. Also, the bug management by using BTS is widely used in OSS projects around the world. The general fault factors recorded in the BTS have been used in several researches [6] [7]. On the other hand, the BTS has collected the other specific data of OSS projects. This paper has the unique feature that we discuss the difference between the general data and the specific data on BTS. We show the general data recorded in many BTS in Table 1. Table 2 shows the specific data of OSS. Table 1 and Table 2 are obtained from RedHad Openstack [10].
In order to use these data in deep learning, it is necessary to convert above mentioned data to numerical values. In this paper, the general data have been converted into numerical values as shown in Table 3. Similarly, the specific data have been converted into numerical values as shown in Table 4.
Mean Time Between software Failures (MTBF) is one of reliability assessment measures. The software is the reliable system if the MTBF becomes large. Then the software will operate continuously without failure. The MTBF grows as development progresses because the software faults are removed during the software operation. i-th
is given by the following equation
Table 1. The list of general fault data in BTS.
Table 2. The list of specific fault data in BTS.
Table 3. Method of numeric conversion for each factor in general data.
Table 4. Method of numeric conversion for each factor in specific data.
, (1)
where each parameters are defined as follows:
: The total number until i-th fault occurred during operation,
: The time of software operation until i-th fault.
We consider that MTBF is used for the systems that can be repaired. On the other hand, the Mean Time To Failure (MTTF) is used for the non-repairable systems. In this paper, MTBF is used as the assessment measure for OSS reliability because the OSS is repairable system. In this paper, the MTBF is defined as the time between the correction of a fault and the occurrence of the next fault.
3. Estimation of OSS Reliability Based on MTBF and Deep Learning
Several researchers have proposed the useful deep learning algorithms. In this paper, we use MTBF-based deep learning to estimate OSS reliability. In this paper, we have constructed a feed-forward neural network in which signals propagate in the order of input layer, intermediate layer, and output layer, respectively. Then, we distinguish between the general data recorded on BTS and the specific data on OSS project. We show three cases based on deep learning in Table 1 and Table 2 as explanatory variables. Moreover, we consider the case of the case with both general and special data as explanatory variables.
4. Optimization of Deep Neural Network
Several algorithms have been proposed by several researchers for parameter optimization of deep learning networks [11] [12] [13] [14] [15]. In this paper, we use the Adam optimizer well known as deep learning optimization algorithm [16]. Figure 1 shows the details of the hyper-parameters used for Adam. Adam is a stochastic gradient descent method that improves on AdaGrad and RMSProp. It has the characteristic of updating values more frequently where the gradient of the loss function is large and reducing the update range. Then, the gradient is highly variable, the facilitating convergence with a small amount of computation.
4.1. Activation Function
Activation function is the function used to convert the sum of inputs from the network into outputs. This function is typically a nonlinear function in order to achieve in Figure 1 outputs that cannot be represented by a single node. In this paper, ReLU is used as the activation function. ReLU is following as
(2)
In deep learning, the gradient method defines a loss function as Equation (2). Also, there is the other loss function with weights to modify the derivative toward near 0. This is often proposed when learning proceeds. However, depending on the activation function, as the number of layers increases, nodes with gradients close to 0 are more likely to appear, and the Vanishing gradient problem, where the weights are not updated, is more likely to occur. ReLU is a function that is less prone to this problem. In mathematics, a value of 0 is not differentiable. In general, when the value is 0, including when it is less than 0, it follows that
(3)
From Equation (3), the gradient is sufficiently large to prevent the gradient loss problem. It also has the advantage of being less computationally expensive due to its simple formulas.
4.2. Loss Function
Loss function is a function that evaluates the error between the output value of the network and the correct value when optimizing the weights of each node in deep learning. In this paper, the estimation is a regression problem to predict the output value of MTBF. The widely known loss functions for regression problems are MSE (Mean Squared Error), which squares the error between the correct solution and the output value, and MAE (Mean Absolute Error), which takes the absolute value of the correct solution and the output value, respectively. MSE uses the error squared, making MSE a more error-tolerant network than MAE. Since there are many outliers in the MTBF estimated in this paper that are far from the overall trend, MAE is used as the loss function. MAE is following as
(4)
where each parameter is as follows:
N: the total number of data,
: estimate,
: output data from network.
5. Development of Prototype Deep Learning Tool Based on GUI
Various research papers have used the deep learning to estimate the reliability of OSS. In order to perform such deep learning, there are time constraints for users who are not familiar with programming language, such as the need to build a scientific environment for program installation, and computers for deep learning, as well as time constraints for building the environment. This makes it difficult for the average user to understand the reliability of OSS by using deep learning. Therefore, we have developed a GUI-based reliability analysis tool to solve this problem. Our software works as follows
1) The data of Input OSS fault data are send to Node.js server from client software.
2) The Python read the transfer data, and start the algorithm of deep learning.
3) The Python send the estimation results to Node.js server.
4) The Python transfer the estimation results to client software. Then, the user can see the results by several graphs.
The software workflow is shown in Figure 2. The environment for deep learning is not built on the user’s side, but on the server, making the software independent of the user’s execution environment. The program operation screen is shown in Figure 3. The intuitive GUI-based software can be used by users who are not familiar with programming. The method proposed in Section 3 is used for deep learning. In addition, as shown in Figure 4, the user can choose whether to use general data, specific data, or both as parameters for deep learning. This enables optimal deep learning at the user’s discretion. After the training is completed, the server returns three graphs, MTBF, cumulative MTBF, and the Error Scatter, respectively. Then, the user can view the estimation results using the application. This paper presents as the examples for numerical results of the method proposed in Section 3 using the prototype tool.
Figure 2. The workflow of reliability assessment tool.
Figure 3. The screen of software readme.
6. Numerical Illustrations of Our Tool
We have retrieved 20,000 large fault data sets from RedHat Openstack’s BTS and input them into our deep learning tool. Then, the results were obtained from three deep training runs: one with general data, one with specific data, and one with both general and specific data, respectively. Figures 5-7 show the estimated instantaneous MTBF. Figure 5 and Figure 7 show that the estimation results generally capture the trend of the testing data. On the other hand, the graph in Figure 6 shows that the estimation result of MTBF is not so large even though the MTBF of the testing data is higher.
Figures 8-10 show the estimated cumulative MTBF. The cumulative MTBF shows the MTBF when all the instantaneous MTBFs up to a certain point are summed. The graphs in Figure 8 and Figure 10 show that the testing data is well estimated for the actual data. In particular, the shape of the graphs of the testing data and the estimation results in Figure 10 are very similar. On the other hand,
Figure 5. The estimated MTBF for faults using general parameter.
Figure 6. The estimated MTBF for faults using specific parameter.
the graph in Figure 9 shows a lower level of cumulative MTBF than the testing data.
Figures 11-13 show the scatter of errors. Also, Figure 11 and Figure 12 show that the testing data and estimation results tend to plot near the same when the MTBF of the testing data is low. We found that the estimate is not well done when the MTBF of the testing is high. Figure 13 plots the testing data and the estimation results in the same vicinity regardless of the testing data.
Figure 7. The estimated MTBF for faults using general and specific parameter.
Figure 8. The estimated cumulative MTBF for faults using general parameter.
Table 5 shows the correlation coefficient for the scatter of errors. The correlation coefficients are the highest for the general and specific data. This indicates that the accuracy of estimation is the highest for the general and specific data.
From above results, we find that combining general and specific data for learning may provide more accurate estimation than using general data.
Table 5. The correlation coefficient of errors.
Figure 9. The estimated cumulative MTBF for faults using specific parameter.
Figure 10. The estimated cumulative MTBF for faults using general and specific parameter.
Figure 11. The relation of testing data and estimate for faults using general parameter.
Figure 12. The relation of testing data and estimate for faults using specific parameter.
Figure 13. The relation of testing data and estimate for faults using general and specific parameter.
7. Concluding Remarks
In this paper, we have discussed the availability of our method by using specific data for estimate MTBF. As a result, the proposed estimation results using the specific data are not well than the general data. The estimation using a combination of general and specific data was more accurate than the estimation using general data. This has shown that specific data is useful as an explanatory variable with different characteristics from general data.
In addition, we have developed a reliability assessment tool based on deep learning that can be used by users who are not familiar with deep learning. The tool has been designed to be more practical by allowing users to choose whether to use the parameters recorded only in the specific OSS projects proposed in this paper.
In BTS, there are various types of failure data other than those used in this paper. Therefore, as a future study, we are planning to investigate the construction of reliability models by using fault data collected in BTSs of other OSS projects. On the other hand, some specific data collected from OSS projects may adversely affect the estimation. We would like to discuss the selection of appropriate specific data in the future.
Acknowledgements
This work was supported in part by the JSPS KAKENHI Grant No. 20K11799 in Japan.