New Trend in Fintech: Research on Artificial Intelligence Model Interpretability in Financial Fields

With the development of Fintech, applying artificial intelligence (AI) technologies to the financial field is a general trend. However, there are some in-appropriate conditions, for instance, the AI model is always treated as a black box and cannot be interpreted. This paper studies the AI model interpretability when the models are applied in the financial field. We analyze the reasons of black box problem and explore the effective solutions. We propose a new kind of automatic Regtech tool—LIMER, and put forward policy suggestions, thereby continuously promoting the development of Fintech to a higher level.


Introduction
In recent years, the rapid development of innovative technologies, for instance, artificial intelligence (AI) has had a great influence on the global financial industry. However, the black box phenomenon of AI models has also attracted the attention of many international government agencies and financial regulatory authorities. The black box phenomenon of AI models is that AI models are extremely complex and cannot be interpreted, which are always treated as a black box. Some institutions have emphasized the importance of model interpretability when AI is applied in the financial field.
For example, in report Big Data Meets Artificial Intelligence-Challenges and Implications for the Regulation of Financial Services (July 2018) [1], BaFin pointed out that the precondition of applying AI to the financial field is that financial institutions, such as banks, have some methods providing how AI models work and why decisions are made (i.e., model interpretability), thus preventing models from being treated as a pure black box.
Moreover, the Financial Stability Board (FSB) issued the report Artificial Intelligence and Machine Learning in Financial Services: Market Developments and Financial Stability Implications in November 2017 and noted that AI and machine learning are likely to bring many challenges to the financial stability [2]. In particular, AI models and machine learning algorithms are extremely complex and generally lacking in interpretability. It is difficult for users to know how these applications will affect the market. They may bring unexpected shocks to the financial stability, even causing systematic risks. At the present stage, on the basis of well and truly evaluating the risks of AI and machine learning with respect to data privacy, network security, etc., AI model interpretability should be constantly improved, and the supervision of applications of AI and machine learning in the financial field should be strengthened.
Financial regulatory authorities in China made similar requests. In April 2018, the People's Bank of China (PBOC), China Banking and Insurance Regulatory Commission (CBIRC), China Securities Regulatory Commission (CSRC) and State Administration of Foreign Exchange (SAFE) jointly issued Guidance on Regulating the Asset Management Business of Financial Institutions [3]. It noted that financial institutions should report to the financial regulatory authorities the main parameters of AI models and the logic of asset allocation. In addition, financial institutions should not only establish specific intelligent investment management accounts for investors, but also fully prompt inherent defects and risks of AI algorithms. Moreover, they should know clearly the process of transactions and monitor the trading positions of intelligent investment management accounts.
What's more, Li et al. in China Banking and Insurance Regulatory Commission (CBIRC) also pointed out that when financial institutions use intelligent systems to provide intelligent investment management advices, similar risk indicators and trading strategies may lead to the phenomenon of buy and sell at the same time, so that rise and fall at the same time, thus exacerbating the market fluctuations [4]. According to the classification of Fintech by the Basel Committee on Banking Supervision, i.e., payment, deposit and loan, investment management and market facilities, the regulatory authorities should focus on information disclosure and investor protection in intelligent investment management, i.e., model interpretability.
To sum up, from the perspective of regulatory authorities, mastering the internal mechanism of the AI models applied by financial institutions can better protect the rights and interests of consumers, which is helpful to remove the discriminatory factors in the model design. In addition, the interpretation of the AI models applied by financial institutions enables regulators to control financial risks and maintain the financial market stability.  When AI models are applied in the financial field, if the models are not fully   understood while the models make the business decisions, the users may gradually become indifferent to the risks, thus accumulating financial risks. For example, when AI models are applied to the risk management in the pre-loan credit evaluation, if the probability of default of the customer is predicted without understanding the internal mechanism of the models, improper credit scores may be given. In the asset management business, through the traditional way, for instance, either technical analysis or fundamental analysis, investors will be able to know every detail of decision-making. However, when using AI models, i.e., the intelligent investment, the models may provide similar advices to large number of investors. If investors do not understand the reasons behind the models' recommendations, a buy and sell at the same time phenomenon may occur, which magnifies the single financial risk. Therefore, the model interpretability has become a major obstacle for the application of AI in the financial field.

Research by the International Institute of China Construction Bank in 2018
found that the application of AI in the financial field has a large imbalance [5]. It is far more applied in loan and asset management than in information provision, payment and other businesses. We believe that one of many factors restricting the balanced application of AI in the financial field is that when the AI model is applied, the internal mechanism of the model is not understood, i.e., the model cannot be interpreted.
The rest of this paper is organized as follows. We first give the reason why the AI models cannot be interpreted in Section 2. In Section3, we formally define the model interpretability. We review the related work of interpreting the AI models in Section 4 and present our solutions in Section 5. In Section 6, we give some policy suggestions and conclude the paper in Section 7.

The Reason Why the AI Models Cannot be Interpreted
We believe that the reason why AI models applied in the financial field cannot be interpreted lies in technology precedes the rules. On the one hand, with the rapid development of AI, the models become more and more complex, which leads to the fact that the models cannot be interpreted. On the other hand, the regulatory rules remain unchanged, that is, the regulatory authorities have not introduced relevant policies in time to adapt to the development of technologies.
Both factors result in the situation of technology precedes the rules. In this paper, we take the Risk-weighted Assets (RWA) calculation of commercial banks as an example to illustrate the lag of regulatory rules in adapting to the application of AI models.

AI Is Developing Rapidly
With the rapid development of technologies, the performance of AI models is continuously enhanced, the accuracy of various prediction tasks is constantly improved, and the complexity of the model is also increasingly high.

The Regulatory Rules Remain Unchanged
Since the 1990s, with the continuous progress of computer technology, com-  However, in terms of the models used to estimate the above three indicators, currently, the regulators reject the models generated by AI due to the high complexity of these models (they cannot be interpreted), which will bring greater obstacles to the regulators' supervision. Although currently, the regulators require that the model parameters and internal mechanism used to estimate these indicators should be easily understood, the regulators have not given any suggestions on how to make the models be interpretable, nor have they issued any relevant policies to be implemented to adapt to the application of new technologies.

What is the Model Interpretability?
At present, there is no unified definition of model interpretability in both academia and industry fields. Therefore, we give its definition based on the relevant research of model interpretability.

Why Does the Model Need to be Interpreted?
From a broad sense, the necessity for interpretability comes from the fact that human beings do not know enough about a certain problem or task. With respect to the field of AI, although complex AI model, such as deep neural network (DNN) has high expression ability, cooperating with some parameter tuning technologies that can be called as modern alchemy, can achieve high accuracy in many specific tasks. However, for humans, the trained model is just a nonlinear function formula with a pile of seemingly parameters and its results have very high accuracy. We believe the model itself also means knowledge. When

Find Technical Methods for Interpreting the AI Models
In order to solve the problem that AI models applied in the financial fields cannot be interpreted (i.e., the black box phenomenon), we try to find and select technical methods-model interpretable methods and discuss which method is more suitable for the financial field. In the report released in July 2018, BaFin pointed out that the problem could be solved, but it did not give specific methods or relevant policy suggestions [1]. Therefore, this paper carries out further research on the model interpretable methods and analyzes the specific methods applicable to the financial field that can be used to interpret the AI models.

Hidden Neuron Analysis Methods
The hidden neuron analysis methods interpret a pre-trained deep neural net- Based on the above studies, the hidden neuron analysis methods provide useful qualitative insights into the properties of each hidden neuron. However, qualitatively analyzing every neuron does not provide much actionable and quantitative interpretation about the overall mechanism of the entire neural network.
More importantly, the visualization method has a better interpretable effect on the image data as input, especially the convolutional neural network (CNN). In the financial field, AI models are mostly applied to risk management or asset management business. In relevant scenarios, the application of AI models on image data is not too much. Therefore, this model interpretable method will not show obvious effects in the financial field.

Model Mimicking Methods
By imitating the classification function of a neural network, the model mimicking methods build a transparent model that is easy to interpret and achieves a high classification accuracy.

Local Interpretation Methods
The local interpretation methods compute and visualize the important features for an input instance by analyzing the predictions of its local perturbations.

A Brief Summary
The hidden neuron analysis methods, the model mimicking methods and the local interpretation methods have their own advantages and disadvantages (as shown in Table 1). In terms of the applicability in the financial field, we believe that the local interpretation methods are most applicable, while the model mimicking methods are not as applicable as local interpretation methods, and the hidden neuron analysis methods are least applicable for the financial field. When AI models are applied in the financial field, the data is not image data in most cases. Therefore, the hidden neuron analysis methods are not suitable for the financial field. If we only want to have a general understanding of the internal mechanism of the model, we can adopt the model mimicking methods, but the interpretation effect is not ideal, because the shallow model cannot completely represent the complex model. Therefore, the model mimicking methods are generally applicable in the financial field. If we want to get the reason behind the corresponding prediction of the model for a specific input instance, we can adopt the local interpretation methods, which can interpret the actual internal mechanism of the complex model with high accuracy and thus is most suitable for the financial field. Therefore, regulators can interpret the AI models applied by commercial banks in their relevant businesses by using the model interpretable scheme based on the local interpretation methods.

Model Interpretable Methods that Meet the Existing Regulatory Rules to the Maximum Extent-Taking the Risk-Weighted Asset (RWA) Calculation Process as an Example
Based on the selection of technical means to solve the problem that AI models in financial field cannot be interpreted, we believe that there are feasible technical  The complex models are imitated by shallow model, and the overall behavior of the model is easy to be interpreted.
1) There is a gap between the interpretation of the shallow model and the actual overall mechanism of the complex model. 2) The reasons behind the models make predictions are not given.

Local Interpretation Methods
The interpretation of a single prediction can interpret the actual internal mechanism of a complex model and give the reason for the prediction of the model with high accuracy.
Compared with above two types of methods, local interpretation methods have no obvious disadvantages.
High that regulators do not allow commercial banks to adopt when building AI models. Regulators then select specific individual assets from the set of assets that commercial banks use in calculating Risk-weighted Assets (RWA) (sovereign, financial institution, corporate and retail assets). Using the local interpretation methods, regulators can interpret the AI models used by commercial banks to estimate the above three indicators of the asset. If there is no factor cautiously used in the interpretation, the model can be regarded as compliance; otherwise, the commercial bank needs to rectify the model. The time complexity of LIMER is O(n), where n is the number of assets that need to be inspected. Also, LIMER has a good stability and scalability.

The Proposal for Basel III-Explicitly Allowing Commercial Banks to Use AI Models in IRB
Basel III improves the details of risk exposure calculation using the internal Open Journal of Applied Sciences ratings-based approach (IRB), but not clearly point out that the models used to estimate probability of default (PD), default loss rate (LGD) and default risk exposure (EAD) can be AI models. As above stated, when using AI models to calculate the Risk-weighted Assets (RWA), we already have the feasible interpretable methods. Therefore, we suggest that the Basel III explicitly allows commercial banks to use AI models in IRB, and at the same time, starts the study of principle of building the set of factors that cautiously used when the models are built.

The Proposal for Regulators in China-Use Automated Regtech Tool to Interpret the AI Models Used by Commercial Banks
Based on the above model interpretable methods during the process of Risk-

The Proposal for Large Commercial Banks-Adopting Model Interpretable Methods to Continuously Improve the Control of AI Risks
For large commercial banks, we suggest that they should use AI technologies to build new models and use more data sources to evaluate important indicators such as probability of default (PD) and default loss rate (LGD) of assets. At the same time, when large commercial banks design and develop models to be ap-Open Journal of Applied Sciences plied in the financial field, the adoption of model interpretable methods is expected to comprehensively improve the control of AI risks. For example, in business areas such as risk control and asset management, model interpretation also has many advantages. First, it can make the model more effective (the process of model interpretation is also the process of knowledge discovery, and commercial banks can use new knowledge to optimize and improve the effects of the model). Second, it can make it easier for commercial banks to meet regulatory requirements. Third, it can protect the practitioners of commercial banks.
Fourthly, it can check the model errors caused by mixing data in the data set that will not appear in the actual situation and inconsistency between training data and test data. Therefore, we also suggest that large commercial banks, on the premise of meeting regulatory requirements, actively use model interpretation methods to continuously improve their ability to control risks of AI. Therefore, they can apply AI to business development, and promote the digital transformation of the banking industry with high quality.

Conclusion
This paper studies the AI model interpretability when the models are applied in the financial field. We analyze the reasons of black box problem and explore the effective solutions. We propose a new kind of automatic Regtech tool-LIMER, and put forward policy suggestions. This work may open up many promising directions for future work. Firstly, it is worth supporting more scenarios when AI models are applied in the financial field. Secondly, it is worth studying more model interpretable methods.