Research on Test Methods for AI Industrial Application Capabilities of Smart Mobile Terminals

Tong Zhang¹, Yang Bai¹, Mingyan Song¹, Hao Tang², Yingchao Ren³, Jian Tan²

¹Innovation Center for Intelligent System on Recognition and Decision, Beijing Jinghang Research Institute of Computing and Communication, Beijing, China.
²School of Digital Media and Art Design, Beijing University of Posts and Telecommunications, Beijing, China.
³College of Resources and Environment, University of Chinese Academy of Sciences, Beijing, China.

DOI: 10.4236/jcc.2021.912008                

Abstract

A test method and system for AI industrial application capabilities of smart terminals. The method includes the following steps: According to the temperature change values of different AI industrial applications executed by standard smart terminal equipment under different equipment parameters, the equipment parameters and temperature changes of standard smart terminal equipment are constructed. Correlation model; obtain the device parameters of the smart terminal device to be tested, and obtain the predicted value of the temperature change of the smart terminal device to be tested based on the correlation model between the device parameters of the standard smart terminal device and the temperature change; Measure the actual value of the temperature change of the intelligent terminal device executing different AI industrial applications; obtain the test result of the intelligent terminal device to be tested according to the predicted value of the temperature change and the actual value of the temperature change.

Keywords

AI, Software Testing, Industrial Application Capabilities, Automation

1. Introduction

The automated production process required by Industry 4.0 will require a large number of AI testing equipment, including the main types of image, audio, and text automatic recognition [1]. The current AI operating capability testing is mainly testing tools and methods at the AI software level, and the result of the test is only to evaluate the equipment’s ability to run a specific AI model and its associated software and hardware environment [2]. TestCraft is an AI-based test automation platform that can perform regression and continuous testing on Selenium. It is also used to monitor web applications. The role of artificial intelligence (AI) technology is to reduce maintenance time and costs by automatically overcoming changes in the application. Applitools is an application visual management and AI-based visual UI testing and monitoring software. It provides an end-to-end software testing platform supported by Visual AI, which can be used by professionals in engineering, test automation, manual QA, DevOps, and Digital Transformation teams. Functionalize is cloud-based automated testing technology for function, performance, and load testing-a one-stop service for all the above tests. In addition, the tool uses machine learning and artificial intelligence to accelerate the creation, diagnosis, and maintenance of tests. Sauce Labs is another popular cloud-based test automation tool that utilizes machine learning and AI. Sauce Labs supports a comprehensive list of browsers and operating systems, mobile emulators and emulators, and mobile devices, and supports the needs of users to test the speed of their applications. It claims to be the world’s largest continuous testing cloud, providing more than 800 browser and operating system combinations, 200 mobile emulators and simulators, and thousands of real devices [3]. Testim automated functional testing tools use artificial intelligence and machine learning to speed up the writing, execution, and maintenance of automated tests.

The test software and methods listed in the background art essentially test the AI computing capabilities of devices (not limited to mobile devices such as smartphones) through different AI task processes. Different tools and software have different types of AI tasks and different communication frequencies from the remote end, but they are not able to test the stability of the continuous AI work of smartphone devices in an industrial environment [4]. It can be seen that the current main tools and methods cannot evaluate the AI application capability of equipment in an industrial environment, especially the long-term operating capability under long-term operating conditions.

2. Research Methods

The device parameters described in this article include CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and convergence time for executing different AI industrial applications [5].

The beneficial effects of the above technical solutions are: using CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and the convergence time of executing different AI industrial applications as independent variables, and temperature changes as dependent variables through changes in different parameters. The relationship between equipment parameters and temperature changes can be accurately measured, and the correlation model obtained is more accurate, providing a basis for subsequent accurate prediction of temperature changes of the equipment under test [6].

Further, the correlation model includes a linear regression model or a logistic regression model.

Further, according to the temperature change values of the standard smart terminal device executing different AI industrial applications under different device parameters, obtaining the correlation model between the device parameters of the standard smart terminal device and the temperature change includes:

Normalize the device parameters;

Based on the equipment parameters after normalization, a multiple linear regression method is adopted to establish a linear regression model between equipment parameters and temperature changes of standard intelligent terminal equipment;

Using the least-squares method to determine the model parameters of the linear regression model;

Perform residual analysis on the linear regression model, and if there are abnormal data, remove the abnormal data to re-determine the model parameters of the linear regression model [7].

The beneficial effects of the above technical solution are: the linear regression model can be constructed accurately and quickly through the least square method, and the residual analysis of the model can find out whether there are abnormal data in the model and affect the accuracy of the model. The constructed model is more accurate.

Further, determining the model parameters of the linear regression model by using the least square method includes:

The regression model is $Y=BX+U$ , where $X=\left[\begin{array}{cccc}1& x_{11}& \mathrm{...}& x_{1n}\\ 1& x_{21}& \mathrm{...}& x_{2n}\\ \mathrm{...}& \mathrm{...}& \mathrm{...}& \mathrm{...}\\ 1& x_{m1}& \mathrm{...}& x_{mn}\end{array}\right]$ , $B=\left[\begin{array}{c}{b}_0\\ b_1\\ \mathrm{...}\\ b_n\end{array}\right]$ , $U=\left[\begin{array}{c}{\mu }_0\\ {\mu }_1\\ \mathrm{...}\\ {\mu }_n\end{array}\right]$ and Y represent the operating temperature change value, X is the device parameter, n is the number of device parameters, m is the number of statistical samples, B is the regression coefficient, and U is the regression constant;

Using the least-squares method, with the minimum sum of squared deviations between the actual value and the estimated value of the model as the goal, calculate $X=\left[\begin{array}{cccc}1& x_{11}& \mathrm{...}& x_{1n}\\ 1& x_{21}& \mathrm{...}& x_{2n}\\ \mathrm{...}& \mathrm{...}& \mathrm{...}& \mathrm{...}\\ 1& x_{m1}& \mathrm{...}& x_{mn}\end{array}\right]$ , $B=\left[\begin{array}{c}{b}_0\\ b_1\\ \mathrm{...}\\ b_n\end{array}\right]$ , $U=\left[\begin{array}{c}{\mu }_0\\ {\mu }_1\\ \mathrm{...}\\ {\mu }_n\end{array}\right]$ , where $X_i$ and $Y_i$ represent the observed value of the ith sample, and $Y_i^{\ast }$ represents the regression value of the ith sample.

Further, the normalization processing on the device parameter data includes the normalization processing on the device parameter data using the following formula: $X^{*}=\frac{X-X_{mean}}{X_{std}}$ , where $X^{*}$ represents the normalized device parameter, and X Is the original equipment parameter, $X_{mean}$ is the mean value of the equipment parameter, and $X_{std}$ is the standard deviation of the equipment parameter.

The beneficial effects of the above technical solutions are: normalizing the data to eliminate data differences between different dimensions and data and provide a data basis for the subsequent construction of the model.

On the other hand, the embodiment of this document provides an AI industrial application capability test system for smart terminals, which includes the following modules:

The model building module is used to construct the correlation model between the device parameters of the standard smart terminal device and the temperature change according to the temperature change value of the standard smart terminal device under different equipment parameters to execute different AI industrial applications;

The temperature prediction module is used to obtain the device parameters of the smart terminal device to be tested, and obtain the predicted value of the operating temperature change of the smart terminal device to be tested based on the correlation model between the device parameters of the standard smart terminal device and the temperature change;

The test result acquisition module is used to acquire the actual temperature change value of the AI industrial application executed by the intelligent terminal device under test; obtain the test of the intelligent terminal device under test according to the predicted temperature change value and the actual temperature change value result.

Further, the device parameters include CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and convergence time for executing different AI industrial applications.

Further, the correlation model includes a linear regression model or a logistic regression model.

Further, the model building module includes:

The normalization module is used to normalize the device parameters;

The linear regression model building module is used to establish a linear regression model between the equipment parameters and temperature changes of standard intelligent terminal equipment based on the normalized equipment parameters and using multiple linear regression methods;

The model parameter acquisition module is used to determine the model parameters of the linear regression model by using the least square method;

The model correction module is used to perform residual analysis on the linear regression model, and if there is abnormal data, remove the abnormal data to re-determine the model parameters of the linear regression model.

3. Implementation

According to any type of semiconductor, it will become an intrinsic semiconductor at high temperatures―intrinsicization, resulting in the disappearance of pn junctions (PN junctions. PN junctions have unidirectional conductivity); and now almost all electronic devices have pn junctions (There are also active and drain pn junctions in MOS devices), so electronic products will fail at high temperatures. The smaller the temperature rise of the AI application running on the smart terminal, the longer it can run stably for a long time. In different industrial environments, equipment reliability and temperature changes conform to the bathtub curve.

The energy consumption in the calculation is related to the reversibility of the calculation. The Landauer principle states that energy must be consumed to erase information. (First form) If the computer erases 1 bit of information, the total amount of energy dissipated to the environment is at least KBTln2, where KB is Boltzmann’s constant and T is the temperature of the computer. In AI calculations, various logic gate operations are required, and bits are constantly being erased. In particular, AI tasks have a great demand for CPU and GPU calculations. The calculation speed is extremely fast and the heat generation is inevitable. The influence of AI tasks on the calculation of heat generation is a non-linear or simple model. First of all, different types of AI tasks have different effects on the heating of similar computing terminals. The calculation density of AI models for text, pictures, voice, etc. is different. Secondly, the same AI task has different heat distribution in different computing terminals. GPU mainly deals with matrix grid information, CPU mainly deals with comprehensive scheduling tasks, and memory and fixed storage are used to store temporary data and long-term data in the calculation phase [8].

Therefore, based on the consideration of CPU, GPU, storage and different AI tasks, and temperature change as a measurement index, it can quickly determine whether the smart terminal meets the requirements of stable operation capability in an industrial environment. On this basis, a specific embodiment of this article discloses a method for testing the AI industrial application capability of a smart terminal, which includes the following steps:

S1, according to the temperature change values of the standard intelligent terminal equipment performing different AI industrial applications under different equipment parameters, construct the correlation model between the equipment parameters of the standard intelligent terminal equipment and the temperature change.

Based on GB/T 5170 “Methods for the verification of basic parameters of environmental testing equipment for electrical and electronic products, low-temperature test equipment”, GB 5170.3-85 “Methods for verification of basic parameters for environmental testing equipment for electrical and electronic products, high-temperature test equipment” and GB 5170.4-85 “Environmental testing for electrical and electronic products Test equipment basic parameter verification method temperature change test equipment”, convert the stable operation time of the smart terminal in a specific industrial application environment into the temperature rise value of the AI industrial application, and select the smart terminal that performs the temperature rise value review requirements of the AI industrial application It is a standard intelligent terminal. For example, the working environment of a production line has an annual average temperature of 20 degrees, air humidity of 10%, and a dust density of 3 mg/m³. The intelligent terminal used in the production line executes AI applications for the comprehensive use of image recognition and text recognition for printing defective products detection and voice recognition to determine abnormalities and working conditions of the production line. The overall switching time of the smart terminal is consistent with the assembly line, and the annual working time exceeds 3000 hours. According to the specific industrial application environment, and the average temperature rise time of the smart terminal device performing different AI tasks should not exceed the standard smart terminal that selects the smart terminal corresponding to the standard.

Specifically, the device parameters include CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and convergence time for executing different AI industrial applications.

Exemplarily, the CPU operating parameter may be π to calculate the test time, and the number of calculation digits is 1 million digits. The GPU operating parameter is the repeated drawing test time, for example, repeated drawing of the full screen of a smartphone for 60 frames. The memory throughput rate is the memory random access test time, such as random access to 10 M data. The fixed storage throughput rate is the fixed storage random access test time, such as random access to 10 M data. Different AI industrial applications include text-based AI recognition tasks, voice-based AI recognition tasks, and picture-based AI recognition tasks. During implementation, in the same test environment as the specific industrial environment, the above-mentioned device parameters of the standard smart terminal are changed, and the temperature change measurement is performed on the standard smart terminal to perform different AI industrial applications to obtain sample data [9].

The correlation model may be a linear regression model or a logistic regression model.

Exemplarily, constructing a linear regression model between the device parameters of the standard smart terminal device and the temperature change includes the following steps:

S1.1. Perform normalization processing on the device parameters;

Specifically, the following formula is used to normalize the device parameter data: $X^{*}=\frac{X-X_{mean}}{X_{std}}$ , where $X^{*}$ represents the normalized device parameter, X is the original device parameter, $X_{mean}$ is the average value of the device parameter, and $X_{std}$ is the device parameter The standard deviation.

Through normalization, the difference of different dimensions and orders of magnitude is eliminated, and it is convenient to model and analyze the data.

S1.2. Based on the equipment parameters after normalization, a multiple linear regression method is adopted to establish a linear regression model between the equipment parameters and temperature changes of standard intelligent terminal equipment.

Specifically, the linear regression model is $Y=BX+U$ , where $X=\left[\begin{array}{cccc}1& x_{11}& \mathrm{...}& x_{1n}\\ 1& x_{21}& \mathrm{...}& x_{2n}\\ \mathrm{...}& \mathrm{...}& \mathrm{...}& \mathrm{...}\\ 1& x_{m1}& \mathrm{...}& x_{mn}\end{array}\right]$ , $B=\left[\begin{array}{c}{b}_0\\ b_1\\ \mathrm{...}\\ b_n\end{array}\right]$ , $U=\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\\ \mathrm{...}\\ {\mu }_n\end{array}\right]$ , $Y=\left[\begin{array}{c}{y}_1\\ y_2\\ \mathrm{...}\\ y_n\end{array}\right]$ and Y represent operating temperature change values, X is a device parameter, n is the number of device parameters, m is the number of statistical samples, B is the regression coefficient, and U is the regression constant;

S1.3. The least-square method is used to determine the model parameters of the linear regression model.

Specifically, the least-squares method, estimation of the model of the actual value and the sum of squares between the values $Q={\displaystyle \underset{i=1}{\overset{m}{\sum }}{(Y_i-Y_i^{\ast })}^2}={\displaystyle \underset{i=1}{\overset{m}{\sum }}{(Y_i-U-B{X}_i)}^2}$ minimum target, calculated $\{\begin{array}{l}U=\frac{{\displaystyle \sum X_i^2}{\displaystyle \sum Y_i}-{\displaystyle \sum X_i}{\displaystyle \sum X_i}{Y}_i}{m{\displaystyle \sum X_i^2}-{({\displaystyle \sum X_i})}^2}\\ B=\frac{m{{\displaystyle \sum X_{i}Y}}_i-{\displaystyle \sum X_i}{\displaystyle \sum Y_i}}{m{\displaystyle \sum X_i^2}-{({\displaystyle \sum X_i})}^2}\end{array}$ , wherein, $X_i$ and $Y_i$ represents the observed value of ith sample, $Y_i^{\ast }$ denotes ith sample The regression value.

S1.4. Perform residual analysis on the linear regression model, and if there are abnormal data, remove the abnormal data to re-determine the model parameters of the linear regression model.

After the linear regression model is established, the model is verified by residual analysis of the model. If there is abnormal point data, the abnormal data is eliminated, and the model parameters of the linear regression model are re-determined to make the model meet the requirements of equal variance. Exemplarily, a residual graph can be used to perform residual analysis on the model.

Exemplarily, the linear regression model can also be tested for fit and significance, and abnormal data in the sample data can be removed so that the regression model has high goodness of fit and significance. The specific goodness of fit test and significance test are commonly used statistical methods and will not be described in detail here.

S2. Obtain the device parameters of the smart terminal device to be tested, and obtain the predicted value of the temperature change of the smart terminal device to be tested based on the correlation model between the device parameters of the standard smart terminal device and the temperature change.

Specifically, after obtaining the CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and convergence time of executing different AI industrial applications of the intelligent terminal to be tested, the parameters are brought into the correlation model obtained in step S1. Obtain the predicted value of the operating temperature change of the device under test executing the AI industrial application.

S3. Obtain the actual temperature change values of the intelligent terminal device under test executing different AI industrial applications; obtain the test result of the intelligent terminal device under test according to the predicted temperature change value and the actual temperature change value.

Specifically, the temperature of the smart terminal to be tested is tested before and after the smart terminal device to be tested executes different AI industrial applications, and the actual value of the temperature change of the smart terminal device to be tested executing different AI industrial applications is obtained. Among them, multiple measurements can be performed on the intelligent terminal to be tested, and the average value of the multiple measurements is taken as the actual value of the temperature change.

According to the difference between the predicted value of temperature change and the actual value of temperature change, the test result of the intelligent terminal device to be tested is obtained. For example, according to the difference between the predicted value of temperature change and the actual value of temperature change within the preset threshold range, it means that the AI industrial application capability of the intelligent terminal under test meets the requirements, otherwise it is considered that the AI industrial application capability of the intelligent terminal under test does not meet the requirements. Require. In different industrial application environments, the requirements for the temperature change value of the intelligent terminal to execute AI industrial applications are different, and the threshold range will be different. Therefore, it needs to be set according to the actual industrial environment requirements.

In addition, even if the CPU operating parameters, GPU operating parameters, memory throughput rate, fixed storage throughput rate, and convergence time for different AI industrial applications are the same, the differences between the circuit layout and accessory collocation of different terminals will be reflected in the temperature. The difference. Therefore, using the AI industrial application capability test method for smart terminals provided in this article can not only test the AI industrial application capability of smart terminals but also further reflect the internal wiring of the terminal under test and the standard terminal based on the predicted value of temperature change and the actual value of temperature change. The difference in method, heat dissipation layout, and component level can be used to measure the quality level of the wiring method, heat dissipation layout, and component level of the smart terminal to be tested. The AI industrial application capability test method using the smart terminal provided in this article can be directly implemented in multiple related fields such as equipment testing before the deployment of AI models, AI terminal equipment procurement evaluation, and AI terminal equipment design prototype evaluation.

To conveniently view the test results, the evaluation results of the device to be tested can also be visually displayed in the form of scoring. For example, if the score of a standard smart terminal device is 100 points, the smart terminal to be tested can be scored according to the formula $\text{Score}=\theta \times {\text{e}}^{(-\frac{\Delta T^{\text{3}}{}_{score}}{2})}$ , which is convenient for the inspectors To obtain the evaluation results. Among them, $\theta$ represents the percentile coefficient, and $\Delta T_{score}$ represents the difference between the predicted value of the temperature change and the actual value of the temperature change of the smart terminal to be tested.

4. Summarize

On the one hand, the embodiment of this document provides a method for testing the AI industrial application capability of a smart terminal, which includes the following steps:

According to the temperature change values of the standard intelligent terminal equipment executing different AI industrial applications under different equipment parameters, construct the correlation model between the equipment parameters of the standard intelligent terminal equipment and the temperature change;

Obtaining the device parameters of the smart terminal device to be tested, and obtaining the predicted value of the temperature change of the smart terminal device to be tested based on the correlation model between the device parameters of the standard smart terminal device and the temperature change;

Obtain the actual temperature change values of the intelligent terminal device under test executing different AI industrial applications; obtain the test result of the intelligent terminal device under test according to the predicted temperature change value and the actual temperature change value.

The beneficial effects of the above technical solution are: by constructing the correlation model between the device parameters of the standard smart terminal device and the temperature change, the temperature change of the device under test in different AI industrial applications can be predicted, and by comparing with the predicted value and the actual value, it can be quickly judge the test results of the intelligent terminal to be tested, the test time is short, the test efficiency is high, and it can greatly save time and labor costs.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Zhang, S. (2014) Industry 4.0 and Smart Manufacturing. Mechanical Design and Manufacturing Engineering, 43, 1-5.
[2]	Deng, H.W. (2020) The Application of Artificial Intelligence in Mechanical Equipment Fault Detection. Collections, 11.
[3]	Wang, F. and Di, P. (2021) Equipment Test Evaluation Method Based on Virtual Test Data Fusion. Ship Electronics Engineering, 41, 131-134.
[4]	Winter AIPerf: Large-Scale AI Arithmetic Benchmark Program. Big Data, 7, 2021032.
[5]	Wang, B., Liu, W. and Zhou, C. (2021) Reduce Chip Testing Costs Based on Data Analysis Science. Electronics and Packaging.
[6]	Yu, J., Gao, J., Tang, J.H. and Li, L. (2017) Application of Serial Port Polling in Temperature Testing Based on Virtual Instruments. Industrial Instrumentation and Automation, No. 5, 91-93.
[7]	Wang, Q.D., Jin, G.S., Li, Q., Wang, W., Yang, Z.Y. and Wang, H. (2021) The Present Situation and Prospect of Industrial Edge Computing Research. Information and Control, 50, 257-274.
[8]	Yan, M., Duan, H.C. and Guo, W. (2018) AI Application and Development in the Field of Software Testing. ICT and Policy, 44, 65.
[9]	Wang, Y. and Wang, J.W. (2018) AI Opportunities for Development on Edge Devices. Electronics World, 25, 13-17.