Bridge Girder Crack Assessment Using Faster RCNN Inception V2 and Infrared Thermography ()
1. Introduction
Bridges constitute a sizable number of infrastructures in our environment and are often quite expensive to construct and maintain. Their integrity, safety, sustainability, reliability and maintenance are as important as the initial construction. These factors, however, are often impeded by deteriorating effects due to age and long-term service and exposure to harsh environmental conditions such as wind and earthquakes. In order to mitigate this fast deteriorating effect of bridges, the science of health monitoring emerged.
It is not a gainsaying that the role of infrastructure is critical, and serves as a significant index, in measuring the development of a nation. However, modern challenges in infrastructure development transcend merely building of roads, bridges, and other social facilities, but rather exploring means to mitigate the deterioration. Certainly, the rate of deterioration of US infrastructure, for example, has been a subject of concern amongst politicians, engineers and the public at large [1]. According to the ASCE 2017 report card, the US infrastructure received a cumulative grade of D+ (i.e. fair condition) with the bridges averaging a grade of C+ (i.e. good) [1] [2].
Collapse of critical infrastructures such as bridges can have far reaching implications on the safety and economy of a nation. The failure of the Silver Bridge in Ohio in 1967 which led to the death of 46 people, the collapse of the I-35 W Bridge in Minnesota in 2007 which killed 13 people, injured 145 and resulted in a direct economic loss of US$17 million and US$43 million in 2007 and 2008, respectively [3] [4], the recent Florida International University (FIU) pedestrian bridge disaster in 2018 and the Tennessee Bridge Railing Collapse in 2019 are relatable experiences showing the implications of bridge collapse.
Bridges, just like other infrastructure, are very costly to construct, and their failure cannot be allowed. To prevent sudden failure, bridges are now routinely inspected for member or connection failures and their performances are measured or predicted regularly, with necessary actions taken to reduce their rate of deterioration. Bridge condition assessment techniques that have been explored include to not limited to: non-destructive techniques e.g. ultrasonic pulse velocity method, impact-echo/impulse-response method, acoustic emission method, radiographic method, eddy current method, eddy current method, and infrared thermographic methods; and dynamic characteristics-based methods e.g. use of natural frequency changes, use of modal damping changes, use of FRF changes, use of mode shape curvature changes, use of modal strain energy changes, and flexibility changes [5]. More recently, the use of techniques, such as Digital Imaging Correlation and various classes of Deep Learning such as Convolutional Neural Network (CNN), Region-CNN (RCNN) and Faster-RCNN, are becoming increasingly popular in identification and classification of defects in concrete structures due to advances in computing as researchers are exploring means to automate structural health assessment.
Deep learning is a machine learning technique that utilizes the deep neural network [6]. It allows computational models of multiple processing layers to learn representations of data with multiple levels of abstraction [7]. By harnessing this feature of machine learning, it is possible to use a well-trained neural network to detect, and classify, defects in concrete structures such as bridges thereby aiding engineering judgements of the conditions of bridges. The Objectives of this paper are to develop a framework that can be used for automation of bridge inspection, train a network that can be used for concrete crack classification, develop an algorithm to obtain info. of crack size and location in structure and build a 3D crack visualization model to assist maintenance engineer to determine whether the crack needs immediate attention.
2. Literature Review
Many researchers developed different image processing algorithms to detect infrastructural health monitoring. But most of the algorithms are limited to analyze a single picture instead of real-time video analysis. Tsai, Y. C. et al. (2009) analyzed Sobel model and Otsu method for pavement crack detection [8]. After comparing Sobel and Otsu method with the segmentation based on fractal method, they found fractal method is better than Sobel and Otsu method to analyze pavement crack surface. They analyzed 1280 × 1024 size pavement crack surface image to compare the results. The Sobel method includes edges due to differential operations and noises. The Otsu method produces error segmentation due to uneven grey-level distribution on background and lower contrast of grey level between crack and background. Some of the existing algorithms with their objectives are discussed in Table 1.
We used Sobel, Canny edge, Roberts, Prewitt, and Laplacian of Gaussian filter detection algorithms to detect crack from a hand mobile phone capture image. But all the algorithm failed to produce satisfactory result to identify the crack as shown in Figure 1. With the same image, we used different corner detection algorithm such as BRISK Feature Corner Detection, Harris Feature Corner Detection, FAST Features Corner Detection, Min Eigen Features Corner Detection, MSER Features Corner Detection, SURF Features Corner Detection, Extract Corner Features, HOG Features Extraction around corner point. Although some of the corner detection algorithm shows better detection compared to Figure 1 algorithms, some of the algorithm’s detections are over-exceeded as shown in Figure 2.
In summary, the available image processing techniques are impractical to use crack detection because of their lack of identification capabilities. Moreover, most of the techniques are limited by a single image while in practice the technique should perform on a sequence of images rather than a single image. In this research a state-of-art technique named as a convolution neural network to produce useful detection information.
3. Data Acquisition
In this work, we used UAV (i.e. drone) to get access to bridge or anywhere we need to collect images from. With the drone a FLIR 6.8 mm f/1.3 thermal imaging camera were incorporated to capture the thermal images, damages inside
![]()
Table 1. Comparison of image processing models [1].
![]()
Figure 1. Crack detection using (b) Sobel, (c) Canny edge, (d) Roberts, (e) Prewitt, and (f) Laplacian of Gaussian filter detection.
![]()
Figure 2. (a) BRISK Feature Corner Detection, (b) Harris Feature Corner Detection, (c) FAST Features Corner Detection, (d) Min Eigen Features Corner Detection, (e) MSER Features Corner Detection, (f) SURF Features Corner Detection, (g) Extract Corner Features, (h) HOG Features Extraction around corner point.
structural components can be identified, classified and evaluated. A short tutorial was made for the demonstration of data acquisition in this YouTube video link (https://www.youtube.com/watch?v=q4ZDH7MDh9E). A total of 2088 images of crack and not-crack were collected [37]. All the images were split into two groups of images: one for training and the other for testing or validation purpose. In the next step, a neural network model was developed and compared it with a commercially available deep convolutional Neural Network by using the labeled training images. Finally, the trained network applied to classify unknown images. Figure 3 shows the conceptual framework of data acquisition and model training. Figure 4 and Figure 5 show the DJI Panthom 4 pro Drone and the infrared thermal camera respectively. The difference between the infrared image and the naked eye image were shown in Figure 6.
4. Data Pre-Processing
All the images were labeled and annotated using LabelImg Software. The images
![]()
Figure 3. A conceptual framework of data acquisition and model training.
![]()
Figure 6. Comparison of the Infrared image and the naked eye image.
were annotated as crack and non-crack image types. Additionally, PIX resizer software was used to reduce the image dimension to 838 × 600. Image data augmentation technique was implemented to enlarge the training and validation data sets. Figure 7 displays the sprite image of all 2086 images (before augmentation) used in this study.
5. Model Development
A 22-layer (20 hidden layer, one input layer and one output layer) convolution model was developed names as Visual Geometry Group (VGG 22). During the training process, different ratios of train vs. test images were implemented. One example is 1668 images used for training the model and 418 images used for validating purpose. The input size of images was maintained as 224 × 224 × 3 (height × width × channel). In Figure 8, the VGG 22 model architecture is described.
6. Model Training
To train the model, a high-performance supercomputer in UTC SimCenter was used to run the deep learning simulations. Python is the computational tool used to implement the deep learning for damage assessment. During the VGG-22 model training, we tried multiple combinations of epoch and batch sizes. Ten
![]()
Figure 7. Sprite image of infrared thermography.
percent of all images were used for the testing/validation and the rest images were used for training purpose. Data augmentation techniques were deployed to enlarge the training image dataset. Figure 9 represents a feature map of the crack image in hidden layer 3 after the application of 32 filters. Stochastic Gradient Descent (SGD) optimizer had been used to update the network. We get to a validation accuracy more than 78% (Figure 10), which is pretty good considering that the images are noisy and limited number of images are used for training the network. The gap between the training and validation accuracies because of overfitting. It is observed from Figure 11 that the loss tends to be relatively stable after 20 epochs. Figure 12 demonstrates the confusion matrix which is erroneous detection of the model. It measures the performance of an algorithm to detect the accurate class.
Although the VGG-22 model has 78% validation accuracy, faster RCNN model was chosen to increase the validation accuracy. The architecture of Faster RCNN has two network modules: first module is region proposal network (RPN) and the second module is used to generate bounding box for the detected objects [38]. With Faster RCNN, Inception v2 was used as a feature extractor.
![]()
Figure 9. VGG-22 feature map of hidden layer 3.
![]()
Figure 10. VGG-22 training vs. validation accuracy with respect to number of epochs.
![]()
Figure 11. VGG-22 training and validation loss with respect to number of epochs.
The feature extractor defines as the pattern recognition image processing algorithm. The input image size was taken as 838 × 600. The model was trained up to 60 k global steps. After the training, the model reached almost zero loss as shown in Figure 13. In Figure 14, the localization loss and in Figure 15 localization loss were shown compared to number of global steps.
7. Crack Location
Google Earth and Geotag software were used to retrieve the GPS coordinates
![]()
Figure 13. Faster RCNN total loss vs. global steps.
![]()
Figure 14. Faster RCNN classification loss vs. global steps.
![]()
Figure 15. Faster RCNN localization loss vs. global steps.
information. Figure 16 shows the Google Earth view and the crack location in the UTC old garage building. To find the GPS coordinate of crack images, an open-source software, named as Geotag, was used to extract the image latitude, longitude, altitude, county name, city name, and province. The software can generate GPS coordinate based on the image metadata. In Figure 17, the red marked box shows the sample of the GPS coordinate found from our collected image data.
8. Crack Length
3D Drone mapping was used to determine the area and the length of cracks. Pix4D software was used to process the aerial photogrammetry images into 3D modelling. Using Pix4D software dimension tool the cracks measurement was completed as shown in Figure 18.
9. Results
The VGG-22 model shows 80.35% validation accuracy and 70.03% training accuracy at 19th global steps. The faster RCNN with Inception V2 shows 99% confident at 60 k global steps. A test image was provided in the faster RCNN and the model successfully detected the crack and non-crack structure with an average confident 99% as demonstrated in Figure 19. The 99% confidence means the trained model is 99% confident to classify the crack and non-crack image which is the best confident model percentage.
10. Conclusions
The bridge girder health assessment using deep learning with infrared thermography and drone is pushing the technology of traditional health monitoring. This method can save time, cost and make the structural health assessment more robust and efficient.
![]()
Figure 16. Location of crack (Google Earth view).
![]()
Figure 17. GPS coordinate (latitude, longitude, and altitude) generated by Geotag Software.
In this article, first we developed a 22-layer model and then used an open source model called faster RCNN with Inception V2. The VGG-22 model shows 80.35% validation accuracy and 70.03% training accuracy at 19th global
![]()
Figure 19. Crack detection using infrared thermography.
steps. The faster RCNN with Inception V2 shows 99% confidence level at 60k global steps. The faster RCNN with Inception V2 model can successfully detect the crack and non-crack image based on the training output. Moreover, infrared thermography images overcome the limitations of normal images especially dark place or inside of a structure. Using infrared thermography with convolution neural network and drone for the structural health assessment is a beneficial and cost saving technology.
Acknowledgements
Supercomputer resources used for deep learning simulations provided by Dr. Ethan Hereth and UTC SimCenter is greatly appreciated. We are also grateful to Dr. Yu Liang at UTC Department of Computer Science for his valuable suggestions and comments.