Image background filtering for concentrated damage detection via fusion with point cloud data

Chao Lin1[0000-0002-1517-8392], Yu Chen1[0000-0002-1187-4761], Kenta Itakura2[0000-0001-7259-271X] and Pang-jo Chun1[0000-0002-9755-8435]

1 The University of Tokyo, Tokyo 113-8485, Japan

2 ImVisionLabs Inc., Tokyo 113-8485, Japan
chun@g.ecc.u-tokyo.ac.jp

Abstract. With the increasing demand for efficient and accurate structural health monitoring, the integration of multi-source data has gained growing attention in infrastructure operation and maintenance (O&M). Among them, images and point cloud data (PCD) are widely used due to their complementary characteristics—images provide detailed texture information, while PCD offers precise geometric structure. However, their joint application remains underdeveloped, limiting their potential for mutual reinforcement. To address this gap, this study proposes a novel method for image background filtering based on image-PCD fusion. High-resolution images and PCD are simultaneously captured using a top Lidar 3D camera, with spatial correspondence preserved via the E57 file format. After extraction, points corresponding to target components are projected onto 2D images to generate accurate masks for cropping. This process effectively removes complex and irrelevant background elements, which often cause misclassification in damage detection tasks. Experimental validation on a prestressed concrete bridge shows that the proposed method significantly improves the performance of a deep learning-based crack detection model. By directing the model’s focus toward actual structural surfaces, the method enhances detection accuracy and reduces false positives. This approach offers a practical and efficient solution for improving damage detection and overall assessment of infrastructure conditions in real-world scenarios.

Keywords: Image Background Filtering, Damage Detection, Point Cloud, Data Fusion.

Introduction

With the rapid development of deep learning (DL) models in image segmentation tasks, image-based techniques have become the dominant method for damage detection in civil engineering [1–3]. A significant challenge associated with image-based damage identification is the risk of background elements being misclassified as damage, leading to false positives and reducing overall detection accuracy. To mitigate this limitation, filtering out background elements is essential for improving the precision of damage detection. Numerous studies have shown that damage detection models achieve higher accuracy when concentrated on localized areas of the damaged surface rather than processing entire images with irrelevant background information [4]. However, achieving effective background filtering becomes problematic when relying exclusively on images as variations in lighting conditions can obscure components and lead to reduced accuracy.

Point cloud data (PCD) presents a promising solution to this challenge by enabling background filtering through 3D-to-2D projection. By integrating PCD with images, structural components of interest can be precisely segmented while irrelevant background elements are removed. However, the effectiveness of any image-PCD fusion application depends on the quality and alignment of both kinds of data. High-quality data with well-established relationships between images and PCD are essential for achieving accurate background filtering and reliable damage detection. While it is possible to obtain one type of data and generate the other through post-processing or to collect them separately and align them afterward, such approaches often introduce errors and inconsistencies. In contrast, using a top light detection and ranging (Lidar) 3D camera ensures both high data quality and precise correlation between images and PCD.

In this study, we propose an image background filtering method that enhances damage detection through image-PCD fusion. High-resolution images and detailed PCD are simultaneously captured using a top Lidar 3D camera and saved in the E57 file format, which clearly preserves their relationship to support effective management. Then points of interested components are projected onto corresponding images to accurately filter out background elements. An experiment on a real-world bridge demonstrates that our framework can successfully eliminate background interference, allowing the DL model to focus solely on the structural components. Consequently, this leads to a significant improvement in detection accuracy, reducing false positives and enhancing the reliability of the damage assessment process.

Methodology

The E57 file format [5] is an innovative and versatile standard that is widely adopted for storing PCD, images, and metadata in a single, unified structure. Unlike traditional formats that store PCD and images separately, the E57 format natively records not only the data itself but also the critical sensor pose information that is necessary to mate it together. A top Lidar 3D camera (Matterport Pro3 [6]) that supports the E57 format is selected as the data collection equipment. During each scan session, the device generates a single PCD file in its local coordinate system, accompanied by six panoramic images captured from multiple directions—specifically top, bottom, front, back, left, and right. By jointly leveraging the top Lidar 3D camera and the E57 format, the built-in spatial information is organized to accurately align PCD with corresponding images, enabling seamless data fusion and eliminating the errors typically introduced through manual registration or post-processing techniques.

The 3D-to-2D projection between a 3D point P and its corresponding 2D point p can be represented as

(1)

where (u, v) and (X, Y, Z) respectively denote the coordinates of p in the image coordinate system and the corresponding point P in the world coordinate system, K and are intrinsic and extrinsic parameter matrices that can be acquired by camera and Lidar calibration. Notably, six groups of K and are needed for Matterport Pro3 because of the six image directions.

The workflow for image background filtering is illustrated in Fig. 1. First, the 3D points of the target components are transformed into their local coordinate system and projected onto each corresponding image. Images that do not contain projected points can be discarded. Then a polygon mask is generated for each remaining image that encloses the projected points using geometric algorithms such as the alpha shape algorithm [7] which effectively captures the concave boundary of a point set in 2D space. Finally, each image is cropped using the generated polygonal mask, resulting in background-filtered images that isolate the structural components of interest. These processed images are then ready to be fed into the DL model for damage detection, thereby reducing background interference and improving detection accuracy.

Fig. 1. The workflow of image background filtering.

Experimental validation

To validate the proposed framework in a real-world scenario, a prestressed concrete bridge named Kizuna Bridge was selected as the experimental subject. Kizuna Bridge locates at Fukushima Prefecture, Japan. It was built in 2011 with a length of 11.1 m and a width of 6.7 m. Considering its size, Kizuna Bridge was scanned from 22 spots using Matterport Pro3. Following data acquisition, 22 PCD files and 132 images were extracted from the E57 file. All PCD files were merged with corresponding scan indices and annotated into five categories, namely abutment, girder, deck, parapet, and others. Fig. 2 shows the original and annotated PCD of Kizuna Bridge.

Fig. 2. Original and annotated PCD of Kizuna Bridge.

Image background filtering

Among all component categories, the girder appeared in 77 images which accounts for the most, making it the optimal target for evaluating the proposed background filtering method. Points corresponding to the girder were extracted from the complete PCD and then redistributed into 22 individual PCD files based on their respective scan indices. Then all 132 images were processed through the workflow illustrated in Fig. 1 with alpha set to 500 in the alpha shape algorithm. As a result, all 77 images that contain girder were successfully identified. The workflow demonstrated excellent performance in image background filtering, as shown in Fig. 3. The girders were accurately extracted from the background and undesired component categories. Notably, the segmentation masks aligned precisely with the girder’s irregular curves, highlighting the method’s strong effectiveness.

Fig. 3. Results of image background filtering.

Crack detection

Cracks are the most commonly targeted damage type in image-based detection studies. Therefore, crack detection was carried out on the Kizuna Bridge to evaluate the improvements brought by the background filtering method. DeepLabv3+ [8] was selected as the detection model due to its widespread adoption and proven effectiveness. The model was trained using an open-source crack dataset [9] comprising over ten thousand images. The dataset was randomly split into 90% for training and 10% for testing. The trained model was then evaluated on the testing subset, with the results presented in Table 1.

Table 1. Training results (%) of DeepLabv3+ for crack detection.

Overall accuracy	Mean accuracy	Mean Intersection over Union
98.15	81.64	73.84

The girder and deck were selected as the target components for crack detection, as these areas are prone to cracking. A total of 78 images containing both girder and deck were successfully selected and filtered using the proposed workflow. These filtered images were then input into the trained model for crack detection. The results are illustrated in Fig. 4. When background filtering was not applied, edges of background elements—such as vegetation, buildings, and wires—were often misidentified as cracks. By removing the background, crack detection accuracy improved significantly. The proposed method effectively reduced interference from background lines, thereby enhancing the performance of the deep learning model.

Fig. 4. Examples of crack detection results before and after background filtering.

Conclusion

This paper presents an image background filtering method based on image-PCD fusion to enhance the performance of DL models by directing their focus toward actual damage surfaces. A novel data acquisition and organizational framework was employed, utilizing a top Lidar 3D camera and the E57 file format to capture high-quality data while clearly preserving spatial relationships between images and PCD. Target components were identified in the PCD and projected onto corresponding images to generate precise masks, which were then used to crop out irrelevant background regions. Experimental results on a bridge demonstrate that the proposed method effectively filters out complex and irregular backgrounds, significantly reducing the likelihood of misclassification and thereby improving the accuracy of damage detection.

To further enhance the automation, scalability, and versatility of this approach, several future research directions are proposed. First, replacing the time-consuming manual annotation of PCD with DL-based semantic segmentation techniques would greatly streamline the workflow [10]. Second, incorporating additional types of structural damage into the training dataset could provide a more comprehensive assessment of structural health [11]. Finally, integrating the detected damage back into the PCD as 3D visualizations would enable advanced structural condition modeling and support the development of functional digital twins for infrastructure management [12].

References

Chu, H., Deng, L., Yuan, H., Long, L., Guo, J.: A transformer and self-cascade operation-based architecture for segmenting high-resolution bridge cracks. Automation in Construction 158, 105194 (2024).

Chu, H., Chun, P.J.: Fine‐grained crack segmentation for high‐resolution images via a multiscale cascaded network. Computer‐Aided Civil and Infrastructure Engineering 39(4), 575–594 (2024).

Chun, P.J., Kikuta, T.: Self‐training with Bayesian neural networks and spatial priors for unsupervised domain adaptation in crack segmentation. Computer‐Aided Civil and Infrastructure Engineering 39(17), 2642–2661 (2024).

Xiao, J.L., Fan, J.S., Liu, Y.F., Li, B.L., Nie, J.G.: Region of interest (ROI) extraction and crack detection for UAV-based bridge inspection using point cloud segmentation and 3D-to-2D projection. Automation in Construction 158, 105226 (2024).

Committee E57 on 3D Imaging Systems, https://www.astm.org/get-involved/technical-committees/committee-E57, last accessed 2025/01/21.

Matterport Pro3, https://matterport.com/pro3, last accessed 2025/01/25.

Edelsbrunner, H., Kirkpatrick, D., Seidel, R.: On the shape of a set of points in the plane. IEEE Transactions on information theory 29(4), 551–559 (1983).

Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Laura Leal-Taixé, Stefan Roth (eds.) European conference on computer vision (ECCV) 2018, LNCS, vol. 11129, pp. 801–818 Springer, Munich (2018).

Concrete Crack Conglomerate Dataset, https://data.lib.vt.edu/articles/dataset/Concrete_Crack_Conglomerate_Dataset/16625056, last accessed 2025/02/24.

Lin, C., Abe, S., Zheng, S., Li, X., Chun, P.J.: A structure‐oriented loss function for automated semantic segmentation of bridge point clouds. Computer‐Aided Civil and Infrastructure Engineering (2025).

Santos, P.A.S., Carvalho, M.T.M.: Multi-class segmentation of structural damage and pathological manifestations using YOLOv8 and Segment Anything Model. Automation in Construction 172, 106037 (2025).

Gao, Y., Li, H., Fu, W., Chai, C., Su, T.: Damage volumetric assessment and digital twin synchronization based on LiDAR point clouds. Automation in Construction 157, 105168 (2024).