Automated 3D Bridge Modeling from Non-Layered Drawings Using Transformer-Based Deep Learning

Keitaro Taniguchi1 and Tatsuro Yamane2[0000-0003-4027-1959]

1 National Institute of Technology, Tokuyama College, Gakuendai, Shunan, Yamaguchi, 745-8585, JAPAN

2 National Institute of Technology, Tokuyama College, Gakuendai, Shunan, Yamaguchi, 745-8585, JAPAN

yamane@tokuyama.ac.jp

Abstract. To improve the efficiency of maintaining existing infrastructure, such as bridges, there is growing interest in utilizing 3D models that represent damage conditions and structural information. Current approaches for creating 3D models of existing infrastructure include photogrammetry and reconstruction from technical drawings; however, fully automated approaches have yet to be realized. This study proposes a methodology for automating 3D model construction by extracting dimensions from blueprint images using a Transformer-based object detection model and Optical Character Recognition (OCR). In the initial phase, the object detection model identifies each numerical text string in the drawing, focusing on classifying its semantic meaning (e.g., girder height, road width), rather than the numerical value itself. Subsequently, OCR extracts the specific numerical values from these detected text strings, linking each numerical value with its corresponding meaning within the drawing, enabling automated 3D model construction. The proposed method was applied to actual bridge drawings, successfully generating accurate 3D models. Further development of this approach into a practical application could lead to more accessible and user-friendly 3D models.

Keywords: 3D Modeling, Object Detection, Transformer.

Introduction

The deterioration of infrastructures, including bridges, has become a critical global issue. In Japan, many structures built during the period of rapid economic growth are now aging. According to the Ministry of Land, Infrastructure, Transport, and Tourism, by 2030, 54% of road bridges in Japan will have been in service for over 50 years. Similarly, in the United States, a 2021 report indicated that 42% of all bridges had exceeded 50 years of service, with 7.5% classified as being in poor condition due to structural deficiencies. Additionally, severe labor shortages in the construction industry have led to inadequate bridge maintenance. Addressing this issue requires enhancing the efficiency and sophistication of maintenance processes to improve overall productivity.

Given this background, there is growing interest in leveraging machine learning technologies and 3D data for infrastructure maintenance. The use of 3D data for bridge maintenance facilitates the tracking of temporal changes in damage conditions and the systematic analysis of spatial relationships between different types of damage. This approach can contribute to more accurate structural diagnosis.

However, most existing structures lack pre-existing 3D models. Consequently, many studies focus on reconstructing 3D shapes from images using techniques such as Structure from Motion/Multi-View Stereo (SfM/MVS) or acquiring surface geometry using Light Detection and Ranging (LiDAR).

Nevertheless, the accuracy of generated 3D data depends on the quantity and quality of captured images or measured point clouds. Additionally, for large-scale or geometrically complex bridges, data collection takes an enormous amount of time and money. Moreover, utilizing BIM or FEM analysis requires converting detected surfaces from point cloud data into 3D models. However, achieving accurate modeling presents numerous challenges that must be addressed.

To overcome these challenges, constructing 3D models directly from design documents and inspection reports, which are already created during the design and inspection phases, would eliminate the need for additional data collection, thereby reducing associated costs. Therefore, this study proposes a methodology for the automatic generation of 3D bridge models based on design drawings and inspection reports. The study also develops an algorithm for this approach and applies the proposed method to multiple drawings extracted from an inspection record to construct a 3D model.

Automated 3D modeling method

Related Studies and Challenges

Several studies have been conducted on generating 3D models from drawings of existing structures. Lu et al [1]. presented a semi-automated approach for extracting information from CAD drawings to construct 3D models. Furthermore, Zhao et al [2]. demonstrated a method utilizing Faster R-CNN to detect structural members from drawings and generate BIM data.

However, existing approaches require predefining various patterns, such as the spatial relationships of structural members and the locations of dimension annotations in drawings, which must be incorporated into the algorithm. To apply these methods to a wide range of drawings, it is necessary to comprehensively account for all possible variations, which inherently limits versatility and scalability. This limitation is particularly significant for civil engineering structures, including bridges, as they are typically site-specific, one-of-a-kind productions. Given the diversity of drawing formats, applying conventional methods to bridge drawings remains challenging.

On the other hand, if CAD data is available, where dimension information is pre-separated by layers, the difficulty of extracting relevant data is significantly reduced. However, if the layer classification criteria are not clearly defined, data extraction becomes challenging. Additionally, in many cases, CAD data may not exist at all,

Fig. 1. Flow diagram of method proposed in this study.

leaving only PDF or paper-based drawings as the available sources of information.

To address these challenges, this study proposes a methodology that leverages an object detection model incorporating a Transformer, which can effectively capture spatial relationships between objects within an image. This approach, as shown in Fig. 1, enables the automatic identification of the positions of textual elements in drawings stored as images and classifies the meaning of each detected text. By extracting numerical values from the detected text and linking them to their corresponding meanings, the proposed method enhances the scalability of 3D model generation.

Classification of Dimension Types Using Detection Transformer

Detection Transformer. In the proposed method, the first step is to determine the positions of textual elements in a drawing image and classify their meanings (e.g., bridge width, span length, girder height). In recent years, Deep Learning-based machine learning algorithms have been widely applied for object detection in images. Typical Deep Learning-based object detection models, such as Faster R-CNN, employ Convolutional Neural Networks (CNNs) to extract feature maps from candidate regions. These feature maps are then used to predict bounding boxes and classify detected objects. In this process, shape-based features are primarily used for object detection and classification. However, in bridge drawings, dimension values are represented as simple sequences of numbers. Since different dimension types share similar numerical appearances, conventional object detection models, such as Faster R-CNN, which primarily rely on shape-based features, struggle to distinguish dimension types accurately without considering their positional context within the drawing. To address this issue, this study employs a dimension information extraction model based on Detection Transformer (DETR) [3], which incorporates a Transformer mechanism that considers the spatial relationships of input elements in its output. This approach enables more accurate dimension classification by incorporating positional relationships between dimension values.

Fig. 2. Sample images from the constructed dataset

Model Training. In this study, DETR is applied to drawings within bridge inspection reports to detect dimension positions and classify dimension types. Since the placement and types of dimensions differ significantly between cross-sectional views and side views, two separate models were developed: one for cross-sectional view analysis and another for side view analysis. Each model is trained to detect and classify dimensions based on the characteristics unique to its respective type of drawing.

Since manually creating annotation data to specify the positions and types of dimensions based on actual bridge drawings is time-consuming, a program was developed to automatically generate synthetic drawings resembling bridge blueprints (2048 px × 1024 px), as shown in Fig. 2, to create training data. Simultaneously, annotation data specifying the meaning and position of each numerical value was generated from the same information.

Note that if the dataset were created using commonly used dimension values for actual bridges, the model might learn to classify dimension types based on the numerical values themselves, rather than their placement in the drawing. Ideally, the meaning of a dimension should be inferred from the shape of structural members, leader lines, and other contextual information, without relying on specific numerical values. Using realistic values could therefore reduce the model’s generalizability. To ensure that the model learns to classify dimensions based on their spatial positions rather than their values, the dimension values in the synthetic drawings were generated as randomized text strings. The dataset for training the models consisted of 2,000 training images and 100 validation images for cross-sectional views, and 500 training images and 25 validation images for side views.

For training, ResNet50 was used as the Backbone Network in DETR. The hyperparameters were set identically for both the cross-sectional view and side view models, with the learning rate for the Backbone Network set to 10-5, the overall model learning rate set to 10-4 and the batch size set to 2. The cross-sectional analysis model was trained for 120 epochs (1000 steps per epoch), and the side view analysis model for 200 epochs (250 steps per epoch).

Fig. 3. Detection results for test images

Table 1. Accuracy of each model (%)

AP@.5	AP@.75	AP@[.5:.05:.95]
cross-sectional view analysis model	92.9	49.8	50.3
side view analysis model	69.6	34.3	36.3

Accuracy Evaluation. The trained models were evaluated by detecting dimension values in test images, and their accuracy was assessed. The test images were generated programmatically, using the same method as the training dataset. For each model, 100 test images were used to ensure a consistent evaluation.

An example of the detection results is shown in Fig. 3. The model successfully detects and classifies the positions and types of dimension values within the drawings with high accuracy.

The detection accuracy of each trained model was quantitatively evaluated based on the test image results. The accuracy evaluation results for each model are summarized in Table 1. In the cross-sectional view analysis model, the AP@.5 exceeded 90%, indicating that the model can classify the meaning of dimensions with very high accuracy. Additionally, the model achieved approximately 50% accuracy in detecting dimension regions for both AP@.75 and AP@[.5:.05:.95].

On the other hand, in the side view analysis model, the AP@.5 reached approximately 70%, demonstrating high detection accuracy. However, the AP@.75 and AP@[.5:.05:.95] values were around 30%. This lower performance at higher IoU thresholds is attributed to the strictness of the 0.75 and 0.95 IoU thresholds, which impose very stringent localization accuracy requirements. However, as discussed in the following section on dimension value recognition, some tolerance in detection position offsets is acceptable. Therefore, the relatively lower accuracy at AP@.75 and AP@[.5:.05:.95] does not significantly impact the overall effectiveness of the methodology.

Fig. 4. Numeric recognition results

Dimension Value Recognition Using OCR

Value Recognition. In this study, the bounding box information detected by DETR was first used to extract individual images containing each dimension value from the original input image. Slight misalignment of DETR-detected bounding boxes could result in partially cropped text, reducing OCR accuracy. To mitigate this issue, the width and height of each cropped region were expanded by 20%, ensuring that the entire dimension value is captured within the extracted image. After extracting the dimension images, dimension values were recognized using PaddleOCR [4], a Deep Learning-based OCR algorithm. These recognized values were then assigned to their corresponding dimension types, as determined by DETR.

Accuracy Evaluation. In the previous section, the dimension position detection using DETR was performed on randomized text strings. Therefore, 600 test images simulating dimension values were programmatically generated to evaluate PaddleOCR’s numerical value recognition accuracy. The test images were created as 128 px × 64 px white background images, with random numerical values rendered in various fonts. To simulate noise typical of actual bridge drawings, straight lines were added to the images.

An example of the recognition results is shown in Fig. 4, with an overall recognition accuracy of 84%. It was noted that dimension values with smaller occupied regions relative to the image size, as well as those with thin-stroke characters or partially occluded digits (e.g., the digit “7”), were especially prone to missed detections or misrecognition. However, actual bridge drawings typically have fewer font variations and generally use high-visibility fonts. Therefore, it is expected that recognition accuracy in practical applications will be higher than the percentage reported in this test.

3D Mesh Modeling

Finally, using the dimension values extracted through OCR, a 3D model was constructed. The bridge inspection report drawings used in this study are simplified schematics, which do not explicitly provide detailed dimensions for all structural members. To address this limitation, assumptions were made for elements such as girder spacing and width, assigning identical dimensions to multiple similar members. Based on these assumptions, a simplified 3D model was generated. Additionally, the 3D model constructed in this study was generated as a mesh in STL format. In this

Fig. 5. Detection results by DETR

approach, the extracted dimension values and their corresponding types were used to calculate vertex coordinates.

Application to Actual Bridge Drawings

To validate the practicality of the proposed algorithm, a 3D model was generated from an actual bridge drawing included in a bridge inspection report. In this study, schematic drawings included in the bridge inspection report were used as input. The detection results are shown in Fig. 5. Both cross-sectional and side view drawings were successfully processed, with all dimension values being correctly detected and classified. Additionally, the confidence scores for all detected dimensions exceeded 95%, indicating high reliability of the developed model in interpreting dimension meanings from actual bridge drawings.

Using the detected text positions, corresponding dimension value images were extracted and analyzed by PaddleOCR. In this test case, all extracted images were correctly recognized, demonstrating the accuracy of the OCR process. The recognized dimension values were linked to their corresponding types determined by DETR and structured for automated 3D model construction.

The automatically generated 3D model based on recognized dimension values and types is shown in Fig. 6. The 3D model was successfully constructed according to the input drawing, confirming that the proposed method effectively generates accurate 3D models.

Conclusion

In this study, a methodology was proposed for automatically constructing 3D models from extracted information by applying a Transformer-based object detection model to imaged drawings. Through this process, a fully automated pipeline was developed for converting 2D bridge drawings into 3D models. Based on the proposed method, a 3D mesh model was successfully and accurately generated from the cross-sectional and side view drawings in an actual inspection report.

Fig. 6. Automatically constructed 3D model

Practical implementation requires developing a dedicated front-end application for automated 3D model generation. Additionally, organizing the extracted dimension types and values in a structured format, such as JSON, enables creating schemas representing various bridge specifications. This structured data can facilitate the development of BIM (Building Information Modeling) models for maintenance purposes and FEM (Finite Element Method) models for structural analysis, contributing to more advanced infrastructure management and evaluation.

Acknowledgements

This study was supported by JSPS KAKENHI Grant Number JP24K17333 and by an academic-government collaborative research project with the Civil Engineering and Architecture Department of Yamaguchi Prefecture.

References

Lu, Q., Chen, L., Li, S., and Pitt, M.: Semi-automatic geometric digital twinning for existing buildings based on images and CAD drawings, Automation in Construction, Vol.115, 103183, 2020.

Zhao, Y., Deng, X., and Lai, H.: Reconstructing BIM from 2D structural drawings for existing buildings, Automation in Construction, Vol.128, 103750, 2021.

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S.: End-to-End Object Detection with Transformers, Computer Vision – ECCV 2020, pp.213–229, 2020.

Du, Y., Li, C., Guo, R., Yin, X., Liu, W., Zhou, J., Bai, Y., Yu, Z., Yang, Y., Dang, Q., and Wang, H.: PP-OCR: A Practical Ultra Lightweight OCR System, arXiv preprint arXiv:2009. 09941, 2020.