Digitize historic architectural plans with OCR and NER transformer models

The Swiss Archive for Landscaping Architecture, located at OST in Rapperswil, administers more than 100’000 historic plans. These plans need to be digitzed to make them accessible. This paper proposes a three-model architecture consisting of a layout model to find text on the plans, an optical character recognition model to extract the found words, and finally, a named entity recognition model to label the relevant words like the client, location, or date. K-means clustering is used to group the text blocks from the layout model into related blocks for OCR.

Different deep-learning models are compared and evaluated. The most suitable models are then retrained on the NVIDIA DGX-2 system in a custom-built apptainer image with different training strategies to improve their accuracy. Different pre- and post-processing techniques are employed to improve the accuracy of the pipeline.

The final image pipeline achieves an F1 score of 48% with 35% precision and 77% recall. The chosen NER model ”German BERT” scored an F1-score of 86% after re-training and the OCR pipeline extracted 54% of words correctly and 18% close to correct.

The insights from this SA can be applied to future projects to build an application usable by the archive, enabling it to catalog its documents and make them accessible to the world.