ENP-Corpus Creator: An Application for Transforming Scanned Images into Semantically Enriched Corpora

This document introduces the ENP-Corpus Creator application, developed by the ENP-China project. ENP-Corpus Creator is a comprehensive suite of tools designed to facilitate the critical steps involved in transforming print sources into a semantically enriched, full-text version of these sources.

The initial version of the application was developed in early 2024 by Baptiste Blouin, a computer scientist with the ENP-China project. Since then, the toolset has been significantly enhanced based on feedback from expert annotators (historians) after multiple rounds of testing on sample sources. The version we present today is the product of these productive, iterative exchanges between historians and computer scientists. Although the functions were initially developed to meet the specific needs of a small team of historians specializing in China, we believe they are versatile enough to accommodate a broader range of sources and serve a wider community of researchers working primarily with print texts.

The ENP-Corpus Creator suite comprises several features, each corresponding to a critical step in the process from scanned images to semantically enriched full text, as illustrated below:

1. File Preparation

Functions:

• Pdf to Image
• TIF to PNG

To begin, you will need a digitized version of your source with sufficient resolution (at least 300 dpi). Supported formats include PDF, TIFF, PNG, and JPG. ENP-Corpus Creator provides two functions to help prepare your file in the appropriate format:

1. Pdf to Image: This function automatically converts PDF files into separate images, optimizing for the best resolution while maintaining a manageable file size.
2. TIF to PNG: If your documents are originally in TIFF format, ENP Corpus Creator provides a function to convert TIFF files to PNG, as the Optical Character Recognition (OCR) function in ENP-Corpus Creator only accepts PNG or JPG files (see step 2).

2. Optical Character Recognition (OCR)

Functions:

• Google Credentials
• OCR Data

After preparing your images in a suitable format, the next step is to apply Optical Character Recognition (OCR) to extract text from the images. ENP-Corpus Creator integrates Google Vision’s technology into its workflow, chosen for its superior performance compared to alternatives like Tesseract or ABBYY FineReader, particularly with documents that have complex layouts (e.g., multiple columns, multiple languages, and various reading directions). Although Google Cloud is a paid service, it offers a three-month free trial that allows processing of a substantial number of documents before reaching any limits, and you can cancel the trial at any time without incurring charges.

To use the OCR Data function, you must first activate Google Vision Cloud on your Google Account and generate a JSON key. This key is required for the Google Credentials function within the ENP-Corpus Creator application.

Once the key is uploaded, you can apply the OCR Data function. This function generates a `.txt` file for each image file, creating image/text file pairs that will be used in the next step (Annotation), as shown below:

3. Annotation

Functions:

• Annotate
• Script

The annotation step is designed to create meaningful semantic units from the raw text, such as identifying and classifying individual articles within a journal issue. This process is essential for enabling refined, targeted searches beyond the page level and making connections across pages, for instance, linking articles that span multiple pages or sections that appear in successive issues.

Before beginning the annotation task, it’s recommended to examine several sample pages to define the labels and segmentation levels that best align with your research needs. These labels will serve as metadata that you can reuse in the digital version of your text, so it’s important to plan them thoughtfully in advance.

Expert annotators involved in the development of ENP-Corpus Creator have established a set of default annotations based on standard metadata commonly found in textual sources. While these baseline annotations cannot be modified, each user is free to create custom annotations that suit their specific needs. The default annotations are explained on the welcome page of the Annotate function (see below). During the annotation process, you can toggle these instructions on or off by clicking the “Hide Instructions” or “Show Instructions” buttons.

Once you launch the Annotate function, it opens an intuitive, interactive annotation page in your web browser:

To begin, use the “Upload Image/Text” button to upload the full set of image/text file pairs created in Step 2. It is recommended to store all these files in a dedicated folder, where you can also save your annotations.

The annotation interface consists of three main panels, as illustrated below:

1. Left Panel: Displays the image of the page, allowing you to reference the original document during annotation. This is particularly helpful for correcting OCR errors or deciding on appropriate labels for different article levels.
2. Middle Panel: Shows the extracted text, split into individual lines.
3. Right Panel: A blank section where your annotations are stored. When you complete the annotation process for your document set, you will export this data as a “tab” file.

Some lines in the text are highlighted in yellow. These highlights, generated automatically by the application, indicate potential titles, unusual typesetting, or displaced text. This feature allows you to focus on key sections without reading the entire page.

ENP Corpus Creator supports three types of annotations: reordering, editing, and labeling.

1. Reordering: If the OCR process has disrupted the reading order, you can adjust it by moving lines into the correct sequence. Simply click on the line(s) you want to reposition and drag them to the desired location.

2. Editing/Correction: You can correct OCR errors or add missing information by clicking on the line you want to edit and pressing “e” on your keyboard. After making changes, click “OK.” You can also add new lines by pressing the green “+” button in the toolbar or delete lines by selecting the line and pressing “d.”

3. Classification/Enrichment: Label lines with specific information, such as genre, section types, or metadata. Available shortcuts include:

• “g” for advertisements,
• “t” for tables,
• “i” for images,
• “c” for tables of contents,
• “n” for footnotes or endnotes,
• “a” for author names, and
• “p” for page numbers.
• Text levels from 1 to 9 for journal sections or book chapters.

To add a label, click on the relevant line and press the assigned key. You can also create custom labels, as long as the key is not already in use.

Note: Ensure you have completed reordering and editing (steps 1 and 2) before proceeding to add labels and metadata (step 3).

Important Tips for Best Practices

1. It is recommended to add a basic description of your document at the top of the annotation tab, such as the source name, publication date, or volume/issue for periodicals. Use “#” to distinguish these descriptive elements from subsequent annotations, as shown in the example below:

2. If you’ve made changes to the text (e.g., editing, reordering, adding, or deleting lines), remember to save your work before moving to the next page. To save, click the red “Save Text” button, replacing the initial file in your folder with the updated text file. A pop-up window will remind you to save before leaving the current page.

3. After annotating a line, it will be highlighted in green (see below). You can still select and edit the line, or modify annotations directly in the Annotation panel.

4. If making manual changes directly in the Annotation tab, be careful not to alter the annotation structure, such as accidentally removing brackets or other important separators. Ensure you understand the syntax before making manual edits.

5. If you need to pause the annotation process, save your progress by clicking the black “Save Notes” button. Your annotations will be saved as a “tab” file, which you can reload later with the red “Upload tab file” button.

Once annotation is complete, use the Script function to document the keys used in your annotations and to add custom metadata.

4. Further Applications: Text Mining with HistText

The resulting text files and annotations can be uploaded to a SolR server to create a new collection in the Modern China Textual Database (MCTB). This collection can then be explored using the HistText application and R package, both developed by the ENP-China project. HistText enables users to build custom corpora, conduct various text analyses, and apply advanced data extraction and Natural Language Processing (NLP) techniques, such as Named Entity Recognition (NER).

Conclusion

ENP-Corpus Creator is ideally suited for small- to medium-sized corpora that can be processed manually within a reasonable timeframe, such as digitized books, archives, or monthly journals (up to several tens of thousands of pages). However, it is not suitable for very large collections, like newspapers spanning several decades, as the annotation workload would be too time-intensive.

While the current version of ENP-Corpus Creator relies on manual annotations, future development aims to incorporate machine learning capabilities, enabling users to train models based on manual annotations from sample pages. These models could then be used to automatically segment and classify articles across larger collections of similar texts—a functionality already available in applications like Transkribus and eScriptorium.

In addition to being entirely free (download link), ENP-Corpus Creator offers several distinct advantages over existing applications. It is efficient and user-friendly, more flexible than traditional encoding systems like the Text Encoding Initiative (TEI), and does not require programming skills, unlike other excellent methods such as OCR with Google Vision and Tesseract.

---

---