A historian’s view of NER (named entity recognition)

Following up on Pierre’s inspiring post, I wish to start a dialogue between historians and computing specialists around the tools we share in the ERC project. This series of methodological posts also answers our common concern with keeping track of the research process, making transparent every decisions we make and the procedures we follow at every stage of the project – particularly for our remote colleagues in Taiwan, Bristol or Paris.

When I first heard of machine learning about three years ago, it sounded like science fiction. That we could apply that “sectarian doctrine” to historical research went beyond my imagination. As a PhD candidate struggling to complete her dissertation, I preferred to stick to more dependable and established tools at the time – Geographical Information System (GIS) or relational databases. This post today records my first steps into machine learning. What happened in the meantime? How did I convert to the devil?

Our project makes the hypothesis that machine learning (ML) and natural language processing (NLP) tools offer promising ways to extract information from historical sources at an unprecedented scale. Yet most of the tools available off the shelf today have not been designed for historical purposes. Entity corpora for instance have been built from contemporary texts with contemporary concerns. In order to be able to extract relevant information from our historical sources, we first need to train the machine to recognize entities that might interest us as historians. Put another way, we need to build a corpus of entities that are suited for historical purposes in general – and for our specific research questions as historians of elites in modern China.

For this purpose, we — Pierre (our NLP specialist), Christian and I (both historians) — decided to conduct a small experiment with named entity recognition (NER). We started from Pierre’s preliminary experiment with the Boorman dictionary. Although the entity corpus applied by Pierre on the Boorman was built from contemporary sources of a different nature (1980s American newspapers) with specific concerns (security, terrorism, criminality), we found that most entities were historically relevant, at least at the most general level (date, location, person, organization).

The question then was to determine precisely to what extent the existing corpus matched our own concerns, how we could correct possible mismatches and fill the gaps. We discussed with Pierre the best method to proceed. We had two options:

  • Either to correct/reannotate the text samples already processed by Pierre using Standford CoreNLP
  • Either to start from scratch and annotate new samples of text – blank from any previous annotations

Pierre recommended that we choose the second solution because it would allow us to start from our own needs instead of being driven by a priori categories.

Interface of the biographical database developed at the Institute of Modern history, Taipei, Taiwan

For Christian and I, the first step was to select relevant people and text samples from the directories made available through “The Modern and Contemporary Person Integrated Information System” project developed at the Institute of Modern history, Taipei, Taiwan. We started from the Boorman’s index and selected a sample of five representative actors from different backgrounds (politics, business, intellectual, and a woman) in Powell’s directories, spanning from 1918 to 1940:

  • Chang Chia-ao 張公權  [Zhang Jia’ao] (1918, 1931, 1936) (business)
  • Soong T. V. 宋子文 [Song Ziwen] (1928, 1936) (business/politician)
  • Wu T’ing-fang 伍廷芳 [Wu Tingfang] (1918) (politician)
  • Feng Yü-hsiang 馮御吞 ( 馮玉祥) [Feng Yuxiang] (1920, 1931, 1940 (military)
  • Liang Ch’i-ch’ao 梁啟超 [Liang Qichao] 1918, 1931 (intellectual)
  • Cheng Yü-hsiu 鄭毓秀 [Zheng Yuxiu] (1928, 1931) (woman)

We decided to  annotate every notice separately — so as not to influence each other — and to compare our respective findings. We annotated the texts using the existing basic categories that Pierre slightly customized to adjust our first reactions to the Boorman’s experiment: person, location (subdivided into country, province, prefecture, district, city); institution (company, organization), miscellaneous, non entities (this served to annotate interesting words that were not entities stricto sensu but seemed important for further research on concepts).

Preliminary Results

Aside for the method we chose for annotating texts (either a linear reading or an entity-driven way of reading), Christian and I generally agreed on which entities we needed to annotate and the names we used to annotate them. The few divergences stemmed from our difficulty to stay at the level of named entities – a very thin layer between lexical analysis and semantics. Its revealed the extent to which entity annotation is far from a “natural” operation for us – historians who have been trained to read historical texts differently – hermeneutics, external/internal critics, etc. We nevertheless agreed upon the following principles, i.e. the adjustments that had to be made to train the machine in order to better suit the historians’ needs – given below from the most consensual to the most debatable from the historians’ point of view, from the easiest to the most challenging for Pierre and from a technical point of view:
    • To refine the existing typology of institutions into: administration, company, education, military, organization. As for locations, the number of categories we decided upon reflects the level of granularity that we found the most appropriate or desirable for our analysis.
    • To correct the persons’ qualifications, we opted for the following categories: title (Mr., Miss, Dr., Prof….); degree (B.A., M.A., PhD…); occupation (financier, lawyer, journalist…); position (minister of finance) (the former can stand alone, whereas the latter implies a relation to an institution).
    • To add new entities (meta-categories) in order to translate historical conventions and to deal with issues of periodization: event (Boxer uprising, Northern expedition, Sino-Japanese war…) and regime (Qing dynasty, First Republic, Nanjing Decade). The former refers to a point in time, whereas the latter defines a longer period or a thicker slice of time. We historians argued a lot with Pierre over the concept of event, which has a different meaning for linguists. As neither historians nor linguists are ready to renounce their “natural” language, we may have to pick up a different term (more neutral/consensual) in order to avoid any possible confusion. But which one?
  • Last, what to do with such vague terms as movement, circles, and so on.. and terms that pertains to ideology (revolutionary, anti-Japanese, Confucianist, and even vegetarian!)? According to Pierre, these are non-entities that we could fruitfully use for machine learning.
To conclude, while we can reuse a good deal of existing entity corpora, there is still a lot to be done so as to adjust to historical sources and the historians’ needs, particularly if we want the machine to properly work for us and discover things we did not expect at first sight.
To be continued

Cécile Armand

Ancienne élève de l'Ecole Normale Supérieure (ENS) de Lyon (2006), agrégée d'histoire (2009), docteure en histoire (2017), postdoctorante à Stanford University (2017-8) puis Aix-Marseille Université dans le cadre du projet ERC “Elites, Networks and Power in modern China” (2018-). Ma thèse soutenue à l’ENS Lyon en juin 2017 proposait une histoire spatiale de la publicité à Shanghai (1905-1949). Mes recherches actuelles portent sur l’émergence des experts, “l’influence” américaine, la naissance de la presse et d’une “opinion publique” en Chine moderne. Dans une démarche d’histoire sociale, mes centres d'intérêt englobent aussi l'histoire spatiale, les outils et méthodes numériques et l'épistémologie des sciences sociales en général.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search