A historian’s view of NER (named entity recognition)

Following up on Pierre’s inspiring post, I wish to start a dialogue between historians and computing specialists around the tools we share in the ERC project. This series of methodological posts also answers our common concern with keeping track of the research process, making transparent every decisions we make and the procedures we follow at every stage of the project – particularly for our remote colleagues in Taiwan, Bristol or Paris.

When I first heard of machine learning about three years ago, it sounded like science fiction. That we could apply that “sectarian doctrine” to historical research went beyond my imagination. As a PhD candidate struggling to complete her dissertation, I preferred to stick to more dependable and established tools at the time – Geographical Information System (GIS) or relational databases. This post today records my first steps into machine learning. What happened in the meantime? How did I convert to the devil?

Our project makes the hypothesis that machine learning (ML) and natural language processing (NLP) tools offer promising ways to extract information from historical sources at an unprecedented scale. Yet most of the tools available off the shelf today have not been designed for historical purposes. Entity corpora for instance have been built from contemporary texts with contemporary concerns. In order to be able to extract relevant information from our historical sources, we first need to train the machine to recognize entities that might interest us as historians. Put another way, we need to build a corpus of entities that are suited for historical purposes in general – and for our specific research questions as historians of elites in modern China.

For this purpose, we — Pierre (our NLP specialist), Christian and I (both historians) — decided to conduct a small experiment with named entity recognition (NER). We started from Pierre’s preliminary experiment with the Boorman dictionary. Although the entity corpus applied by Pierre on the Boorman was built from contemporary sources of a different nature (1980s American newspapers) with specific concerns (security, terrorism, criminality), we found that most entities were historically relevant, at least at the most general level (date, location, person, organization).

The question then was to determine precisely to what extent the existing corpus matched our own concerns, how we could correct possible mismatches and fill the gaps. We discussed with Pierre the best method to proceed. We had two options:

• Either to correct/reannotate the text samples already processed by Pierre using Standford CoreNLP
• Either to start from scratch and annotate new samples of text – blank from any previous annotations

Pierre recommended that we choose the second solution because it would allow us to start from our own needs instead of being driven by a priori categories.

Interface of the biographical database developed at the Institute of Modern history, Taipei, Taiwan

For Christian and I, the first step was to select relevant people and text samples from the directories made available through “The Modern and Contemporary Person Integrated Information System” project developed at the Institute of Modern history, Taipei, Taiwan. We started from the Boorman’s index and selected a sample of five representative actors from different backgrounds (politics, business, intellectual, and a woman) in Powell’s directories, spanning from 1918 to 1940:

• Chang Chia-ao 張公權  [Zhang Jia’ao] (1918, 1931, 1936) (business)
• Soong T. V. 宋子文 [Song Ziwen] (1928, 1936) (business/politician)
• Wu T’ing-fang 伍廷芳 [Wu Tingfang] (1918) (politician)
• Feng Yü-hsiang 馮御吞 ( 馮玉祥) [Feng Yuxiang] (1920, 1931, 1940 (military)
• Liang Ch’i-ch’ao 梁啟超 [Liang Qichao] 1918, 1931 (intellectual)
• Cheng Yü-hsiu 鄭毓秀 [Zheng Yuxiu] (1928, 1931) (woman)

We decided to  annotate every notice separately — so as not to influence each other — and to compare our respective findings. We annotated the texts using the existing basic categories that Pierre slightly customized to adjust our first reactions to the Boorman’s experiment: person, location (subdivided into country, province, prefecture, district, city); institution (company, organization), miscellaneous, non entities (this served to annotate interesting words that were not entities stricto sensu but seemed important for further research on concepts).

Preliminary Results

• • To refine the existing typology of institutions into: administration, company, education, military, organization. As for locations, the number of categories we decided upon reflects the level of granularity that we found the most appropriate or desirable for our analysis.

• • To correct the persons’ qualifications, we opted for the following categories: title (Mr., Miss, Dr., Prof….); degree (B.A., M.A., PhD…); occupation (financier, lawyer, journalist…); position (minister of finance) (the former can stand alone, whereas the latter implies a relation to an institution).

• • To add new entities (meta-categories) in order to translate historical conventions and to deal with issues of periodization: event (Boxer uprising, Northern expedition, Sino-Japanese war…) and regime (Qing dynasty, First Republic, Nanjing Decade). The former refers to a point in time, whereas the latter defines a longer period or a thicker slice of time. We historians argued a lot with Pierre over the concept of event, which has a different meaning for linguists. As neither historians nor linguists are ready to renounce their “natural” language, we may have to pick up a different term (more neutral/consensual) in order to avoid any possible confusion. But which one?

• Last, what to do with such vague terms as movement, circles, and so on.. and terms that pertains to ideology (revolutionary, anti-Japanese, Confucianist, and even vegetarian!)? According to Pierre, these are non-entities that we could fruitfully use for machine learning.