Enabling Quantitative Analysis of a Noisy Historical Hebrew Corpus: The Computational Pipeline Behind ~38,000 Israeli Collective Agreements (1957–2022)
תקציר
Quantitative, longitudinal analysis of employment relations requires turning a large archive of historical documents into clean, queryable data. We present the computational pipeline behind nearly 38,000 Israeli collective bargaining agreements (1957–2022) — a corpus too large to read by hand, yet whose difficulty is text quality, the morphological richness of Hebrew, and accessibility for non-programmer researchers, not volume. The pipeline has four stages: (1) OCR of degraded scans, selecting Google Vision after benchmarking against ground truth; (2) Hebrew-specific cleaning (tokenization, stopwords, root reduction, bigrams); (3) FastText sub-word embeddings robust to OCR noise and morphology, enabling semi-automatic discovery of term variations that experts validate; and (4) an interactive, Elasticsearch-backed tool running supervised dictionary-based and unsupervised BERTopic analyses. Using it, we trace long-term shifts in the contractual language of Israeli labor relations across sectors and decades. The approach generalizes to other noisy, morphologically-rich historical corpora.