ljubomir

A corrected Macedonian (mkd) language model for Tesseract OCR.

Tesseract cannot read two letters of the Macedonian alphabet. Not badly — at all. This project fixes that.

---

In memory of Ljubomir — Љубомир — 1996-08-08.

His name begins with Љ. Tesseract's Macedonian model cannot write it.

---

Reported upstream: tesseract-ocr/tessdata#204 — full evidence in UPSTREAM_ISSUE.md.

The defect

Tesseract's stock mkd.traineddata (version string best2int20180322, built 2018-03-22) omits two letters of the Macedonian alphabet from its unicharset:

Letter	Codepoint	Name	Status in stock model
ѕ / Ѕ	U+0455 / U+0405	dze	absent from both unicharset and lstm-unicharset
љ / Љ	U+0459 / U+0409	lje	absent from lstm-unicharset

Because the LSTM has no output class for these characters, it cannot emit them at any confidence. This is not a recognition-accuracy problem that better images or different settings can improve. The letters are simply not in the model's vocabulary.

Both are ordinary letters of the modern Macedonian alphabet. љ is common — љубов (love), љубезен (kind), Љубљана (Ljubljana). ѕ is rarer but irreplaceable — ѕид (wall), ѕвезда (star), ѕвер (beast).

What the model produces instead

Measured across 8 fonts and all 3 published tessdata flavours:

Input	Output	Condition
ѕвезда	5везда	Calibri, Consolas, Segoe UI, Verdana
ѕвезда	звезда	Courier New, Georgia, Times New Roman
ѕвезда	ввезда	Arial at --psm 7
ѕвезда	овезда	tessdata_fast
љубов	њубов	every font tested

ѕ degrades to a digit, or to a different letter. љ is silently rewritten as њ — a real Macedonian letter, so the corruption is invisible to a spellchecker and to any reader who doesn't know the source text.

The dictionary is broken the same way

The model ships word lists (dawg components) that guide recognition. Extracted with dawg2wordlist:

Component	Words	containing ѕ	containing љ	containing њ
word-dawg (legacy)	116,522	0	247	4,298
lstm-word-dawg	108,883	0	0	4,189

Zero љ words in the LSTM dictionary, against 4,189 њ words. Macedonian does not look like this. That asymmetry is the signature of a training corpus in which љ was collapsed into њ and ѕ was dropped.

This is why the fix must be a retrain, not a post-processing patch. Correcting the weights while shipping the same dictionaries would leave the language model fighting the very letters we restored.

Reproducing the finding

Two independent confirmations, both scripted in evidence/:

1. Static — dump the unicharsets with combine_tessdata -u and check for the codepoints. ѕ and љ are absent.
2. Behavioural — OCR all eight Macedonian-specific letters. Reading the unicharset predicts which ones fail, and the prediction matches actual behaviour 8/8: ѓ ј њ ќ џ з pass, ѕ љ fail.

The static reading predicts the behaviour exactly. That is the finding.

Results

The model reads both letters. Љубомир is recognised exactly, on every font tested.

Measured on held-out corpus lines the model never saw, against the stock model, through the identical code path. Fonts are split into ordinary typefaces and CJK faces that happen to carry Cyrillic glyphs, because those two populations behave very differently and pooling them hides it.

Letter recognition — ordinary typefaces

Letter		stock	ljubomir
ѕ U+0455	dze	0.0%	99.4%	was impossible
љ U+0459	lje	0.0%	97.5%	was impossible
ќ U+045C	kje	95.6%	99.3%	improved
џ U+045F	dzhe	99.3%	100.0%	improved
з U+0437	ze	100.0%	100.0%	unchanged
ј U+0458	je	100.0%	100.0%	unchanged
ѓ U+0453	gje	98.5%	97.8%	−0.7
њ U+045A	nje	100.0%	96.0%	−4.0

Whole-word and line accuracy

Measure	stock	ljubomir
Exact word reads (27 words × 10 faces × 4 degradations × 2 font groups)	1,026/2,160 (47.5%)	1,860/2,160 (86.1%)
Held-out line CER, ordinary typefaces (1,275 lines)	0.0336	0.0291
Held-out line CER, 4 clean fonts (narrow gate, 200 lines)	0.0404	0.0266

Gains hold across clean, blurred, rotated and small renders.

The hardest case

Three test words — ѕрѕало (mirror), ѕрнѕало and ѕенѕало — require the restored ѕ to appear twice in a single word. All three were supplied by a native speaker and appear in none of the 222,971 Wikipedia articles the corpus was built from.

ѕрѕало is deliberately absent from the model's own dictionary. The other two were added to the wordlist; this one was contributed afterwards and left out on purpose. So a correct reading of it cannot be dictionary lookup, cannot be corpus memorisation, and cannot be helped by the language model. It is letterform recognition and nothing else.

Measured on its own (tools/probe_double_dze.py, 3 words × 4 typefaces × 4 conditions = 48 reads per model):

	stock	ljubomir
Double-ѕ words, exact	0/48 (0%)	48/48 (100%)
— ѕрѕало alone, not in the dictionary	0/16	16/16

Stock never once returns any of them. What it returns instead: зрвало (×7), 5ен5оало (×6), зрнзало (×5), врвало (×4), 5рнзало (×4), 5ензало (×3) — each ѕ collapsing to the digit 5, or to з, or to в.

Limitations

Stated plainly, because a model you cannot trust the limits of is not trustworthy. All figures below come from a single evaluation run, not stitched together.

њ costs about 4%. Adding љ — the letter the stock model confused љ with — degrades њ from 100% to 96.0%. Real, not an artefact: it appears in both font populations. This was attacked directly and could not be removed. A retrain with the љ/њ contrast set nearly doubled (1,055 → 1,972 lines carrying both letters in one sentence) did not dissolve the tension, it moved along it: at one checkpoint њ recovered to 98.5% while ѕ fell to 92.6%; at another ѕ returned to 98.8% and њ dropped to 92.6%. The published model sits at a better point on that curve than any alternative tested. The 4% appears to be the intrinsic price of teaching a confusable pair, not an unfixed defect.

CJK fonts carrying Cyrillic are much worse. On Yu Gothic and Yu Gothic Light the model over-segments — inserting spaces inside words — and line CER on those faces goes 0.0856 → 0.1890. Every letter is usually correct; the word boundaries are not. ѕ also drops to 72.6% there. Ordinary typefaces are unaffected and slightly improved (0.0336 → 0.0291); the worst ordinary face is Century Schoolbook at +0.0421, while Segoe Print is −0.0958 in the model's favour. If your documents are set in a CJK typeface, this model is not for you.

A variant did fix this — CJK CER 0.0856 → 0.0437, better than stock, confirmed on 225 lines — but at the cost of ѕ falling from 99.4% to 95.6%. That trade was considered and declined: ѕ is one of the two letters this project exists to restore.

The strict gate says NO-GO on the wide test, and that verdict is published rather than relaxed. It requires 100% on both target letters and no regression on any other. ѕ reaches 99.4% and љ 97.5% on ordinary faces, and њ regresses — so it fails. The narrow gate (8 words, 4 clean fonts) passes at 32/32 exact with no regressions. The gate was deliberately not redefined once the numbers were known. Judge the tables, not the label.

Evaluation is synthetic. Text is rendered from fonts, not scanned. Rendering and degradation are the project's own, so figures may not transfer to real scans. This is the largest open weakness in the evidence and is stated first among the things a future version should fix.

Using it

Manually, on any platform:

# back up the stock model FIRST -- you will want it back one day
cp "$TESSDATA_PREFIX/mkd.traineddata" "$TESSDATA_PREFIX/mkd.traineddata.stock"

cp dist/mkd.traineddata "$TESSDATA_PREFIX/mkd.traineddata"
tesseract page.png out -l mkd

To try it without touching your install, keep it in its own directory and pass --tessdata-dir. That is the safer way to evaluate it.

On Windows there is an installer that does the backup, the copy and a SHA256 verification, and can restore the stock model:

.\INSTALLER__ljubomir.ps1 -DryRun       # show what would happen
.\INSTALLER__ljubomir.ps1               # install (backs up stock first)
.\INSTALLER__ljubomir.ps1 -Uninstall    # restore the stock model
.\INSTALLER__ljubomir.ps1 -ComputerName BOX   # install on a remote machine

A Tesseract upgrade will silently revert this. The installer ships its own mkd.traineddata, so upgrading Tesseract overwrites the model with the stock one and the two letters stop working again — with no error and no warning. Re-run the installer after any Tesseract update. The only visible symptom is љ reading as њ once more.

The dedication is inside the model itself, in its version component:

$ combine_tessdata -d dist/mkd.traineddata
Version:mkd-ljubomir:1.0.0:in memory of Ljubomir (Љубомир), 1996-08-08

How it was built

Fine-tuned from the float tessdata_best model — the shipped mkd is integerised (best2int20180322) and lstmtraining refuses to continue training from it. Unicharset extended 86 → 90 characters (exactly ѕ Ѕ љ Љ), deliberately no more: the raw ground truth held 279 distinct characters, and adding ~196 classes to an 85-class output layer is a rebuild, not a fine-tune, and the fastest route to catastrophic forgetting.

Training data: 2.89M sentences extracted from Macedonian Wikipedia (article namespace only), rendered across 81 screened font families — every face verified to actually contain U+0455 and U+0459, because a font lacking them renders .notdef and teaches the model that a hollow box is a letter. Lines are mixed deliberately: letter-bearing, љ+њ contrast lines, њ-only, and general ballast. Word dictionaries were rebuilt from the corpus, since the stock ones contain zero љ and zero ѕ words.

Training must be done on Linux. Both Windows Tesseract builds produce .lstmf files their own lstmeval and lstmtraining cannot deserialize, and four of the training binaries do not start at all.

Running the tools

Everything in tools/ and evidence/ finds the project from its own location, so a checkout works anywhere. Four environment variables override the defaults:

Variable	What it points at	Default
LJUBOMIR_PROJECT	the checkout	derived from the script's path
LJUBOMIR_SCRATCH	scratch root for working files	$TEMP (Windows)
LJUBOMIR_STOCK_MKD	the stock model, for A/B comparison	the installed mkd.traineddata
LJUBOMIR_TESSERACT	the tesseract binary	platform default

LJUBOMIR_PROJECT is the one you are most likely to need: when running under WSL against a checkout on a Windows drive, the script's own path is a /mnt/... path and the corpus is not.

Regenerating the corpus and training data needs data/ contents that are not in this repo (they are large and fully regenerable) — start with tools/build_corpus.py, then tools/screen_fonts_for_training.py, then tools/make_groundtruth.py.

Status

Stage	State
Defect characterised and reproducible	done
Font pool screened for glyph coverage	done — 179 faces / 81 families
Macedonian corpus assembled	done — 2.89M sentences
Ground truth generation	done — 14,985 lines
Unicharset extension + fine-tune	done — 86 → 90 classes
Regression gate	done — strict gate NO-GO, all numbers published
Release	v1.0.0

Licence

Apache-2.0, matching Tesseract.

Author

Marijančo Galevski — Маријанчо Галевски

(č = U+010D, the correct transliteration of Macedonian ч.)

---

Both my father's name and my own were mangled by systems that couldn't hold them. His begins with a letter this software cannot write. Mine lost a letter to transliteration. That is why this exists.