Adding and fixing words

Where things live

The generated evidence under data/reference/ is listed, with its sources, in Data integrity. The files you write, correct or regenerate are these:

Commands

data:etymology is offline and takes seconds: it reads the Wiktionary dump from Kaikki.org, a 385 MB file that is never committed. Put it in the repo root (it is git-ignored) or point --dump at it; its name, date and checksum are pinned in scripts/lib/wiktionary-dump.mjs and any other file is refused. The reference builders read JMdict and KANJIDIC2 from two files in the repo root, JMdict_e.gz and kanjidic2.xml.gz (git-ignored). EDRDG overwrites both every day, so their size and checksum are pinned in scripts/lib/jmdict.mjs: a missing file is downloaded from the address there, any other copy is refused, and you should keep the two files if you need to rebuild. The word lists are fetched from GitHub.

data:sentences is offline too: it also reads the two pinned EDRDG files the reference builders share, to check the furigana against JMdict and KANJIDIC2, and runs kuromoji and UniDic from node_modules. It reads five files of a Tatoeba export, about a minute's work on about 175 MB that is never committed. Download them from the paths listed in scripts/lib/tatoeba-export.mjs (each is a bz2; the indices come in a tar), extract them into a tatoeba directory in the repo root (it is git-ignored), and the script verifies their size and checksum before it reads anything. Tatoeba overwrites its exports every week, so keep the files if you need to rebuild: the committed snapshot, not the export, is what CI checks. It reads the word plans, so it runs before data:words, which copies the sentences into each entry; data:kanji runs after it, because it reads the entries' spellings.

data:pitch and data:strokes download their pinned files the first time (Kanjium's accent list at a fixed commit, UniDic's release, whose compiled dictionary checks the accents, and KanjiVG's release; size and checksum are in scripts/build-pitch-reference.mjs, scripts/lib/unidic-accent.mjs and scripts/build-strokes-reference.mjs), cache them under node_modules/.cache/, and are offline after that. The UniDic archive is about half a gigabyte; only its dictionary file is kept. Like data:sentences, data:pitch reads the word plans and the level snapshots, so it runs before data:words; data:strokes reads data/reference/kanji.json, so it runs after data:kanji. Neither guesses: a word the accent list does not give for that exact spelling and reading shows no pitch, an accent UniDic does not give for it too is not shown (UniDic is silent on some words, and the snapshot's heldBack names each one with what the two said), and a kanji whose stroke count KanjiVG and KANJIDIC2 disagree on shows no strokes. data:components downloads and caches its pinned archive (EDRDG's KRADFILE, pinned in scripts/build-components-reference.mjs) the same way, reads data/reference/kanji.json, and so also runs after data:kanji; a kanji the file has no line for shows no shapes.

Adding a month

1. Choose the words (a person or a model may do this). They must be pool words (N5–N1) that no plan holds yet. Every pool word is planned, spelling and reading, so a further month means a new word list.
2. Write the plandata/word-plan/YYYY-MM.json: one { "date", "term" } per day, every day of the month. Add "kana" only when a spelling has several pool words (明日, 梅雨). Add a "headline" only to a word that has an Etymology section: one sentence that earns the click and keeps to what the quoted evidence says, which may be the entry's own section or a section of another reading that the entry shows beside it. A word with a headline promises an origin, so the generator refuses it if the dump has no section for its reading; a word without one is built with whatever the sources hold. A new month continues directly after the last day, with no gap; any day already past opens at once.
3. Pin the evidence: pnpm data:etymology, and pnpm data:sentences for the example sentences and pnpm data:pitch for the pitch accent (a word with none of either is fine). Every term in the plans is included; add --terms 電話,友達,… for words with no plan yet. Most pool words have no usable Etymology section, so pin a batch with --skip-missing: a word without one is skipped and its entry says the origin is missing.
4. Generate: pnpm data:words, then pnpm data:kanji, pnpm data:strokes and pnpm data:components for any kanji the new words add. It prints every entry it could not build and why, and writes nothing until they are fixed (or use --keep-going).
5. Read the result once. Entries with no breakdown are normal (the source gave no clean split, or there is no source), as are entries with no layer (irregular spellings). Skim each headline against the quoted lines: it must not claim more than they do.
6. Register the month in shared/words.ts (one import line plus the MONTHS array). Keep shared/words.ts out of app/ (the import rule).
7. Rebuild the share-image fonts if the new words use a kanji they lack: pnpm assets:og-font. The test fails and names the characters until you do.
8. pnpm test:run. The range and count the docs quote are read live from data/words/, so there is nothing to update.

If the generator refuses an entry

Correcting a word's form, reading or meaning

1. Confirm against the JMdict evidence in data/reference/<level>-reference.json.
2. Add to VOCAB_FORM_CORRECTIONS (wrong form or reading) or VOCAB_MEANING_ENRICHMENTS (fuller gloss) in shared/meanings.ts, keyed by term kana, with a reason citing the JMdict entry id. The word's id stays unchanged.
3. Rebuild: pnpm data:reference (N5) or pnpm data:reference:jlpt (N4–N1), then pnpm data:words, and commit the diffs.

Refreshing the sources

1. JMdict, KANJIDIC2 and word lists: bump WORD_LIST_SOURCES and/or JMDICT_SOURCE (put the new files in the repo root and record their date, size and checksum), then run pnpm data:reference and pnpm data:reference:jlpt together, then pnpm data:sentences, pnpm data:words and pnpm data:kanji, and review every diff. A newer JMdict can reword a tag or a gloss, and then a test or a correction in shared/meanings.ts says so.
2. Wiktionary: point WIKTIONARY_DUMP in scripts/lib/wiktionary-dump.mjs at the new file (date, size, checksum), run pnpm data:etymology, review the text diff, then run pnpm data:words: a changed etymology changes the entry, and the diff shows it.
3. Tatoeba: download the five files named in scripts/lib/tatoeba-export.mjs, extract them into a tatoeba directory, and put their date, sizes and checksums in TATOEBA_EXPORT. Run pnpm data:sentences, review the diff (sentences change, and some words gain or lose one), then pnpm data:words.
4. Kanjium, UniDic and KanjiVG: change the commit or release, size and checksum in PITCH_SOURCE, UNIDIC_ACCENT_SOURCE or STROKES_SOURCE, run pnpm data:pitch and pnpm data:words, or pnpm data:strokes, and review the diff. For KRADFILE, change the size and checksum in COMPONENTS_SOURCE and run pnpm data:components.
5. Dropped a word? pnpm data:etymology --prune removes pins nothing uses, and pnpm data:sentences, pnpm data:pitch, pnpm data:kanji, pnpm data:strokes and pnpm data:components drop what the word alone used.
6. pnpm test:run: new gaps show up as failing entries.

When a check fails

What the checks cover, and what they cannot, is in Data integrity.