Data integrity

Where the data comes from

The evidence is committed, because dictionary data isn't in the repo's tests and etymology isn't in a dictionary at all. CI checks every entry against it, offline. Word-list commits are pinned in scripts/word-list-source.mjs, and the JMdict and KANJIDIC2 files in scripts/lib/jmdict.mjs (EDRDG overwrites both every day, so a pin is the only way to rebuild the same data, and the files are kept beside the code, git-ignored), so no builder can fetch different upstream data for the same level. The Wiktionary dump's date and checksum are pinned in scripts/lib/wiktionary-dump.mjs; the dump itself is too large to commit, so only the sections the entries quote are. A pool word with no Etymology section for its reading is still an entry, but it quotes nothing and is taken apart into nothing (Daily words). The Tatoeba export is pinned the same way, by size and checksum in scripts/lib/tatoeba-export.mjs, and is too large to commit, so only the sentences the entries show are. The accent list, the UniDic archive that checks it and the KanjiVG release are pinned by size and checksum in scripts/build-pitch-reference.mjs, scripts/lib/unidic-accent.mjs and scripts/build-strokes-reference.mjs, and only the accents and strokes the entries use are committed. The KRADFILE archive is pinned by size and checksum in scripts/build-components-reference.mjs and read the same way (pnpm data:components).

Data is processed offline once the pinned files are in place. The reference builders read the pinned JMdict and KANJIDIC2 files from the repo root (downloading a missing one, checksum-verified) and fetch the word lists; pnpm data:pitch and pnpm data:strokes download their pinned files the first time and cache them (the UniDic archive is large, and only its dictionary file is kept; Adding and fixing words says how much); pnpm data:sentences reuses the same JMdict and KANJIDIC2 files to check furigana, and every other step, from pnpm data:etymology to pnpm data:kanji, pnpm data:words and the tests, reads only local files. The running site reads none of these sources: the data is committed and served by the app.

How an entry is built

scripts/lib/word-entry.mjs builds an entry from sources only (scripts/generate-word-entries.mjs runs it, and fetches nothing). Each step refuses rather than guess:

• pickSections picks the Wiktionary section whose declared reading covers the word's.
• evidenceLines yields one verbatim quote per line, minus Wikipedia furniture.
• parseMorphemes reads A (a, “gloss”) + B (b, “gloss”) chains and keeps them only if the parts spell the word and join to its reading (rendaku allowed). It also accepts a stem glossed through its base verb (缶詰), a dropped okurigana (出る → 出口) and bare kana parts (お), and picks nothing when two different splits fit.
• parseLiteral pairs a “warm + spring” style gloss with KANJIDIC2's unambiguous per-kanji readings; parseLoan handles loanwords.
• parseKanji is the last resort for a word written with kanji (and the okurigana after them) that the text doesn't split: one part per kanji, with the okurigana that follow it (手続き is 手 + 続き, 言い出す is 言い + 出す), KANJIDIC2's reading and a meaning the text or the JMdict meaning already uses, marked glossSource: "kanjidic2". It is refused for ateji, jukujikun, an ambiguous split, kana KANJIDIC2 does not list as okurigana (何でも), and a single kanji with its okurigana, which would only repeat the word.
• processesOf finds keyword tags in the quoted text and adds what JMdict, a source independent of Wiktionary, records for the matched sense: a coinage made in Japan, an ateji or jukujikun spelling, a loan source (jmdictFactsOf). stratumOf analyses KANJIDIC2's on and kun readings, and takes a JMdict loan source as the loanword layer unless the cited text names Chinese.

What each field comes from

Where AI may help, and where it may not

A model's output can be wrong in ways that read as confident, so AI is only given tasks whose output is allowed to be inaccurate: nothing it produces is shown to a reader as a fact unless a source backs it.

• OK: choosing which words go in a month (a poor pick costs nothing; the generator refuses a word with no usable source), drafting a headline (a hook, not a claim; a test limits its Japanese to what the evidence or pool contains), and writing code or tests, which CI then checks.
• Not OK: supplying a reading, meaning, level, layer, morpheme split, etymology or any other claim about a word.

Before handing a task to a model, ask: if this came back wrong, would a check catch it or would it simply not matter? If neither, it must come from a source. The same goes for any new feature's data.

What the checks prove

The content tests (test/content/, their own Vitest project, run by pnpm test in CI) regenerate every month and check each entry independently of the generator, so a generator bug can't vouch for itself.

• Generation (word-generation.test.ts): a committed entry equals, field for field, what the generator derives, so a derived field can't be edited by hand or left stale.
• Catalogue: every date is unique and so is every word (a spelling with its reading), no day between the first entry and the last is missing (the first and last months may be partial), and entries are sorted.
• Every entry (words.test.ts):
  • the term, reading, level and meaning equal the word list's, and its layer and processes are ones the site knows;
  • pos holds only tags JMdict gives that word, and register, field and dialect only tags every matched sense carries;
  • every sources[].quote is found verbatim (modulo whitespace and direction marks) in a Wiktionary Etymology section declared for the entry's own reading, from the dump the entry names (a page like 大人 has one section per reading);
  • every otherReadings quote is found verbatim in a section of the same page that declares other readings than the entry's, and an entry with no sources has none;
  • an entry with no sources has no headline, no parts and no dump date, and the dump has no section for its reading (or only one that opens “From Middle Chinese” for a native word);
  • the morphemes literally spell the word and join to kana; each non-irregular single-kanji morpheme has a reading KANJIDIC2 lists and a gloss KANJIDIC2 or the cited text backs;
  • the headline mentions only Japanese the entry's evidence or the pool contains;
  • an unclear entry quotes a hedged line, no snapshot pin is orphaned, each pin sits in the shard of the month that plans its word, and the snapshot records its source and licence.
• JMdict, independently (jmdict-crosscheck.test.ts): an entry's priority codes are ones JMdict gives its spelling or reading; its register, field and dialect are exactly the tags every matched sense carries (and the register tags still occur in the pinned JMdict, so a renamed tag cannot silently stop matching); a word written with kanji that JMdict marks as a coinage made in Japan or an ateji, gikun or jukujikun spelling or reading carries that process; a European loan source means the borrowing process and the loanword layer (unless the cited text itself puts Chinese forward and hedges, as for 缶). It only confirms: the JMdict file pinned here gives no loan source for many loanwords (カメラ, ノート), so silence from JMdict proves nothing.
• Cross-references (also in jmdict-crosscheck.test.ts): each of an entry's antonyms and seeAlso is an ant or xref of a sense its meaning matches, and every one JMdict gives (but the word itself) is there.
• Example sentences (examples.test.ts): each sentence is the committed snapshot's, holds the form it says the word takes, is a form of that word (its own spelling, an inflection that keeps the first character, or kana), has a translation and a readable length, and the snapshot names the export it came from. The sentences are not reviewed one by one: the export is the community's, and a short filter keeps out the plainly unsuitable; of a word's qualifying sentences, those with every kanji read come first. Each sentence's furigana joins back to its text, puts a reading only over kanji, in hiragana, and never across the edge of the word's form; almost every kanji of the sentences shown has one; the snapshot says how readings were checked.
• Pitch accent (pitch.test.ts): the snapshot names its source, licence and the file it was read from, holds only words in the catalogue with their own reading, gives each accent as a whole number no larger than the word's count of morae, and every entry shows the snapshot's accents and no others. The second source is named too, and every word that lost an accent is listed with what each source said, adding up to what was kept.
• Kanji (kanji.test.ts): data/reference/kanji.json holds exactly the kanji the entries are written with, each as the level snapshots have it.
• Shapes inside a kanji (components.test.ts): the snapshot names its source, licence and the archive it was read from, holds only kanji of kanji.json, each with at least one shape, and covers nearly every kanji the entries use.
• Stroke order (strokes.test.ts): the snapshot names its source, licence and release, holds only kanji of kanji.json, each with as many strokes as KANJIDIC2 counts, and covers nearly every kanji the entries use.
• Every word-list word, at every level (vocabulary.test.ts in test/content/ and test/content/{n4,n3,n2,n1}/): it is in JMdict with that reading, has no meaning that reverses JMdict's, has its reading attested (reading-attested.test.ts), and every enrichment in shared/meanings.ts is backed by JMdict. A staleness check fails if a reference file no longer matches what servedVocab() would produce.

To change a word or add a month, see Adding and fixing words.

Attribution

Written once in shared/sources.ts; the footer, each entry's citation and the README read it from there.