diff --git a/AGENTS.md b/AGENTS.md index 64927c10..55bd1486 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -410,7 +410,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`), there being no trailing shape rule. The bare word is where "positional" still holds whole, and the reason it must: `TITLES` holds words that are in no suffix set — 746 of them, measured on this tree with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (decisions.md#P5's trailing-position bullet, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules.md#S3`). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole, and the reason it must: `TITLES` holds words that are in no suffix set — 746 of them, measured on this tree with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (decisions.md#P5's trailing-position bullet, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/customize.rst b/docs/customize.rst index 68ec3383..13ce608a 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -202,13 +202,27 @@ they narrow how an existing entry is read when it appears alone. If you're not sure whether a word you're adding is one of these ambiguous cases, weigh how often it is a name against how often it is the credential. Marking it ambiguous is -not free in either direction: an ambiguous acronym counts as a suffix -only when written with its periods, so the bare spelling reads as a -name and the parse reports the fork, and the comma form moves with it -— ``Smith, BA`` reads first ``BA`` rather than suffix ``BA``, and a -bracketed ``John Smith (BA)`` falls through to nickname parsing. A -wrong unambiguous claim takes the credential reading silently and can -lose a real person's surname. For ``particles_ambiguous`` the default +not free in either direction: written with its periods, an ambiguous +acronym counts as a suffix unambiguously; bare, the reading now +depends on the writing itself, so a bracketed ``John Smith (BA)`` +falls through to nickname parsing and either bare reading reports the +fork. Written in ALL CAPITALS inside a mixed-case name it counts as +a suffix even with no words to spare (the credential lean); written +Title-case there it stays the surname even WITH words to spare (the +surname lean); lacking either signal — an all-lower spelling in a +mixed-case name, or any spelling in a name written wholly in one +case — the reading falls back to whether the name has two or more +words before it. At a comma the count of NAME words before it decides +FIRST, and the case is read only where the count leaves the word a +name: ``John Smith, Ba`` reads suffix ``Ba`` on the count alone (two +name words before the comma), Title-case or not, while ``Smith, BA`` +reads suffix ``BA`` on the CAPITALS lean, one word before the comma +being all the count needs to leave for the lean to promote. What +still reads as the given name is ``Smith, Ba`` (one word, and +Title-case carries no credential lean to promote it) and +``smith, ba`` (one word, one case, no lean at all). A wrong +unambiguous claim takes the credential reading +silently and can lose a real person's surname. For ``particles_ambiguous`` the default runs the other way: a particle that is not borne as a given name belongs in the never-given half, which is where ``mc`` and ``ste`` were moved (#360). The other @@ -225,8 +239,10 @@ that swallowed it as a title would misparse "Dean Martin" for everyone.) ``ma`` is a shipped example. It is both a credential and a common -surname, so it is listed in ``suffix_acronyms_ambiguous`` and counts as -a suffix only when written with periods: +surname, so it is listed in ``suffix_acronyms_ambiguous``: written +with periods it counts as a suffix unambiguously, and bare it takes +the case reading above -- Title-case stays the surname, capitals lean +the credential: .. doctest:: @@ -234,6 +250,8 @@ a suffix only when written with periods: 'Ma' >>> parse("Jack M.A.").suffix 'M.A.' + >>> parse("Jack MA").suffix + 'MA' ``particles_ambiguous`` is the same idea for surname particles. A particle listed there may also be a given name, which is what makes a @@ -428,6 +446,39 @@ listed below. family ``田中さん``, given ``V.`` when ``False`` — though a comma around a CJK name is tolerated input (``rules.md#W3``) and this reading can change. + * - ``unlisted_dotted_suffixes`` + - ``bool`` + - Reads an unlisted token of two or more period-separated chunks + as a credential where the position allows it: + ``"John Smith X.Y.Z."`` gives suffix ``X.Y.Z.`` while + ``"Jack X.Y.Z."`` keeps family ``X.Y.Z.``, and either reading + is reported. Case is irrelevant — the periods are the signal. + Whole-token vocabulary still wins (``M.A.``, ``Ph.D.``), and a + single trailing period is not this shape + (``"John Smith Xyz."`` keeps family ``Xyz.``). Two further + gates keep it from over-reaching: every chunk must be + alphabetic, so a digit anywhere refuses it + (``"John Smith 1.4"`` keeps family ``1.4``, on or off), and a + script with no period abbreviations of its own refuses it too + (a CJK word glued into periods, ``"John Smith 田.中."``, keeps + family ``田.中.``). Defaults to ``True``; ``False`` reads such a + token as name material everywhere and still reports the fork — + it does NOT revive the pre-2.4 reading of a chunk that is a + single ASCII character — a roman numeral, or the digit ``2`` — + as a credential (``"Jack X.Y.I."`` still keeps family + ``X.Y.I.`` either way, and a dotted version string such as + ``"John Smith 1.4.2"`` keeps family ``1.4.2``; that retirement + is not behind this switch). + * - ``unlisted_caps_suffixes`` + - ``bool`` + - Reads an unlisted all-caps word of two or more letters, with no + period in it, in a name written in more than one case as a + credential where the position allows it: ``"John Smith XYZ"`` + gives suffix ``XYZ``. Defaults to ``False``, and deliberately: + an all-caps surname is a real writing convention that shape + cannot separate from a credential, so ``"Jean Pierre DUPONT"`` + gives family ``Pierre``, suffix ``DUPONT`` with this on. Off, + nothing changes and nothing is reported. * - ``strip_emoji`` - ``bool`` - Excludes emoji from tokenization — they appear in no field or diff --git a/docs/design/decisions.md b/docs/design/decisions.md index bd8393f9..1fcbd1e6 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -216,6 +216,7 @@ Declined: - 2026-07-17 (M12, Derek-approved) — ma and do joined the ambiguous acronym set because both are common surnames; the two-word "Jack Ma" is kept intact by S2's words-to-spare guard, while the periods gate governs the dotted spellings ("M.A." counts unambiguously). Documented side effect: parenthesized bare "(MA)"/"(DO)" no longer escape to suffix as in 1.x. +- 2026-09-15 #289 — AMENDS the sentence above for one spelling and leaves it standing for the other. "The two-word 'Jack Ma' is kept intact by S2's words-to-spare guard" is still true of `Jack Ma`, and false of `Jack MA`: an all-caps member of the ambiguous set inside a mixed-case name is now read as a credential with nothing to spare, so `Jack MA` gives given 'Jack', suffix 'MA' — v1's reading was last 'MA', so this is a deliberate parity break and not a restoration. The periods gate is untouched and still governs the dotted spellings. The two ledger comments that quote this sentence carry the same amendment. See #S2. ### vocabulary-collisions — when a word earns the ambiguous marking @@ -331,6 +332,7 @@ The reconciled v1-style banks (`tests/test_*.py`) carried eight `@pytest.mark.xf 2026-09-13 #528 — AMENDS the paragraph above: CLOSED THE SAME DAY. `_facade.py` was touched after all — #528 shipped the fix the paragraph predicted, with the R4 shape it predicted: the facade consults the PARSED token wherever the part maps to one and falls back to the vocabulary only where it does not. Both views give "j. e. s." and "J. G." now (measured 2026-09-13 after the fix); the contrastive test the paragraph above describes became the agreement test `tests/v2/test_render.py::test_facade_initials_follow_the_one_case_fork`. `decisions.md#R3`'s 2026-09-13 entry carries the fix, its accepted costs and the one corpus name where the two views still differ. `capitalize()` was never split: it followed the parse on both surfaces throughout the window the paragraph above describes, giving "John E Smith" for that whole window — not "all along", since 2.3.0, before #383/#479 landed, gave "John e Smith" for the same name. TWO OTHER CAUSES SHARE THE `_initials` FIELD IN THE LEDGERS AND NEITHER IS THIS PR'S, recorded because a reader meeting them under these names will reach for this entry. The Arabic `محمد و علي` diffs on `_initials` at 1.4.0 only (`م. و. ع.` → `م. ع.`), measured byte-identical either side of the fork: it is a pre-existing #269 consequence — a recognized non-Latin connective contributes no initial — surfaced by the row entering the contract corpus, and `expected_since_1.4.0.toml` ledgers it as `feat(#269)`. `JUAN Y GARCIA` diffs on `_initials` at 2.0.0 through 2.2.0 for a DIFFERENT reason than it does at 2.3.0: at those three baselines `fix(#462)` already admits the name (the facade moved `J. G.` → `J. Y. G.` there), and this PR's core move `J. Y. G.` → `J. G.` gets its own rule at 2.3.0 alone. Two causes, one field, documented in a dated paragraph on `fix(#462)` in those three ledgers rather than as a competing rule. Out of scope, each its own issue: #492 (whether a cased suffix token counts as case evidence — `is_one_case` is written so R5 and #492 can share it later, but render does not import it here); #478 (hyphenated connective repair — its spaced-form claim now depends on "y" staying OUT of the subset); #461 (R3's clause); #289 and #516 (the same case-class fact read at the suffix and post-comma slots); the render-side conjunction fallback for spliced raw text, which stays vocabulary-keyed as rules.md#R4 states. + - 2026-09-15 #289/#516 — AMENDS the out-of-scope list above: #289 and #516 are no longer out of scope, having shipped in 2.4 on the fact this entry's own fork computes. The fact is now a `ParseState` field rather than a classify local, taken over the SAME own-words span this entry defines, and `_pieces.own_words` is where that span lives for both readers. #492 stays out of scope and stays deferred. See #S2. - 2026-09-13 #383/#479 — `conjunctions_ambiguous` is deliberately NOT registered in `_SUBSET_FIELDS`, so an orphan marker entry raises nothing. An orphan there is INERT rather than harmful: the classify fork tests membership in `conjunctions` before it reads the subset, and the emitter tests it again, so a marker entry whose base word is gone is never consulted by either. AGENTS.md's invariants rule guards harm, not no-ops, and the precedent is this file's #given-name-titles Declined entry, where two attempts at a check each cost a working configuration to forbid a condition that costs nothing. What holds the SHIPPED constant to the subset relation is an import-time assert in `nameparser/config/conjunctions.py` (which also holds it to cased single letters, the only entries the fork can read), and the v1 shim's intersection, which is a provable no-op. So `remove(conjunctions={"e"})` simply works, leaving a stale marker entry behind that does nothing. Pinned by `test_removing_a_conjunction_leaves_its_ambiguous_marker_alone` in tests/v2/test_lexicon.py and by the orphan behavioral pin in tests/v2/pipeline/test_classify.py, both of which cite this entry. @@ -546,6 +548,56 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - **esq left SUFFIX_ACRONYMS (2026-09-08, the #316/#489 bundle).** The same criterion asked of a word the frequency argument cannot reach: does the entry describe the WORD or the machinery. Esquire is a contraction, not an initialism, so the initialism set was the wrong home for it whatever its frequency — this is the criterion's third kind of answer, and it is the one rai and cha did not need. The entry arrived with the same 2019-12-11 bulk import (af5bdab, #93) that brought rai and cha and was never reviewed either; its only unique coverage was the multi-dot spelling `E.S.Q.`, the SUFFIX_WORDS membership carrying every single-token spelling. Measured over the corpus glob: one name moves, `John Smith E.S.Q.`, from suffix to family, at every baseline. What the removal buys is the invariant this entry's neighbours could not have: `SUFFIX_ACRONYMS ∩ SUFFIX_WORDS == ∅` is asserted at import, esq having been the only member of that intersection and the reason the assert could not exist. Recompute the intersection with `L = Parser().lexicon; sorted(L.suffix_acronyms & L.suffix_words)`, which gives `[]` on 2026-09-08. The standing keep-out is in the Excluded block above. - **Not decided here:** sa stays where #296's audit put it (title and suffix dual, position decides); se and om stay unambiguous, there being no surname evidence worth standing behind and OM being the Order of Merit; the S2 comma-form boundary ("Smith, Ed" → given Ed) is not reopened. #342's own comment lists sa, se and om beside ba and cha as model recall rather than corpus-attested, and per the #360 lesson they need a human before they move. - **Measurement (2026-09-07).** Five corpus names move, every one from the radar-tier corpus_issues.jsonl, in three diff shapes with one cause: Aishwarya Rai moves {family, suffix}, Lala Lajpat Rai and John Smith RAI move {middle, family, suffix}, and John Smith, RAI and Ahmad Jayadi, CHA move {given, family, suffix}. Recompute by parsing every name in the tools/differential/corpus*.jsonl glob twice — once with the shipped lexicon, once with `Lexicon.default().add(suffix_acronyms={"rai","cha"})`, which RESTORES the two entries this bundle removed so the diff read forwards is the removal's — and diffing the seven name fields plus `ambiguities`; the ba step of the same sweep moves nothing. Aishwarya Rai is the one name whose diff DISAPPEARS at 1.4.0, that release having read family Rai too, which is why the `fix(#342) NOT WANTED` ledger rule was deleted rather than rewritten and why the 1.4.0 gate lists four names under the replacement where the three 2.x ledgers list five. Read today's intentional counts off the `corpus:` line of `uv run python tools/differential/compare.py --baseline X`; they rose by three at 1.4.0 and by five at each 2.x baseline. +- **2026-09-15 (#289/#516) — the parking lot is CLOSED, and this entry's "measured 2026-09-07, none of the four is read as a suffix today" sentence is superseded rather than rewritten.** The shape-plus-position heuristic this entry parked is what shipped: the DOTTED shape as `Policy.unlisted_dotted_suffixes`, default on, and the ALL-CAPS shape as `Policy.unlisted_caps_suffixes`, default off. Of the four names the bullet measured, `John Smith X.Y.Z.` and `john smith x.y.z.` now read suffix at the default, and `John Smith XYZ` / `John Smith, XYZ` read suffix only with the opt-in switch. See #S2 for the whole decision and for why the two halves have different defaults. +- **2026-09-15 (#289/#516) — the roman-numeral accident is RETIRED, and here is what narrowly means.** This entry's bullet recorded the accident as evidence rather than as behavior anyone relied on. It now fires no longer where every chunk the vocabulary matches is a single ASCII character — measured, `{i, v, 2}` — so `John Smith R.A.I.` and `John Smith X.Y.I.` reach the shape class instead and read by position, which gives the same fields for the first and 1.4.0's family reading for `Jack X.Y.I.`. `John Smith C.H.A.`, the bullet's other half, is unaffected: nothing claimed it before or after. The WIDE retirement was measured and rejected; #S2 records the three names that pay for it. + +### S2 — the case signal at the suffix slot + +- 2026-09-14 #289 + #516 (this PR) — A BARE AMBIGUOUS ACRONYM IS READ BY THE EVIDENCE THE WRITING CARRIES, and written case is evidence the words-to-spare count does not have. In a name written in more than one case, a member of `suffix_acronyms_ambiguous` written in CAPITALS leans credential and is taken as a suffix with nothing to spare (`Jack MA` → given 'Jack', suffix 'MA', as `Jack MD` has always read); one written in any other cased form that is not wholly lower leans SURNAME and stays a name word with plenty to spare (`John Smith Ma` → middle 'Smith', family 'Ma'). An all-lower member carries no lean, a name written wholly in one case carries none at all, and both take the positional rule unchanged — `JOHN SMITH MA` suffix, `ANH DO` family, `john smith ma` suffix. Switch-free and scoped to the listed set: `ba do ed jd ma`. + THE PREDICATE KEEPS THE JUDGED TOKEN IN THE SPAN, and that is what gives the right answers here. `JOHN SMITH Ma` is MIXED because of `Ma`, and a Title-case word against all-caps neighbours is exactly the surname signal this design wants; excluding the judged token would make the name one-case and hand `Ma` to the positional rule. Same for `Jack MA` read the other way — included, the name is mixed and the capitals lean credential; excluded, `own == ['Jack']` is one case and the peel keeps the family. The general question — whether a cased suffix token counts as case evidence — is #492's and stays deferred; this entry records that the INCLUDING reading is what this design rests on. +- 2026-09-14 (Derek) — THE LEAN REACHES THE POST-COMMA GIVEN SLOT: `Smith, MA` → family 'Smith', suffix 'MA', while `Smith, Ma` and `Smith, ma` stay given. Positive evidence outranks position there, so the credential reading fires with ONE word before the comma where the positional rule would not. + AND THE COUNT AT A COMMA IS OF NAME WORDS. For this class, "words to spare" in `X, Y` form means the part before the comma holds two or more NAME words: `Smith Jr., MA` has two tokens and one name word, and a token count hands its family to `given` (measured on the prototype). rules.md#C1 already said both things — "more than one word precedes the comma" in its opening clause and "a part before the comma with more than one name word" later — so the doc edit is a reconciliation inside C1 rather than a new claim. +- 2026-09-15 (spec review) — THE COUNT REACHES THE LISTED SET TOO — one uniform rule for the whole ambiguous class rather than two. `John Smith, Ed` → given John, family Smith, suffix Ed; `john smith, ma` and `JOHN SMITH, MA` move the same way, each being a one-case name that takes the positional rule, which now reaches the comma structure. All three are 1.4.0 RESTORATIONS — measured on the wheel, v1 reads all three as suffixes — so today's tree is what deviates and this restores parity rather than opening distance from it. Corpus population: ZERO (verified by grepping `tools/differential/corpus*.jsonl` for all three; `Royce, Ed` is the corpus's only member of the sibling one-word shape), so no corpus mover count moves and `Davis Royce, Ed` is the case row that carries the decision. +- 2026-09-17, corrected — A CASELESS SCRIPT IS INERT ONLY AS A LEAN, NOT AS THE NAME-WORD COUNT. `is_one_case` answers True trivially for text with no case to write a contrast in, so `ambiguous_lean` returns `None` for `毛泽东, MA`, `마틴 킹, MA` and `田中 太郎, MA` alike — the LEAN really is inert by construction, as first written. But the comma count above is of NAME words "whatever case the name is written in", which is orthogonal to case and reaches a caseless script the same as any other: `마틴 킹, MA` and `田中 太郎, MA` each hold TWO pre-comma name words, so the structure flips and both read suffix 'MA' — a PARITY RESTORATION, not a fresh deviation (1.4.0 read both as suffix on the wheel), with the Hangul/Han surname split then running positionally over what the comma leaves (`마틴 킹` → given '틴', middle '킹', family '마'; `田中 太郎` → given '太郎', family '田中'), unrelated to this design and #271/#272's pre-existing behavior. `毛泽东, MA` does not move — one pre-comma token, the count never reaches two. Pinned `a_caseless_script_wrote_no_contrast_hangul` and `a_caseless_script_wrote_no_contrast_japanese` (tests/v2/cases.py), both `tolerated=True`. +- 2026-09-17, corrected — THE COMMA-PATH REPORT TRACKS THE FORK CONSULTED, NOT THE LEAN. A first pass gated the family-comma report on whether `ambiguous_lean` fired, which wrongly silenced `Smith, ma` (all-lower, no lean) and `毛泽东, MA` (caseless, no lean) — the TRAILING slot has never gated its own report on the lean either (a bare ambiguous acronym reported before #289 too, pick declined or not), because the report is about a fork the parse CONSULTED, and the fork is consulted whenever a class member sits in the slot, lean or no lean. Fixed: the guard is class MEMBERSHIP on the first post-comma piece (`vocab:suffix-ambiguous` or `shape:acronym`, never the lean) — `Smith, ma` reports `"'ma' after the comma is also an ordinary name word; read as the given name"`, `毛泽东, MA` reports the same wording for `'MA'`, and `Smith, MA PhD` (a two-piece post-comma part) reports through its FIRST piece regardless of what follows it, exactly as a one-piece part would. The structure decision reports in `segment`, where that branch is taken, and the family-comma reading in `assign`, so one DECISION never reports twice — a second ambiguous token elsewhere is a second fork and reports on its own (`Smith MA, Ed` carries two `suffix-or-name` reports, one per fork, plus a `given-or-family` for the lone remaining word). +- 2026-09-14 (Derek) — #516's DOTTED HALF IS A SWITCH, `Policy.unlisted_dotted_suffixes`, DEFAULT ON: an unlisted token of two or more period-separated chunks, any chunk length (`X.Y.Z.`, `B.Tech.`, `Q.W.E.R.T.`), joins the ambiguous class BY SHAPE; whole-token vocabulary still wins (`M.A.`, `Ph.D.`, `A.B.C.` — `abc` IS a suffix acronym). The positional rule then decides and reports either way: `John Smith X.Y.Z.` → suffix, `Jack X.Y.Z.` → family, `Smith, A.B.` → given, `John Smith, A.B.` → suffix. Case is irrelevant — the periods are the evidence — so `john smith x.y.z.` reads as its mixed-case twin. Switch OFF: name material everywhere as 2.3 read it, and the fork is STILL reported, the parser having chosen the name reading over a credential one. +- 2026-09-15 (Derek) — THE ROMAN-CHUNK ACCIDENT RETIRES, NARROWLY. rules.md#S3's chunk rule survives except where every chunk the vocabulary matches is a SINGLE ASCII CHARACTER — the accident exactly. Measured, `{w for w in (L.suffix_acronyms | L.suffix_words) if len(w) == 1}` is `{i, v, 2}` plus the glued CJK honorific tails `{様, 殿, 氏, 군, 님, 씨, 양}`, so "single ASCII character" names the first three and only those — CHARACTER because `2` is a digit and is in the set, ASCII because `씨` is the one that must KEEP its chunk claim (`J.씨`). So `John Smith R.A.I.` and `John Smith J.u.n.i.o.r.` reach the shape class and read suffix by POSITION with the same fields plus the report, `Jack X.Y.I.` reads family where it read suffix (1.4.0's reading), and `Msc.Ed.`, `JD.CPA`, `Lt.Gov.`, `J.씨` keep their chunk-derived readings. The WIDE retirement — any chunk match yielding to the shape — was MEASURED AND REJECTED: it moves three more corpus names, `Doe, John Msc.Ed.` losing a real credential to `middle` and the two `김민준씨, J.씨` rows re-routing their honorific peel, for nothing this design wants. +- 2026-09-17/18, #516 review rounds — THE SHAPE VERDICT NEEDS TWO FURTHER GATES, both found by measuring after the shape verdict shipped. (i) ALPHABETIC: `period_joined_vocab`'s "shape" branch now requires every chunk to be `.isalpha()` — a bare digit chunk is not an acronym letter by any reading, so `Smith, 1.4`, `John Smith 1.4` and `John Smith, 1.4` stay unclaimed (no report, no role move), where an unguarded shape verdict had wrongly admitted them; the delimited `Bridge (1.4)` control only proved digits reach extract's escape, saying nothing about the same digits undelimited, which is the gap this closes. (ii) INITIALLESS-SCRIPT: the same test `is_title_shaped` already makes for H2's own leading-abbreviation inference (#323) — `text.isascii() or not in_initialless_script(text)` — now gates the shape verdict too, so `John Smith 田.中.`, `김 민준 이.박.` and `John Smith たな.か.` stay family: a script with no period abbreviations at all has nothing for interior periods to abbreviate, so a CJK word glued into period-separated characters is not spelling an acronym either. `John Smith 田.中.` is pinned `tolerated=True` rather than `shape=` — a Latin period convention glued onto CJK characters is a composed form no writing system produces, and the case table's own validator refuses a Latin shape tag on CJK text; `이.박.` and `たな.か.` are covered by direct `period_joined_vocab` assertions instead of a second corpus row each. +- 2026-09-17, #516 review round — `SMITH, J.R.` READS BY SHAPE FOR A REASON UNRELATED TO WHY IT DECLINES THE LISTED WORD. `_normalize` keeps interior periods ON PURPOSE (v1 parity, pinned 2026-07-17: `'J.R.'` must not collapse to the listed suffix word `'jr'`), so `ambiguous_class_member` correctly answers False for `Smith, J.R.`, `Smith, E.S.Q.` and `Smith, E.T., Jr.` for a reason that has nothing to do with #516. What #516 adds is a SEPARATE route: each chunk (`'j'`/`'r'`, `'e'`/`'s'`/`'q'`, `'e'`/`'t'`) is an unlisted single-letter chunk nothing in the vocabulary claims, so the token joins the class BY SHAPE instead, exactly as `'K.D.'` does in `J.A. K.D.`. The READING is unchanged in all three — one pre-comma name word is never enough to flip the structure — but each now gains `suffix-or-name`, the class having been considered and declined rather than never reached at all. + `is_wholly_suffix` NEVER READS THE BY-SHAPE CLASS. A first version admitted a by-shape member there unconditionally, reasoning it was needed for `John Smith, A.B.`'s structure flip; proved wrong by mutation — that flip already goes through `_segment.py`'s own `pre_comma_names >= 2` disjunct alone, and nothing else in the suite depended on the admission except this predicate's own two unit tests. Left in, the admission combined with C1's LEGACY token-count disjunct (which counts WORDS, not NAME words, and predates the name-word-count decision above) to flip `Smith Jr., A.B.` to given `'Smith'`, suffix `'Jr., A.B.'` with a self-contradicting "holds 1 name words" report. Fixed by dropping the admission outright: the by-shape class reaches the comma form ONLY through `_vocab.ambiguous_class_candidate`, never through `is_wholly_suffix`. `Smith Jr., A.B.` now reads given `'A.B.'`, family `'Smith'`, suffix `'Jr.'` — the same shape the LISTED class reads for `Smith Jr., Ma` and `Smith Jr., ma` (given `'Ma'`/`'ma'`, family `'Smith'`, suffix `'Jr.'`), not for `Smith Jr., MA` (given `''`, family `'Smith'`, suffix `'Jr., MA'` — the caps lean has already moved that one into the suffix run, measured on this tree; at 2.3.0 it read given `'MA'`, family `'Smith'`, suffix `'Jr.'`, the very shape `Smith Jr., A.B.` reads now) — with the report; classification `parity`, not `fix(#516)`: FULL PARITY on the 1.4.0 wheel (first A.B., last Smith, suffix Jr.), only the report being new. +- 2026-09-14 (Derek), corrected 2026-09-18 — #516's ALL-CAPS HALF IS A SWITCH TOO, `Policy.unlisted_caps_suffixes`, DEFAULT OFF, and the asymmetry is the decision. An unlisted all-caps word of two or more letters, with no period ANYWHERE (`isalpha()` rejects a period anywhere, not merely an interior one, so a single TRAILING period, H2's own shape, is unaffected either way), in a name `one_case` says is written in more than one case, joins the class by shape (`_vocab.caps_shape_candidate`). + UNLISTED MEANS IN NO WORDLIST AT ALL, not merely "no whole-token suffix vocabulary" — found only by measuring after a narrower, tag-based guard shipped and reversed: `titles`, `given_name_titles`, `particles`, `particles_ambiguous`, `conjunctions`, `conjunctions_ambiguous`, `bound_given_names`, `suffix_acronyms`, `suffix_acronyms_ambiguous`, `suffix_words` and `maiden_markers` — ELEVEN lists, checked by direct membership rather than by tag, since `segment` calls the shape test before `classify` has tagged anything. Measured with the narrower guard alone (`John Smith DE`, `John Smith NEE` wrongly moved to suffix), and stated here as a recipe rather than a fixed count, since a raw per-list membership tally and a de-duplicated one (a word appearing in two lists, `abu`/`أبو`/`ابو` across `bound_given_names` and `particles_ambiguous`, is one leak, not two) give different totals: `len([w for w in getattr(lexicon, name) if w.isascii() and len(w) >= 2 and w.isalpha()])` per excluded list, run over the shipped `Lexicon.default()`, gives on the order of dozens for `particles` and `particles_ambiguous` and single digits for `conjunctions` and `bound_given_names` — every one of them a wordlist entry the caps shape test's earlier, narrower guard did not check at all. The one figure that reproduces exactly, because ASCII case is what the test needs and every other maiden marker is a non-Latin or non-ASCII spelling the shape test's own `isupper()`/`isalpha()` gate excludes on its own: FOUR maiden markers are ASCII (`geb`, `geboren`, `geborene`, `nee`), out of `len(lexicon.maiden_markers)` total (17, measured 2026-09-18) — `'NEE'`/`'GEB'` are two of those four, capitalized, which is what the narrower guard let through. ACCEPTED GAP, recorded rather than chased: an all-caps German title spelled with `SS` for `ß` (`GROSSFÜRST`) is unlisted by this same lookup too, for a reason predating this switch entirely — `_normalize` uses `lower()`, never `casefold()`, specifically so it does not corrupt `großfürst` to the misspelling `grossfürst`, and classify's own existing `vocab:title` check has never recognized this spelling either. Left for whoever next generalizes the ß-fold question. + THE RUN IS THE CAPS CLASS'S ALONE. The multi-token shape `'LEED AP'` needs a RUN test (`John Smith, LEED AP` → suffix `'LEED AP'`), but the listed and dotted halves are single-token classes by construction — the listed set is inherently bare words and the dotted half never needed more than one token — so testing the run over either of them asks a question this design never posed. An un-narrowed run test measured three reversals before this was caught: `John Smith, Ed Ma` (both listed, Title-case, which #289's OWN lean reads as NAME), `John Smith, ma do` (both listed, lower) and `John Smith, X.Y.Z. A.B.` (both dotted) all wrongly moved to suffix as a run. Narrowed to the caps branch alone; `Smith, LEED AP` still declines and reports, and all three reversals are now pinned identical OFF/ON. + `JACK VI` AND `JOHN SMITH VI` DO NOT MOVE, AND NOT FOR THE REASON A FIRST DRAFT GAVE. `VI` passes the caps shape test and carries both `SHAPE_ACRONYM_TAG` and `vocab:suffix-ambiguous` with the switch on — the predicate reaches it, contrary to what an early write-up of this entry said. What actually holds it unchanged is rules.md#S2's OWN roman-numeral fork (`_pieces.is_trailing_numeral_suffix`), which claims a bare `'VI'` piece and decrements the peel's walk BEFORE the bare-ambiguous/shape branches beneath it are ever reached — a mechanism this switch does not touch and never held `VI` back through non-membership. + THE COST IS A TWO-WORD CONTRAPOSITIVE, not a flat one. Off by default because French and Korean records write the SURNAME in capitals — `Jean DUPONT`, `Minjun KIM` and `Jean Pierre DUPONT` all read family at 1.4.0, at 2.3 and at this default — and `isupper()` is script-agnostic, so the same decline holds in ANY script with a case contrast (`Иван ИВАНОВ`, `Jean ÉCOLE`, measured). Turning it on does not cost every candidate alike: at TWO words there are no words to spare, so `Jean DUPONT` and `Minjun KIM` report the fork and KEEP their family — the class is considered and declined, the mirror of the listed lean's own words-to-spare rule. At THREE words there ARE, and `Jean Pierre DUPONT` gives family `'Pierre'`, suffix `'DUPONT'` — the one genuine cost, and a swallowed family name is the worse of the two failures #vocabulary-collisions C-i already argues from. Default-off is 1.4.0 parity for the whole class regardless; on, nothing but the report changes at two words, and only the three-word shape actually moves. Off, NOTHING is reported either. +- 2026-09-14 (Derek), scope corrected 2026-09-18 — NO NEW AmbiguityKind. Every decision at these slots emits the existing `AmbiguityKind.SUFFIX_OR_NAME` — no new kind — including the comma path, which is the first report of the comma's OWN decision (listing or credential run) in the library (`COMMA_STRUCTURE` already reports on the comma path, but as a shape the parse could not recognize, not a fork it called); the comma-quiet policy in `_types.py` and rules.md#C1 gain an exception scoped to the ambiguous class. Switch B off emits nothing. This is narrower than an earlier wording of this entry claimed: an attachment decided AFTER a family comma is a separate fork, P6's, and it has reported on its own since 2.3 — measured on this tree and at master `1f78bef` (2.3.0), `parse("Berg, Jan vd")` gives given `'Jan'`, family `'vd Berg'`, `ambiguities=['suffix-or-name']`, emitted from `_pipeline/_post_rules.py`'s attachment fork, and `AmbiguityKind.SUFFIX_OR_NAME`'s own docstring in `_types.py` already lists that family-comma case. + THREE READING SITES, NOT TWO. The trailing suffix slot (`_pieces.peel_trailing`/`listed_lean`) and the post-comma given slot (`_pieces.segment_suffix_reading`) are the two the decision names first; `_vocab.is_wholly_suffix` over a `groups[2:]` TAIL segment is the third, and part of the blast radius rests on it: `Steven Hardman, MD, DO, DDS` loses its `comma-structure` flag only because the lean makes segment 2 (`DO`) wholly suffix. That is the one place this arc QUIETS a report rather than adding one, and it has a case row of its own because a quieted flag is invisible to a reader sweeping for new ones. +- 2026-09-14, corrected 2026-09-17 and 2026-09-18 — WHERE THE FACT LIVES, AND WHY THERE. The comma forms need the class decision at SEGMENT time, two stages before the own-words span is computable from marker TAGS. Four options were measured on a runtime probe (baseline 410.00 frames): segment computing it with a direct marker-head walk, +15; the same re-running the whole marker pass, +16; the fact staying in classify while classify RE-DECIDES `structure`, +0 but refused ON ORDERING — `structure` is read between segment and classify by `_script_segment.py`, which returns early on FAMILY_COMMA, so a flip decided in classify would leave the surname-split gate un-run on a name that turns out to have a suffix comma (measured: `Smith 김민준씨, MA` reads given 'Smith', middle '김', family '민준', suffix '씨, MA' on this tree — the figure first written here, given 'MA' with family 'Smith 김민준씨', was the reading before the flip moved to segment, re-measured 2026-09-18 — and a DIFFERENT wrong split again with the flip decided before script_segment — two stages disagreeing about one parse is what mechanisms.md#PIPELINE-STAGE-CONTRACTS exists to forbid); the marker pass and the fact MOVING to segment, +8, refused because killing the duplicated fold means carrying it on the state through CJK splits, a new bug surface in the one stage that changes token COUNT. SHIPPED: one shared helper, `_pieces.own_words(tokens, comma_offsets, markers, marker_tags=None)`, called by classify always (passing its own tag map) and by segment only when a comma form could turn on the fact. Every figure below is a TOTAL against `1f78befd` (the merge base this whole arc branched from), re-measured 2026-09-18 rather than accumulated per-commit: a `sys.setprofile` harness shaped like `tools/perf/call_count.py`'s `calls_for` (one warm-up parse, mean of 50, module-level `nameparser.parse`, run as a SCRIPT FILE outside either checkout so neither run's `sys.path[0]` masks the other — a `-c` one-liner instead would let the invoking shell's cwd shadow the older snapshot and read the current tree twice), against a `git archive 1f78befd nameparser` extraction on one side and this tree's own editable install on the other, same resolved interpreter both sides. Reference name (`tools/perf/call_count.py`'s own): 411 → 412 (+1). A comma-less, non-trigger control, `John Smith`: 175 → 176 (+1) — segment's lazy `own_words` call never fires without a comma, so this is classify's own always-on share of the cost. Comma names: a non-trigger, `Smith, John`: 203 → 207 (+4); `John Smith, Jr.`: 229 → 233 (+4); `Smith, John, Jr.`: 253 → 257 (+4) — every non-trigger comma name pays the SAME +4 total, dominated by classify's shared span cut plus the cheap case-free candidacy tests the three reading sites each add, not by the lazy fact itself (which these names never force). A genuine trigger, `Smith, MA`: 203 → 225 (+22) — the lazy fact plus its three readers' own work once a real candidate is found. (An earlier draft of this paragraph reported "+1 to +2 on non-trigger comma names" and "+19 to +20" on `Smith, MA`; those were PER-COMMIT deltas from the fix-round commentary above, accumulated inexactly rather than measured as one total, and are superseded by the totals here.) `own_words` shares its exact span-cutting walk with classify's own cut through a second shared function, `_vocab.tag_marker_runs` (moved out of `_classify.py` so `_pieces.py`'s own layering rule — it may import `_state` and `_vocab` only — is not broken): both call sites run the identical walk over the identical inputs, including `comma_offsets`, so a marker phrase like `z domu` is bucketed the same way at a comma either side. A corpus probe over the 1174 corpus entries confirms 0 cut differences and 0 verdict differences between the two call shapes (an earlier, text-only approximation that predated sharing `tag_marker_runs` differed on the CUT for 2 names — `Anna z Nowak`, `Anna z (domu) Nowak`, where `z` opens the phrase entry `z domu` but no run completes — while agreeing on the BOOLEAN for all 1174 even then). WITH SWITCH B ON, a genuine caps candidate pays more, recorded rather than chased to zero since nothing gates it — re-measured 2026-09-18 with `Parser().parse` on both sides of the on/off comparison (the convention `tests/v2/test_policy.py`'s own test uses), against this tree's own default: `Smith, John` 206 → 213 (+7), `Smith, XYZ` 205 → 245 (+40). Recompute (`tools/perf/call_count.py`'s own `calls_for` counts only its fixed reference name, so a chosen comma name needs the same shape spelled inline): `uv run python -c "import sys; from nameparser import Parser, Policy +def calls_for(fn, name, n=50): + fn(name); calls=0 + def counter(frame,event,arg): + nonlocal calls + if event=='call': calls+=1 + sys.setprofile(counter) + try: + for _ in range(n): fn(name) + finally: sys.setprofile(None) + return calls/n +off=Parser(); on=Parser(policy=Policy(unlisted_caps_suffixes=True)) +for n in ('Smith, John','Smith, XYZ'): print(n, calls_for(off.parse, n), calls_for(on.parse, n))"` (both cheaper than an unconsolidated spelling would be, after the caps predicate was consolidated into one shared function, `_vocab.caps_shape_candidate`, called from classify, `ambiguous_class_candidate` and `_segment.py`'s run test alike — the usual "a shared call costs the default a frame" objection does not apply, because every caller's own first conjunct is `policy.unlisted_caps_suffixes` itself, False by default; `tests/v2/test_policy.py`'s own dated comment carries the same two figures and the recipe for `tools/perf/call_count.py` reference readers). The band never moved at either policy — `test_benchmark.py` gates only the comma-less reference name, which no comma-path cost can reach — and `_CALL_BASELINE` is untouched throughout. + NO MECHANISMS ENTRY IS OWED, and here is why: the promoted fact is STATE-OFFSET-CHANNELS' "a pre-token fact is recorded on the ParseState and consulted later" applied to a name-level boolean rather than to offsets, and `ParseState.order` is the precedent that entry and ONE-PREDICATE-PER-QUESTION both already name. That entry's own sentence — "recorded as offsets … or by presence alone where the fact is name-level" — already covers it; the one edit made there is to name `one_case` as the third channel. +- 2026-09-15 (Derek) — ACCEPTED, NOT REPAIRED, TWO SHAPES. (i) `Freiherr von Berg MA` MOVES: it was listed as unreachable by the peel and as staying fix(#424)'s, and measured, the caps lean makes it reachable, because the lean ends P2's chain with no words to spare where the count would not. Today title 'Freiherr', family 'von Berg MA'; designed title 'Freiherr', family 'von Berg', suffix 'MA'; 1.4.0 first 'von Berg', last 'MA'. Three readings exist and the designed one ships; it moves a rules.md#P2 example line and re-scopes fix(#424)'s ledger rule, whose `fields` gains `suffix`. (ii) `abdul Smith Jr Ma` reads `Jr` as a MIDDLE NAME: the surname lean breaks the peel AT `Ma`, so the unambiguous `Jr` in front of it is never reached. Today given 'abdul', family 'Smith', suffix 'Jr Ma'; designed given 'abdul Smith', middle Jr, family 'Ma'. One case row rather than real data, but the shape is real — a name-leaning ambiguous acronym BLOCKS the peel of a genuine suffix behind it — and the walk stopping at the declined pick is the rule, not a bug the peel owes a repair. +- OUT OF SCOPE, each its own issue or entry: no tokenizer split (`J.R.R.`, `A.B.` stay single tokens); the trailing-period-only shape is NOT the signal (`John Smith Xyz.` → family, and AGENTS.md's trailing-abbreviation paragraph stands for bare words and sentence-final stops); no new `AmbiguityKind`; no locale packs; #492 stays deferred. NEITHER SWITCH REACHES THE v1 SHIM, and that is a consequence stated rather than repaired: `lenient_comma_suffixes` has no v1 manager either and `_build_snapshot()` simply does not translate it. Switch A is DEFAULT ON, so `HumanName` changes with it and a v1 caller cannot turn it off — the deliberate reading of the bridge, where the facade tracks the parser's defaults and 2.x behavior changes reach it (the #528 precedent). +- 2026-09-18 — RULES.MD STATEMENTS NEVER NAME A POLICY FIELD, AND THIS ARC MADE THAT LOAD-BEARING. rules.md's own preamble binds statements to be implementation-free ("no stage names, no function names, no regexes"), and no existing statement names a `Policy` field either — C1 says "strict mode" and leaves `strict-comma-suffixes` to the annotation slot. S2's and S3's switch clauses follow that precedent rather than setting a new one: the prose describes `unlisted_dotted_suffixes` and `unlisted_caps_suffixes` as caller configuration and points at the example lines, and the ANNOTATIONS carry the exact field names (`unlisted_dotted_suffixes-off`, `unlisted_caps_suffixes-on`, registered in `tests/v2/rules_doc.py`'s `POLICIES` beside `strict-comma-suffixes`). Recorded here rather than as a silent edit to the preamble or to two rules' prose: if a future design wants a field named directly in a statement, that is a preamble amendment for someone to propose, not a convention this entry changes on its own authority. +- 2026-09-18, corrected — `BUILD_RULES_CORPUS.PY` HARVESTING A POLICY-ANNOTATED-ONLY EXAMPLE IS LONG-STANDING, NOT NEW, AND THIS ARC ADDS TWO MORE INSTANCES OF IT. Its own docstring says "Policy annotations are deliberately ignored... a family-first-scoped example is simply one more name to diff" — an earlier draft of this bullet claimed that sentence was written when every annotated example had a default-policy sibling; measured false, at the commit that wrote it and on this tree alike. Recompute: parse every rule's examples with `tests/v2/rules_doc.py`'s own `parse_rules_doc`, group by `text` across the WHOLE document, and keep every text whose every appearance's annotation resolves (`resolve_annotation`) to kind `"policy"` rather than `"locale"`/`"gated_locale"` and never appears bare. That gives **19** pre-existing examples (`Mesnil de`, `Smith, de Mesnil Jean`, `de la Vega y Santos Juan`, `ibn Awf abdul Rahman`, `Nguyen Thi Van`, `Jane Smith (Nee)`, `Jane Smith (Nee) (Jones)`, `Garcia Juan Carlos`, `Nguyễn Thị Minh Khai`, `Del Toro`, `Anh Van Do`, `Beethoven Ludwig van`, `Berg Jan de Jr.`, `Garcia`, `Ménil Christophe de`, `Ménil de`, `Smith (Jones)`, `de Anke van y`, `van Berg Jan de`) plus this arc's own **2** (`Jean DUPONT`, `Jean Pierre DUPONT`, both `unlisted_caps_suffixes-on`, rules.md#S2) — **21** total on this tree. So the shape was accepted the day the docstring was written, not discovered here; what this arc adds is two more members, checked here for the reason the advice below names: `Jean Pierre DUPONT`'s DEFAULT reading (given Jean, middle Pierre, family DUPONT — measured; the annotated ON reading, given Jean, family Pierre, suffix DUPONT, is the only one rules.md states) never appeared in the doc before, and it produces no diff at any of the five baselines either way. Recorded per the same over-collection principle that keeps `(` and `.,` in the corpus (`tools/differential/README.md`'s provenance table): an annotated example's TEXT is a default-policy corpus name regardless of its annotation, full stop, and every one of the 21 — not only this PR's two — is a standing instance a reader should check for a default-policy diff on before assuming it free. +- 2026-09-18 — THE BLAST RADIUS IS THE GATE'S OWN COUNT, NOT A PREDICTION. Measured at 2.3.0 (`uv run python tools/differential/compare.py`, the default baseline, the previous release), and RE-MEASURED after the review round later the same day and again after the verification round that followed it, which is why several sets of figures carrying one date would otherwise sit here: corpus 1234 names, intentional diffs 51, unexplained 0, radar unclassified 7 — SUFFIX_OR_NAME rows the gate reports but no rule yet claims, named so a release note is never quantified from the intentional count alone. The round added TEN corpus names, not the eight first written here (corrected in the verification round of 2026-09-18): `Doe, John X.Y.Z.`, `Dr. 김민준씨, MA`, `John Smith, MD, Ma`, `John Smith, MD, R.A.I.`, `John de Ma`, `Jo김민준씨, DO`, `Kim김민준씨, MA`, `STEVEN HARDMAN, MD, DO, DDS`, `anh van mc`, `毛泽东 MA` — case rows and rules.md example lines both. The verification round itself then added THREE more (`Anh van Do`, `Doe, John MA`, `John Smith, MD, XYZ`, all rules.md boundary examples), taking the gate's count at this baseline from 1231 to 1234. Recompute the delta rather than trusting either figure: load every `tools/differential/corpus*.jsonl` through compare.py's own `_load_entries`, dedup by name in file order, and diff that list against the same list built from `git show :tools/differential/corpus*.jsonl`. That by-name count runs four below the gate's at the 2.x baselines, the gate deduping by (name, order) and four entries carrying a declared order; the DELTAS agree, which is what a "the round added N" sentence is claiming. The round also added a FOURTH rule, `fix(#289/#516) the glued CJK honorific peel reads the case lean` (3 composed CJK names: `Dr. 김민준씨, MA`, `Jo김민준씨, DO`, `Kim김민준씨, MA` — radar-tier by demotion and classified anyway, since a radar diff this arc intended is one a release note may be written from). The four rules carry 20 + 9 + 9 + 3 = 41 of the 51, `John de Ma` having joined the first. The pre-round reading of this paragraph, kept for the shape of the argument rather than the digits: `fix(#289) a written case contrast decides a bare ambiguous acronym` (19 names, incl. `Davis Royce, Ed`, `Freiherr von Berg MA`, `Jack MA`, `Jack Wei Ma`, `John Prof. MA`, `John Smith Ma`, `John Smith, Ed`, `John Smith, MA`, `John Smith, Ma`, `John van der Berg Ma`); `fix(#516) an unlisted dotted acronym is read by position` (9: `Jack X.Y.I.`, `John Smith Q.W.E.R.T.`, `John Smith X.Y.Z.`, `John Smith, A.B.`, `john smith x.y.z.`, `John Smith B.Tech.`, `John Smith C.H.A.`, `John Smith E.S.Q.`, `Smith, E.S.Q.`); `fix(#289/#516) the ambiguous credential class reports at slots that were silent` (9: `John Smith J.u.n.i.o.r.`, `Royce, Ed`, `Smith, A.B.`, `Smith, Ma`, `J.A. K.D.`, `Jack X.Y.Z.`, `John Smith R.A.I.`, `Smith Jr., A.B.`, `Steven Hardman, MD, DO, DDS`). This is LARGER than the plan's original "11 role + 4 report-only" prediction at the default policy — entirely by corpus growth (this arc's own commits admitted the case-row texts above as contract-tier names) and by two mechanisms the runtime prototype that produced that figure predated (`ambiguous_class_candidate`'s existence and the family-comma report's shape-tag branch), never by a wider rule. Recompute with the command above and read the count the gate returns; the numbers here are a snapshot dated 2026-09-18, not a promise. +- 2026-09-18, WHOLE-PR REVIEW ROUND — WHAT IT MOVED, MEASURED BEFORE AND AFTER. Ten findings; the five that changed behavior are here with the shape each was found by, since the shape is what generalizes. (1) A LOST REPORT. Where the prefix chain absorbed an acronym the S2 peel had DECLINED, nobody emitted the flag S2 promises for either reading: `assign` reports from `peel_trailing`'s picks and a pick reaches it only as a LONE piece, so the token it would have reported on no longer existed. Before: `John van der Berg Ma`, `John de Ma`, `Dr. John van Smith Ma`, `John van Smith Ma Jr.` and `John Smith Mc Ma` gave their family names with `ambiguities` empty; after, each names `suffix-or-name` once and no role moves. The emitter is at the chain's own merge and declines where the PARTICLE run took the word (`anh van do`, `anh van mc` — P6's fork, not S2's, and 1.4.0 read both silently). Recompute: parse the five names and read `ambiguities`. (2) TWO STAGES ASKING ONE QUESTION WITH DIFFERENT INPUTS. `_script_segment` asked `is_wholly_suffix` of the post-comma run WITHOUT the `one_case` fact `segment` had recorded one stage earlier, so `Kim김민준씨, MA` read family `Kim김민준씨`, suffix `MA` (씨 unpeeled) while `Kim김민준씨, PhD` read family `Kim김민준`, suffix `씨, PhD` — one name, two credential spellings, two divisions. After: both peel. `Jo김민준씨, DO` and `Dr. 김민준씨, MA` move with them; `김민준씨, MA` does NOT, being wholly one case (Hangul is caseless and `MA` contrasts with nothing), which is the doctrine holding rather than an exception. rules.md#W3's claim that the two stages differ "in what else they require, not in what they ask of the words" was false until this and is true now. (3) A REPORT ABOUT THE PARSER'S OWN NEW READING. rules.md#S3's narrow retirement moved `R.A.I.`, `X.Y.I.` and `J.u.n.i.o.r.` from the vocabulary verdict into the shape class, and `segment`'s tail-segment test reads `is_wholly_suffix`, which is blind to that class — so `John Smith, MD, R.A.I.`, `John Smith, Ph.D., R.A.I.` and `John Smith, MD, J.u.n.i.o.r.` gained a `comma-structure` flag 2.3 did not raise. After: quiet, at the default. The test is narrowed to the BY-SHAPE half deliberately — a LISTED member reaches that reading through the case lean, and `STEVEN HARDMAN, MD, DO, DDS` (one case, nothing leans) keeps its flag as the lean's own recorded negative control; `John Smith, MD, Ma` keeps it too. With `unlisted_dotted_suffixes=False` the flag stands, the token being name material there. C2 is amended to state all of this. (4) THE WORDLIST ROSTER. `_vocab.caps_shape_candidate`'s exclusion named ELEVEN of `_lexicon._VOCAB_FIELDS`' thirteen: `Lexicon.default().add(surnames={"dupont"})` with the caps switch on still read `Jean Pierre DUPONT` as given `Jean`, family `Pierre`, suffix `DUPONT` — a caller listing a word as a SURNAME and getting a credential, which is this switch's own worst failure arriving through the one wordlist that says "family name". After: family `DUPONT`, no report. The roster is derived from the field tuple, with a per-field test; `honorific_tails`, the other omission, was already excluded transitively (⊆ `suffix_words` by Lexicon invariant) and joins for completeness. (5) A LISTED MEMBER READ BY SHAPE. A caller may list a DOTTED entry in `suffix_acronyms_ambiguous` ('a.b'), and the whole token then matches that set while `suffix_as_written`'s period-free acronym lookup ('ab') misses it — so the chunk view reached classify's shape branch, `SHAPE_ACRONYM_TAG` rode along, and `_pieces.listed_lean` (which declines wherever that tag is) silenced the caller's own case lean: `Jack A.B.` read family where `Jack MA` reads suffix. After: suffix `A.B.`, the lean applying. Nothing default moves, the shipped ambiguous vocabulary carrying no periods. Also in the round and behavior-neutral: `Policy.unlisted_dotted_suffixes` and `unlisted_caps_suffixes` moved to the END of `Policy` and `PolicyPatch` — inserted mid-class, they had re-bound every positional argument from `strip_emoji` on, which is API for a dataclass that is not `kw_only`; the bool-validation roster is derived from `dataclasses.fields`, the hand-written twin in the tests having named neither new flag; `ambiguous_class_candidate` lost an optional `one_case` parameter no production caller ever passed, and with it a caps branch that answered False for every name the library has parsed; and `one_case` is REQUIRED on the four piece predicates that own it. +- 2026-09-18 — DIGIT-DOTTED TOKENS MOVE, SILENTLY, AND IT IS ACCEPTED. The narrow retirement's set of single-ASCII-character suffix entries is `{i, v, 2}` — the roman numerals AND the lone digit — so a dotted token whose only chunk match is that digit loses the chunk claim, and the SHAPE class cannot take it either: `period_joined_vocab`'s shape verdict wants every chunk `isalpha()`, which no digit is. Measured against 1f78bef: `John Smith 1.4.2` suffix `1.4.2` → family `1.4.2`; `Smith, 1.4.2` family `Smith`/suffix `1.4.2` → given `1.4.2`/family `Smith`; `John Smith, 1.4.2` given `John`/family `Smith`/suffix `1.4.2` → given `1.4.2`/family `John Smith`. No report at any policy, and the switch does not reach it either way. ACCEPTED: a version string read as a credential was the same accident the retirement removes, and the three readings are pinned as case rows. `John Smith 1.4` was already family at 1f78bef and is unaffected — two chunks, neither a match. docs/release_log.rst's wording is corrected from "a single-character roman numeral" to the digit-inclusive form, which is what the code tests. +- 2026-09-18 — THE COMMA REPORT'S REACH IS THE FIRST POST-COMMA PIECE, AND THE GIVEN SEGMENT'S TRAILING SLOT IS AN OPEN FOLLOW-UP. `assign`'s family-comma emitter reads `state.pieces[1][0]` — the first piece after the comma — so a class member standing at the END of the given part is never asked about. PRE-EXISTING and unchanged by this arc: `Doe, John MA` reads middle `MA` and `Doe, John X.Y.Z.` middle `X.Y.Z.`, both silent, at 1f78bef and on this tree alike. What this arc adds is names that MOVED into that silence: `Doe, John J.u.n.i.o.r.` and `Doe, John X.Y.I.` went suffix → middle, and `John Smith nee Jones R.A.I.` suffix → maiden (the maiden walk absorbs it). `Freiherr von Berg X.Y.I.` was listed here too and does not belong: corrected in the verification round of 2026-09-18, it moved suffix → family but GAINED a report rather than losing one. At 1f78bef and on the 2.3.0 wheel it read title `Freiherr`, family `von Berg`, suffix `X.Y.I.`, reporting `particle-or-given` alone; on this tree it reads title `Freiherr`, family `von Berg X.Y.I.` and reports `particle-or-given` AND `suffix-or-name`, the chain emitter taking it by its shape tag. The role moved into a slot that does not report and the report came from the chain instead, so the name is evidence for the emitter rather than for the silence. The universal claims are narrowed to what ships rather than the emitters widened: `nameparser/_types.py`'s `SUFFIX_OR_NAME` member now names the slots it covers and the one it does not, and rules.md#S3's "and the same at a comma" now says which parts of a comma form it means. Widening it is a MAINTAINER DECISION and is left open, filed as [#531](https://github.com/derek73/python-nameparser/issues/531): the slot is `_assign`'s given-part walk, the fork is genuine there, and whether a middle initial's neighbourhood should start reporting is a judgement about noise (rules.md#A1's "a kind is worth adding only if a reader would hesitate too" applies to a SLOT the same way). And the open question is a 1.4.0 PARITY question as well as a noise one, which the verification round of 2026-09-18 established and the entry had backwards: `Doe, John MA` reads suffix `MA` on the 1.4.0 wheel and middle `MA` at 2.0.0 through 2.3.0 and here, so the silence at this slot is a 2.0-era reading that has never matched v1 — "every release" was the wrong span and it is now "every 2.x release" wherever this is stated (`nameparser/_types.py`'s `SUFFIX_OR_NAME` member, docs/release_log.rst). The dotted sibling does not divide that way: `Doe, John X.Y.Z.` is middle `X.Y.Z.` at 1.4.0 too. So a maintainer widening the slot is choosing between reporting the fork and restoring v1's ROLE, which are different repairs, and only the bare-acronym half has a parity claim behind it. Recorded under 3-0-reevaluations' standing rule that a design citing 1.4 parity as load-bearing says so where the design lives. Two more report movements recorded here rather than repaired: `van MA` LOSES its `particle-or-given` (1f78bef reported both kinds; the caps lean now takes `MA` to suffix, leaving `van` the only name piece, and assign's leading-particle emitter wants more pieces following), and `John Smith Ma Ma` reports ONCE where 1f78bef reported twice (the first `Ma` is a middle name now, not a second declined pick). And a reach of the default-on dotted switch worth naming: `John Smith www.example.com` reads suffix `www.example.com` with a report, where 1f78bef read family — a URL is three alphabetic period-separated chunks and the shape rule cannot tell it from an acronym. +- 2026-09-18 — A KNOWN ROUGH EDGE UNDER `lenient_comma_suffixes=False`, FOR DEREK'S ATTENTION. Measured: `Smith V., Ma` gives given `Smith`, family `V.`, suffix `Ma`, and `Smith V, Ma` gives given `Smith`, suffix `V, Ma`. The cause is the strict knob rather than the credential class: under strict, the initial-shaped `V.` fails the suffix test, so `name_word_count` sees TWO name words before the comma, flips the structure, and `Smith V.` is then read positionally. The control that shows it is the knob's and not this arc's: `Smith V., PhD` reads given `Smith`, family `V.`, suffix `PhD` under the same knob on master. Consistent, then, and left as it is — recorded with case rows so a future change to either the knob or the count has to notice it. +- 2026-09-18 — THE PREDICTION THAT DID NOT SURVIVE ITS OWN MEASUREMENT, kept because the failure mode is this log's recurring one. A review round called THREE `one_case` plumbing sites in `_group.py` inert and proposed removing them. Measured with a runtime wrapper over 1,642 names under six policies — the default, both family-first orders, strict commas, and each 2.4 switch flipped — 9,852 parses: the maiden walk's PAIR is inert, 0 parses move, and it is inert BY CONSTRUCTION rather than by corpus luck (`numeral_only` answers off `peeled.numeral`, and the numeral fork is decided before the peel reads a lean). The chain-tail measure is not: dropping it there moves 18, on `John van der Berg Ma`, `John de Ma` and `Freiherr von Berg MA` under every one of the six. Both figures are at the call sites. The population is the DISTINCT union, in file order, of every `tools/differential/corpus*.jsonl` entry, every `tests/v2/cases.py` text, and `tests/test_variations.TEST_NAMES` with the three comma permutations `test_variations_of_TEST_NAMES` builds (no-comma, family-comma, and suffix-comma where the name has a suffix), empty strings dropped; the wrapper patches the call sites in a copy of the module source and rebinds `_pipeline.STAGES`, which binds stage functions at import and would otherwise leave the mutation inert. This entry first carried 2,338 names and 14,028 parses, which that recipe does not reproduce by any reading of it — the distinct union is 1,642, summing the three sources' distinct counts gives 2,202, and not deduping at all gives 2,690 — so both figures were re-measured on this tree in the verification round of 2026-09-18 and restated here and at the call sites. The FINDING was unharmed: 0 and 18 reproduce exactly, on the same three names, under all six policies, and the control that drops all four sites at once moves those same 18, which is what shows the wrapper can report a difference at all. The argument was right for two sites out of three and would have taken a third with it. +- 2026-09-18 — THE COMMA HARNESS AND THE FRAME BAND, AFTER THE ROUND. Unchanged at the default: 412/449 on the reference name (`uv run python tools/perf/call_count.py`), and the module-`parse` totals above hold exactly — `John Smith` 176, `Smith, John` 207, `John Smith, Jr.` 233, `Smith, John, Jr.` 257, `Smith, MA` 225. WITH SWITCH B ON the two figures recorded above each drop by one: `Smith, John` 213 → 212 and `Smith, XYZ` 245 → 244, `Parser().parse` on both sides as before. The cause is the dead caps branch leaving `ambiguous_class_candidate`: `segment`'s single-token call used to reach a `caps_shape_candidate` call that could only ever answer False, and now does not. Recompute with the recipe above. +- 2026-09-18, VERIFICATION ROUND — THE REPORT INVENTORY AN UPGRADER SEES, 2.3.0 → TREE, AND ITS RECIPE. docs/release_log.rst's comma-report bullet claimed TWO reports go away and named `Steven Hardman, MD, DO, DDS` and `John Smith, MD, R.A.I.`. The second is false and was never measured against the wheel: `John Smith, MD, R.A.I.` reports NOTHING at 2.3.0, having carried a `comma-structure` flag only inside this arc's own development, where rules.md#S3's narrow retirement raised it and the C2 amendment quieted it again. Over the whole corpus exactly ONE report is lost, and one more is SWAPPED. Measured: LOST — `comma-structure` ×1 (`Steven Hardman, MD, DO, DDS`), `given-or-family` ×2 (`Jack X.Y.I.`, and `e and e`, which is PR #527's conjunction change and not this arc's). GAINED — `suffix-or-name` ×37, `conjunction-or-initial` ×9 (#527's again), `given-or-family` ×2 (`Jack MA`, `Jack MA.`). So `Jack X.Y.I.` is the only name whose report CHANGED KIND on this arc's account, `given-or-family` → `suffix-or-name`, and the bullet now names it as the swap rather than counting it among the losses. Recompute: collect the corpus glob through compare.py's `_load_entries` deduped by name, parse each name on the tree and on the 2.3.0 wheel (a PEP 723 `uv run --no-project` script pinning `nameparser==2.3.0`, per AGENTS.md's warning that a same-tree comparison reports parity), and take the per-name multiset difference of the `ambiguities` kind values in both directions. The kind counts are multisets, not name counts — `e and e` contributes two `conjunction-or-initial` gains on its own — which is why a bullet quantified from the gate's per-rule name lists would not answer this question. ### indic-honorifics — the renunciate class and the Indic honorific vocabulary (2026-09-06, #346/#344/#343) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 8ea2697c..d8474950 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -47,7 +47,7 @@ How it works. The two layers compose without ordering bugs because the positiona ## STATE-OFFSET-CHANNELS — early facts ride the state -Problem shape. A fact known during tokenization matters to a much later stage. Contract statement. A pre-token fact is recorded as offsets on the ParseState (comma_offsets, interpunct_offsets) and consulted later by position — or by presence alone where the fact is name-level, as both interpunct consumers do — rather than re-derived from text. How it works. The offsets survive every intermediate stage untouched; #298's transcription marker rides this channel from tokenize to order resolution (rules T3/W4). Lives in. nameparser/_pipeline/_state.py, produced in _tokenize. Reach for it when. You are about to re-scan the original string in a late stage to rediscover something tokenize already knew. +Problem shape. A fact known during tokenization matters to a much later stage. Contract statement. A pre-token fact is recorded as offsets on the ParseState (comma_offsets, interpunct_offsets) and consulted later by position — or by presence alone where the fact is name-level, as both interpunct consumers do, or as a single boolean for the whole name, as `ParseState.one_case` is (#289/#516: the written-case fact three later sites read — the trailing suffix slot, the post-comma given slot, and the tail-segment reading — none of which may disagree about it) — rather than re-derived from text. How it works. The offsets survive every intermediate stage untouched; #298's transcription marker rides this channel from tokenize to order resolution (rules T3/W4). `one_case` differs from the offset channels in WHO writes it: it is computed once by whichever of segment and classify needs it first (classify always, segment only where a comma form could turn it on), recorded as `None` for "not asked yet", and never recomputed once set (decisions.md#S2). Lives in. nameparser/_pipeline/_state.py, produced in _tokenize (the offsets) or in segment/classify (`one_case`). Reach for it when. You are about to re-scan the original string in a late stage to rediscover something tokenize already knew. ## PIPELINE-STAGE-CONTRACTS — the ownership map @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped; and in_initialless_script, the #322/#323 instance and the only one here that is a REPERTOIRE test rather than a vocabulary one — the script half of the #320 initial veto, read by is_initial one function away and by _pieces.is_leading_title, so "a script with no initials has no period abbreviations either" is one predicate over _policy._NO_INITIALS rather than a second reading of that table; it lost its leading underscore when the second caller arrived) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles and peel_walk are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5), and since the /simplify round of 2026-09-09 the SHARED predicate is `tail_reading` instead — the peel-and-chain fixed point that answers where the name pieces end (decisions.md#H5). Assign calls it at its main walk and group's bound-given reserve calls it twice, once per view the join compares, because that reserve reads the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). `peel_trailing` and `trailing_titles` are what that fixed point is BUILT from, and neither is a two-stage question any longer: `peel_trailing` has no caller outside `_pieces.py` at all, `trailing_start` and `tail_reading` being the only two and both in the leaf, and `trailing_titles` has exactly one, assign's family-comma segment-1 walk, which reads the chain without the re-peel; `_group.py` imports neither. The tail reading is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by an AST call-site census over `_pipeline/*.py` — every call node whose callee is one of these names, keyed by module and enclosing function, which is what caught the census claiming a share for `peel_trailing` that the round had just taken away — the rest of it holds unchanged: is_suffix_piece, leading_titles, peel_walk and now tail_reading shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped; and in_initialless_script, the #322/#323 instance and the only one here that is a REPERTOIRE test rather than a vocabulary one — the script half of the #320 initial veto, read by is_initial one function away and by _pieces.is_leading_title, so "a script with no initials has no period abbreviations either" is one predicate over _policy._NO_INITIALS rather than a second reading of that table; it lost its leading underscore when the second caller arrived; and caps_shape_candidate, the #516 instance and the newest, called from three sites that each needed the identical question answered — classify's own tag emission, this module's ambiguous_class_candidate, and _segment.py's multi-token run test — where the usual reason for keeping such copies apart (a shared call costing every default-policy parse a frame it cannot use) does not hold, because every one of the three callers' own FIRST conjunct is the caller-configured switch itself, `Policy.unlisted_caps_suffixes`, False by default, so the shared call is never reached at the default regardless of how many callers share it (decisions.md#S2)) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles and peel_walk are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5), and since the /simplify round of 2026-09-09 the SHARED predicate is `tail_reading` instead — the peel-and-chain fixed point that answers where the name pieces end (decisions.md#H5). Assign calls it at its main walk and group's bound-given reserve calls it twice, once per view the join compares, because that reserve reads the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). `peel_trailing` and `trailing_titles` are what that fixed point is BUILT from, and neither is a two-stage question any longer: `peel_trailing` has no caller outside `_pieces.py` at all, `trailing_start` and `tail_reading` being the only two and both in the leaf, and `trailing_titles` has exactly one, assign's family-comma segment-1 walk, which reads the chain without the re-peel; `_group.py` imports neither. The tail reading is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by an AST call-site census over `_pipeline/*.py` — every call node whose callee is one of these names, keyed by module and enclosing function, which is what caught the census claiming a share for `peel_trailing` that the round had just taken away — the rest of it holds unchanged: is_suffix_piece, leading_titles, peel_walk and now tail_reading shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/design/rules.md b/docs/design/rules.md index 60bdf378..21cf0010 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -133,7 +133,7 @@ H2. Rationale: before a name, an abbreviation is almost always a not open: the vocabulary decides, and "Esq." is the postnominal it is. "Smith, Esq." → suffix="Esq." - history: decisions.md#H2 · interacts: C1, P4, H5, W3, W4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py + history: decisions.md#H2 · interacts: C1, P4, H5, W3, W4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py H3. Rationale: compound titles are written as a run of title words, connectives included; a title word standing inside the name is @@ -298,7 +298,8 @@ H5. Rationale: a word abbreviated with a period at the END of a name is then taken over what stands, not over what stood. "John Smith Prof. Jr." → suffix="Jr." "John Smith Jr. Prof." → suffix="Jr." - "John Prof. MA" → family="MA" + "John Prof. MA" → suffix="MA" + "JOHN PROF. MA" → family="MA" Accepted: the reach is the whole title vocabulary, the ordinary surnames in it included. TITLES holds king, judge and bishop, and a period written behind one of them is enough to make it the @@ -437,7 +438,14 @@ P2. Rationale: a particle is written as part of the surname it a trailing suffix begins — read as assign will read it (S2), over the pieces the chain leaves: a trailing roman numeral, or a bare acronym with words to spare, ends the chain as a suffix word - does — a maiden marker takes the + does, and so does a bare ambiguous acronym written in capitals in + a mixed-case name, which needs no words to spare; the same + acronym written Title-case in a mixed-case name ends nothing and + joins the chain as any name word does, words to spare or not, + and the fork the chain called is reported there exactly as it is + where no particle stands (S2) — + a maiden + marker takes the words after it (M2), or the name ends. The final group reads as the family name; earlier groups read by position. The chain begins wherever the @@ -454,18 +462,34 @@ P2. Rationale: a particle is written as part of the surname it "John van der Berg PhD" → family="van der Berg" "John van der Berg V" → family="van der Berg" "John van der Berg V" → suffix="V" - "John van der Berg Ma" → suffix="Ma" + "John van der Berg Ma" → family="van der Berg Ma" + "John van der Berg Ma" → ambiguities=("suffix-or-name",) + "john van der berg ma" → suffix="ma" "John van der J. V" → family="van der J. V" · boundary - "Freiherr von Berg MA" → family="von Berg MA" + "Freiherr von Berg MA" → family="von Berg" + "Freiherr von Berg MA" → suffix="MA" "Freiherr von Richthofen V" → suffix="V" · boundary "John van der Berg née Jones" → family="van der Berg" Accepted: a particle of the unambiguous suffix vocabulary too (vd, mc) is a suffix piece to the peel, so where it opens the trailing run the chain stops before it as before any suffix word, and the peel takes it; where it continues a prefix run, - the run takes it as a particle, as P6 reads it after a comma. + the run takes it as a particle, as P6 reads it after a comma — + and there the chain reports nothing, the particle reading being + P6's fork rather than S2's. + The same carve-out covers `do`, the one word that is both a + particle and an AMBIGUOUS credential acronym: where a particle + run takes it, the word is the run's and the chain reports + nothing, whatever case it is written in — so `Anh van Do` is + silent where `Anh Do`, with no particle standing, reports. + A word that is both belongs to the particle run and to P6's + fork, not to S2's. "John Smith Mc V" → suffix="Mc V" "John van Mc" → family="van Mc" + "anh van do" → family="van do" · boundary + "anh van do" → ambiguities=() · boundary + "Anh van Do" → family="van Do" · boundary + "Anh van Do" → ambiguities=() · boundary Accepted: a caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field. @@ -503,6 +527,10 @@ P3. Rationale: connective words ("y", "of the") bind name words into and not by this rule. The two questions part at one point: a marker the pass declines and leaves as a word (M2) is a word, and counts toward the three — but its case is still not asked. + That span is not this rule's alone: wherever another rule asks + the case question — the suffix slot (S2) and the post-comma slot + (C1) — it is asked of these same words, so the answer is taken + once and read where each of them stands. "Juan y Eva Garcia" → given="Juan y Eva" "Jose E Maria Santos" → middle="E Maria" "jose e maria santos" → middle="e maria" @@ -581,7 +609,7 @@ P3. Rationale: connective words ("y", "of the") bind name words into same two words unjoined are two name words and H1 does not fire. P1's leading run is the second (#395, landed): its run takes the "Vega y Santos" join whole or stops before it. - history: decisions.md#P3 · interacts: H1, P1, M2, R3, R4 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P3 · interacts: H1, P1, M2, R3, R4 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py P4. Rationale: a particle links forward from inside a name; at the very front there is no name yet to be inside. @@ -666,10 +694,10 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "abdul Smith V" → family="Smith" "abdul Smith V" → suffix="V" "abdul Smith Jr V" → family="Smith" - "abdul Smith Jr Ma" → family="Smith" - "abdul Smith Jr Ma" → suffix="Jr Ma" - "abdul Smith Ma" → suffix="Ma" - "abdul Smith Berg Ma" → family="Berg" · boundary + "abdul Smith Jr Ma" → given="abdul Smith" + "abdul Smith Jr Ma" → middle="Jr" + "abdul Smith Ma" → given="abdul Smith" + "abdul Smith Berg Ma" → middle="Berg" · boundary "abdul Sir Smith Berg" → given="abdul Sir" "Berg, abdul van" → given="abdul van" "Berg, abdul vd" → family="vd Berg" @@ -886,11 +914,53 @@ S2. Rationale: generational suffixes and credentials are recognized abbreviation shape any word can wear and does not. A BARE ambiguous acronym is consumed only when the name has words to spare — as the second of two words it stays the family - name — and either reading carries the ambiguity flag. + name — and at the slots that report, either reading carries the + ambiguity flag. Those slots are the trailing slot of a name, the + first slot after a family comma, and the segments beyond it; the + trailing slot of the GIVEN part after a family comma is not one + of them, and takes its reading in silence (S3). + Written case is the other evidence, and it speaks only in a name + written in more than one case: there a member of the ambiguous + set written in capitals reads as the credential even with no + words to spare, and one written in any other cased form that is + not wholly lower reads as the name even with words to spare. A + name written wholly in one case says nothing about any word in + it, and the count decides alone; so does a script with no case + to write in. After a family comma this evidence is SECOND: the + count of name words before the comma decides first (C1), and the + case is read only where that count leaves the word a name. + An unlisted word joins this same ambiguous class by SHAPE where + the caller asks for it. Two or more period-separated chunks is + one such shape, admitted by default (S3); an unlisted all-caps + alphabetic word of two or more letters, standing in a suffix + position of a mixed-case name and belonging to no wordlist, is + the other, admitted only under the caller switch the example + lines below name. That second shape is OFF by default because + French and Korean records write the SURNAME in capitals, so it + is a surname as often as it is a credential and only the caller + knows which corpus this is; the cost of turning it on is that a + three-word name gives up its family name to the acronym, while a + two-word name keeps it — there are no words to spare there, so + the class is considered and declined and only the fork is + reported. "John Smith Jr." → suffix="Jr." "John Smith M.A." → suffix="M.A." "John Smith PhD" → suffix="PhD" "John Ma" → family="Ma" · boundary + "Jack MA" → suffix="MA" + "Jack Ma" → family="Ma" + "JACK MA" → family="MA" + "JOHN SMITH MA" → suffix="MA" + "John Smith Ma" → family="Ma" + "Smith, MA" → suffix="MA" + "Smith, Ma" → given="Ma" + "Doe, John MA" → middle="MA" · boundary + "Doe, John MA" → ambiguities=() · boundary + "John Smith XYZ" → family="XYZ" + "John Smith XYZ" unlisted_caps_suffixes-on → suffix="XYZ" + "Jean DUPONT" unlisted_caps_suffixes-on → family="DUPONT" + "Jean Pierre DUPONT" unlisted_caps_suffixes-on → suffix="DUPONT" + "Jean Pierre DUPONT" unlisted_caps_suffixes-on → family="Pierre" "Jack Ma." → family="Ma." · boundary "Ph. D. Van Johnson" → family="Van Johnson" "Ph. D. Van Johnson" → title="Ph." @@ -906,13 +976,16 @@ S2. Rationale: generational suffixes and credentials are recognized this decision, since the abbreviation test H2 uses is true of `Ph.` itself, so a scan that stepped over titles would step over the very piece being judged. - Accepted: with words to spare, a bare ambiguous acronym reads - as a suffix even beside an East Asian surname it more likely - belongs to; and an unambiguous suffix is consumed even when - that leaves no family name at all. - "Jack Wei Ma" → suffix="Ma" - "Jack Wei Ma" → ambiguities=("suffix-or-name",) + Accepted: an unambiguous suffix is consumed even when that + leaves no family name at all. "Smith Jr." → family="" + Accepted: the case signal costs a genuine suffix standing behind + a name-leaning acronym. The walk stops at the declined pick + rather than continuing past it, so a suffix word in front of one + is never reached and reads as a name word. + "Jack Wei Ma" → family="Ma" + "Jack Wei Ma" → ambiguities=("suffix-or-name",) + "abdul Smith Jr Ma" → middle="Jr" Accepted: the title chain no longer takes the word this rule needs, and the argument a descriptive note here asked for is made. A title run leaves one NAME word standing and a @@ -933,18 +1006,62 @@ S2. Rationale: generational suffixes and credentials are recognized and unchanged (decisions.md#v1-xfail-triage: `king` stays a title, for the addressing forms). "Dr Jr" → suffix="Jr" - interacts: H1, H2, H3, H5, C1 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py + history: decisions.md#S2 · interacts: H1, H2, H3, H5, C1, S3, P2, P5 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py S3. Rationale: credentials are often written run together with periods; the chunks between the periods are what carry the - vocabulary. + vocabulary, and a word of several chunks that no vocabulary + knows is still written the way a credential is written. A word with interior periods reads as a suffix when any of its - period-separated chunks is suffix vocabulary — any chunk, which - is looser than it sounds, since single letters can be Roman - numerals. + period-separated chunks is suffix vocabulary — except where + every chunk the vocabulary matches is a single ASCII character, + the roman numerals and the lone digit the vocabulary lists, + which are about generations rather than credentials. + A word of two or more period-separated chunks that no + vocabulary claims is read by POSITION instead, as a bare + ambiguous acronym is (S2): a credential where the name has words + to spare, a name word where it does not, either reading + reported at the slots S2 reports at, and the same at a comma — + which means the FIRST piece after a family comma and the part + before a SUFFIX comma. The part before a FAMILY comma never + reports, the comma having already named it the family; neither + does a word trailing the given part after one, where this + reading is taken silently. Case says nothing here — the + periods are the evidence — and three shapes are outside it: a + single trailing period is not this shape at all, a chunk that is + not wholly alphabetic is no acronym letter, and a word carrying + a script that writes no abbreviations is not wearing an + abbreviation's periods (H2 refuses the same word for the same + reason). This second half is a caller switch, ON by default, and + the example lines name it; turned off, such a word is name + material and the chunk rule above still decides the rest. + "John Smith Msc.Ed." → suffix="Msc.Ed." + "John Smith Msc.Ed." unlisted_dotted_suffixes-off → suffix="Msc.Ed." + "Doe, John Msc.Ed." → suffix="Msc.Ed." "John Smith J.u.n.i.o.r." → suffix="J.u.n.i.o.r." - "John Smith Q.W.E.R.T." → family="Q.W.E.R.T." · boundary - implemented: nameparser/_pipeline/_vocab.py + "John Smith Q.W.E.R.T." → suffix="Q.W.E.R.T." + "John Smith X.Y.Z." → suffix="X.Y.Z." + "john smith x.y.z." → suffix="x.y.z." + "John Smith X.Y.Z." unlisted_dotted_suffixes-off → family="X.Y.Z." + "Jack X.Y.I." → family="X.Y.I." · boundary + "John Smith Xyz." → family="Xyz." · boundary + "John Smith 1.4" → family="1.4" · boundary + "Doe, John X.Y.Z." → middle="X.Y.Z." · boundary + Accepted: the initialless-script clause carries no example line + of its own. Every input that exercises it composes a script that + writes no abbreviations with a period that only a Latin + convention writes — `John Smith 田.中.` is the shape — and a + composed form no writing system produces is tolerated input + rather than contract. A normative rule cannot hold an example of + it without putting the string into the corpus that enforces it + at released baselines, so the witnesses are the tolerated row + tests/v2/cases.py's + an_initialless_script_glued_into_periods_is_not_this_shape, + which keeps the name on the differential's radar tier, and the + CJK assertions in tests/v2/pipeline/test_vocab.py's + test_period_joined_vocab_retires_the_single_character_chunk. + W3 states the same precedence for the same reason. + history: decisions.md#S2 · interacts: S2, C1, H2, W3 · implemented: nameparser/_pipeline/_vocab.py ## Nicknames & quoted names (N) @@ -1178,8 +1295,24 @@ C1. Rationale: a credential run after the comma means the name is in the part after the first comma is entirely suffix words and more than one word precedes the comma; otherwise it reads as the listing form, the part before the comma being the family name. - Only the part after the first comma decides. Both modes consult - the vocabulary alone; by default a recognized suffix word counts + Only the part after the first comma decides. + For the ambiguous credential class — a bare acronym the + vocabulary marks as also an ordinary name, and a word admitted + to the class by shape, which S3 defines and bounds — the + count before the comma is of NAME words rather than of words: + two or more of them read the part after the comma as the + credential run, whatever case the name is written in. The count + goes FIRST and written case is asked only after it: where two + name words stand before the comma the part after it is the + credential run however that part is written, and only where the + count leaves the word a name — one name word before the comma — + is the case read, capitals in a mixed-case name making it the + credential there too (S2). A decision either way at this comma + is reported, which is the one place the comma's own decision is + reported; an attachment decided after a family comma (P6) + reports on its own. C2's comma-structure flag reports what the + parse could not recognize, not a fork it called. + By default a recognized suffix word counts even written like an initial ("V."), while strict mode vetoes initial-shaped words. In the listing form the part after the comma is still read for what it is: a part that is nothing but @@ -1233,6 +1366,18 @@ C1. Rationale: a credential run after the comma means the name is in "John Smith, Mr. Jr." → given="John" "Smith Jr., Mr." → family="Smith" · boundary "John Smith, Jones" → family="John Smith" + "John Smith, MA" → suffix="MA" + "John Smith, Ma" → suffix="Ma" + "Smith, MA" → suffix="MA" + "Smith, Ma" → given="Ma" + "John Smith, Ed" → suffix="Ed" + "Davis Royce, Ed" → suffix="Ed" + "Royce, Ed" → given="Ed" · boundary + "Smith Jr., MA" → suffix="Jr., MA" + "Smith Jr., Ma" → given="Ma" · boundary + "John Smith, A.B." → suffix="A.B." + "John Smith, A.B." unlisted_dotted_suffixes-off → given="A.B." + "Smith, A.B." → given="A.B." · boundary Accepted: a word of both the title and the unambiguous suffix vocabulary reads as the postnominal after a family comma in every spelling, the honorific's too — position decides for the @@ -1267,7 +1412,7 @@ C1. Rationale: a credential run after the comma means the name is in V` reads the suffix and `Smith, John PhD I.` continues the run, while adding a suffix comma after either turns that same letter into the middle initial. - history: decisions.md#C1 · interacts: H2, P6, W3 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py + history: decisions.md#C1 · interacts: H2, P6, W3, S2, S3 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py C2. Rationale: text beyond the recognized comma parts should be taken in without silent guessing. @@ -1276,9 +1421,29 @@ C2. Rationale: text beyond the recognized comma parts should be flagged as a structural ambiguity rather than rejected — parsing never fails on content. An empty part between doubled commas is consumed silently. + A part the parse reads as a credential run by some route other + than the suffix vocabulary is recognized and is not flagged: a + run of ambiguous acronyms whose written case leans credential + (S2), and a run every word of which is an unlisted DOTTED word + the position reads as a credential (S3). This is the one place + the ambiguous class QUIETS a report rather than adding one, and + it is narrow in two ways that the examples below pin. A member + whose written case does not lean CREDENTIAL keeps the flag, + whether the name is written in one case so that nothing leans at + all, or the member is written the way a name is written. And + S2's other by-shape half, the unlisted all-caps word, does not + reach here under its switch either: the shape a tail segment is + recognized by is the dotted one alone. "John Smith, MD, Bart" → suffix="MD, Bart" "John Smith, MD,, Jr." → suffix="MD, Jr." · boundary - history: decisions.md#C1 · implemented: nameparser/_pipeline/_segment.py + "John Smith, MD, R.A.I." → suffix="MD, R.A.I." + "John Smith, MD, R.A.I." → ambiguities=() + "John Smith, MD, R.A.I." unlisted_dotted_suffixes-off → ambiguities=("comma-structure",) + "John Smith, MD, Ma" → ambiguities=("comma-structure",) · boundary + "Steven Hardman, MD, DO, DDS" → ambiguities=() + "STEVEN HARDMAN, MD, DO, DDS" → ambiguities=("comma-structure",) · boundary + "John Smith, MD, XYZ" unlisted_caps_suffixes-on → ambiguities=("comma-structure",) + history: decisions.md#C1, decisions.md#S2 · interacts: C1, S2, S3 · implemented: nameparser/_pipeline/_segment.py ## Name order (O) @@ -1497,7 +1662,12 @@ W3. Rationale: a family name declared by a comma is the writer's name's end, and a glued honorific before the comma stays glued. The vocabulary question is C1's own, asked without C1's word-count condition: the two differ in what else they require, - not in what they ask of the words. A period the listing leaves + not in what they ask of the words — the written-case evidence + S2 reads included, so a post-comma acronym the case leans + credential declines the side it stands on here exactly as it + reads as the credential there, and the honorific comes off the + part before the comma as it does behind a settled credential. + A period the listing leaves behind is not what licenses the step past a post-nominal word, and it is not ignored either. The step is W2's, taken on the vocabulary alone and taken with no punctuation anywhere in the @@ -1521,10 +1691,11 @@ W3. Rationale: a family name declared by a comma is the writer's "남궁민수, 지훈" → family="남궁민수" · boundary "田中さん, Dr." → family="田中さん" "田中さん, PhD" → suffix="さん, PhD" + "Kim김민준씨, MA" → suffix="씨, MA" "田中さん 様." → suffix="さん 様." "김민준씨." → suffix="씨." "田中. 太郎" → family="田中." - tolerated: native CJK writing has neither a family-comma convention nor an edge full stop of any width on a name word, so the four comma lines above and the three period lines under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All seven stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD. + tolerated: native CJK writing has neither a family-comma convention nor an edge full stop of any width on a name word, so the five comma lines above and the three period lines under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All eight stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD. history: decisions.md#W3 · interacts: W1, W2, C1, H2 · implemented: nameparser/_pipeline/_script_segment.py W4. Rationale: Chinese, Japanese and Korean all write the family diff --git a/docs/release_log.rst b/docs/release_log.rst index 7b567dab..c5be2352 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -10,6 +10,14 @@ Release Log - **Fix HumanName.initials() reading a one-letter connective by vocabulary and written shape instead of by the parse.** ``HumanName("john e smith").initials()`` gives ``j. e. s.``, where every release from 1.4.0 through 2.3.0 gave ``j. s.``; ``JUAN Y GARCIA`` gives ``J. G.`` where 2.3.0 gave ``J. Y. G.``, and ``JUAN GARCIA Y LOPEZ`` gives ``J. G. L.`` where 2.3.0 gave ``J. G. Y. L.``. Those last two read 1.4.0's way at 2.3.0 and only there: 2.0.0 through 2.2.0 already gave today's answer, by the unrelated bug the 2.3.0 note below records as fixed (the facade dropping a bare capital that is also a one-letter conjunction, #462), so against those three releases neither name moves at all. The v1 facade decided whether a word was the connective by looking the word up and checking its shape, while ``parse(...).initials()`` read the tag the parse recorded -- so the change above, which reads a single letter in a one-case name from the vocabulary rather than from its case, moved one view and not the other. Both views of a parse now give the same answer. Mixed-case names are untouched on both, the writing having decided the letter: ``John E Smith`` is still ``J. E. S.`` and ``Scott E. Werner`` still ``S. E. W.``. So is a one-case name whose letter is outside the marked set -- ``maria y lopez`` is still ``m. l.``, ``y`` having joined before this release and after it. Two costs, and both match what ``capitalize()`` has always done: editing ``C.conjunctions`` after a name is parsed no longer changes its initials until ``full_name`` is assigned again, and a name restored from a pickle, copied with ``copy.copy``/``copy.deepcopy`` (the same state hooks), or built from keyword fields (``HumanName(first=..., middle=..., last=...)``) carries no tags, so its initials come from the vocabulary and can differ from a fresh parse of the same string. One private break, stated because a v1 subclass can hit it: an override of ``_process_initial`` written to v1's ``(name_part, firstname=False)`` signature now raises ``TypeError`` the first time ``initials()`` runs, since ``initials()`` passes the part's tokens. Such an override has to accept a ``tokens`` keyword *and pass it on* -- ``return super()._process_initial(name_part, firstname, tokens=tokens)`` -- to receive this fix. Widening the signature without forwarding still works, but on the pre-#528 STRING path: the token call hands the override the group's own text as ``name_part`` rather than an empty placeholder, so ``john e smith`` initials ``j. s.`` under such an override, not the ``j. e. s.`` above. A subclass overriding one of the public ``first_list``, ``middle_list`` or ``last_list`` properties keeps working too: that member takes the pre-2.4 vocabulary reading instead of the change above, while an un-overridden member still moves. See the ``R3`` entry of ``docs/design/decisions.md`` (closes #528) + - **Fix a credential acronym that is also a surname being read by position alone.** ``HumanName("Jack MA")`` gives suffix ``MA`` where 2.0 through 2.3 gave last ``MA``, and ``John Smith Ma`` gives last ``Ma`` where they gave suffix ``Ma``. In a name written in more than one case, an ambiguous acronym written in capitals is written the way a credential is written and is read as one even where removing it leaves no surname; one written in any other cased form that is not wholly lower is written the way a surname is written and stays one even where there are words to spare (``John Smith ma`` and ``John Smith ed`` -- all lower, no contrast -- give suffix ``ma``/``ed`` instead). A name written wholly in one case says nothing either way and keeps the reading it had: ``JOHN SMITH MA`` is still a credential, ``ANH DO`` still a surname, ``jack ma`` still a surname. The same reading reaches the comma forms, where the words-to-spare count is now a count of NAME words: ``Smith, MA`` gives last ``Smith``, suffix ``MA``; ``Smith Jr., MA`` keeps last ``Smith``; and ``John Smith, MA``, ``John Smith, Ed``, ``john smith, ma`` and ``JOHN SMITH, MA`` all give a suffix again, which is what 1.4.0 read and 2.0 through 2.3 did not. ``Jack Ma`` and ``Anh Do`` are unchanged. The LEAN is inert on a caseless script, but the comma count above is not -- it asks name-word count, not case -- so ``마틴 킹, MA`` and ``田中 太郎, MA`` also give a suffix again (1.4.0 parity on the suffix, two pre-comma name words each) while the single-token ``毛泽东, MA`` does not move, having no case to write a contrast in either way. See the ``S2`` entry of ``docs/design/decisions.md`` (closes #289) + + - **New Policy field unlisted_dotted_suffixes, on by default: a dotted acronym nobody has listed is read by position.** ``HumanName("John Smith X.Y.Z.")`` gives suffix ``X.Y.Z.`` where every release gave last ``X.Y.Z.``, while ``Jack X.Y.Z.`` keeps its surname, the same words-to-spare rule a listed acronym takes -- and both readings are reported. Case is irrelevant here: the periods are the signal, so ``john smith x.y.z.`` reads the same way. Words the vocabulary does know are untouched (``M.A.``, ``Ph.D.``, ``A.B.C.``), a single trailing period is still not this shape (``John Smith Xyz.`` keeps last ``Xyz.``), and a dotted run at the FRONT of a name is untouched (``J.R.R. Tolkien``). One accident retires with it: a dotted word whose only vocabulary matches were SINGLE ASCII CHARACTERS -- the roman numerals the suffix list holds, and the lone digit ``2`` -- was reading as a generational suffix, so ``Jack X.Y.I.`` gives last ``X.Y.I.`` again, as 1.4.0 read it, while ``Msc.Ed.``, ``JD.CPA`` and ``Lt.Gov.`` are unchanged. The digit is why a dotted VERSION STRING moves with them and moves SILENTLY: ``John Smith 1.4.2`` gives last ``1.4.2`` where 2.3 gave suffix ``1.4.2``, and ``John Smith, 1.4.2`` gives first ``1.4.2``, last ``John Smith``. Such a token reports nothing at any policy -- it is no acronym either, the shape reading wanting every chunk alphabetic -- and a version string read as a credential was the same accident this retirement removes. That retirement is NOT behind this switch and stands either way -- setting it to ``False`` reads an unlisted dotted word as name material by position instead (``John Smith X.Y.Z.`` keeps last ``X.Y.Z.``), the pre-2.4 reading for THAT half alone. See the ``S2`` and ``suffix-acronym-collisions`` entries of ``docs/design/decisions.md`` (closes #516) + + - **New Policy field unlisted_caps_suffixes, off by default: an opt-in reading for an unlisted all-caps credential.** It reaches the core parser only -- ``Parser(policy=Policy(unlisted_caps_suffixes=True))`` -- since the field has no v1 ``Constants`` manager. With it on, ``.parse("John Smith XYZ")`` gives given ``John``, last ``Smith``, suffix ``XYZ``, and ``.parse("John Smith, XYZ")`` gives the same three fields. It is off by default because an all-caps surname is a real writing convention that shape cannot separate from a credential: ``Jean DUPONT``, ``Minjun KIM`` and ``Jean Pierre DUPONT`` are surnames in French and Korean records, and the last of those gives given ``Jean``, last ``Pierre``, suffix ``DUPONT`` with the switch on. Off, nothing changes and nothing is reported -- 1.4.0's reading for that whole class. Neither of the two new fields reaches the v1 ``Constants`` API, as ``lenient_comma_suffixes`` does not: a ``HumanName`` tracks the parser's own DEFAULTS, so the dotted reading above (default on) reaches it while this one (default off) cannot be turned on from there. See the ``S2`` entry of ``docs/design/decisions.md`` (closes #516) + + - **The comma's own decision about an ambiguous credential is now reported.** ``parse("Smith, MA").ambiguities`` names ``suffix-or-name``, and so does every other decision at the ambiguous credential class -- before or after a comma, in either direction, with no new ``AmbiguityKind`` (the family-comma attachment fork already reported this way, e.g. ``parse("Berg, Jan vd")``). One report per decision: ``Smith, Ma`` reports that the word was kept as the given name just as ``Smith, MA`` reports that it was taken as a credential. The reading a SURNAME PARTICLE swallows is reported too, which no release before this one did: ``John van der Berg Ma`` gives last ``van der Berg Ma`` and names ``suffix-or-name``, where the chain took a word the credential reading had considered. ONE report goes away, because a comma segment the parser reads as a credential run is no longer called unrecognized: ``Steven Hardman, MD, DO, DDS`` no longer reports ``comma-structure``, on its written case. That is the whole of the losses over the differential corpora -- ``John Smith, MD, R.A.I.`` is quieted on its shape by the same change, but it never reported at 2.3.0 either, having only carried the flag inside this release's own development. The other movement an upgrader sees is a SWAP rather than a loss: ``Jack X.Y.I.`` reported ``given-or-family`` at 2.3.0 and reports ``suffix-or-name`` here, the dotted retirement above having handed it to the ambiguous class. Everything else at this class is a GAIN, which is what the rest of this bullet describes. Where a report is NOT emitted is a boundary rather than an oversight: a credential trailing the GIVEN part of a family-comma listing is read as a middle name silently, as every 2.x release has read it -- ``Doe, John MA`` gives middle ``MA`` and reports nothing. 1.4.0 read that one as a suffix, so the silence is a 2.0-era reading this release leaves where it found it, recorded open in the ``S2`` entry; the dotted sibling ``Doe, John X.Y.Z.`` is a middle name at 1.4.0 too. See the ``S2`` and ``C1`` entries of ``docs/design/decisions.md`` + **Additions** - **Add Lexicon.conjunctions_ambiguous, the one-letter connectives that read as initials.** A subset of ``conjunctions`` holding ``e`` by default; it is the knob for the change above rather than a switch. Portuguese data, where ``e`` links surnames the way ``y`` does in Spanish, takes it out: ``Lexicon.default().remove(conjunctions_ambiguous={"e"})`` restores the joining reading. Dutch data, where a bare single letter is an initial and never a connective, adds the other one: ``Lexicon.default().add(conjunctions_ambiguous={"y"})``. A v1 ``Constants`` has no manager of its own for it -- deleting the word from ``conjunctions`` is what turns the marking off, the same rule the glued-honorific tails follow. See ``docs/customize.rst`` (#383, #479) diff --git a/docs/usage.rst b/docs/usage.rst index 47f3d16e..3f9a8ffc 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -826,8 +826,15 @@ so branching on a kind needs no import: ['Van'] The post-nominals that double as ordinary surnames report the same way. -``MA`` after a full name is read as a credential, but after a single -given name it stays the surname — either way the choice is recorded: +Which reading a bare one gets depends on what the writing says. In a +name written in more than one case the word's OWN spelling is read +first, even where there is nothing to spare: capitals lean the +credential, any other cased form that is not wholly lower leans the +surname (which is why ``Jack Ma`` reads it as the surname), and the +lean wins over the count either way. Only where the spelling gives no +such signal does the words-to-spare count decide: an all-lower or +wholly one-case spelling with words to spare reads the credential. +Either way the choice is recorded: .. doctest:: @@ -835,8 +842,20 @@ given name it stays the surname — either way the choice is recorded: 'MA' >>> [a.kind.value for a in parse("John Smith MA").ambiguities] ['suffix-or-name'] - >>> parse("Jack MA").family + >>> parse("Jack MA").suffix 'MA' + >>> parse("Jack Ma").family + 'Ma' + >>> parse("John Smith Ma").family + 'Ma' + +Two ``Policy`` switches extend the same class to words the vocabulary +does not hold. ``unlisted_dotted_suffixes`` (on by default) reads a +token of two or more period-separated chunks the same way — +``parse("John Smith X.Y.Z.").suffix`` is ``'X.Y.Z.'`` — and +``unlisted_caps_suffixes`` (off by default) does the same for an +unlisted all-caps word, which is opt-in because an all-caps surname is +written that way too. See :doc:`customize` for both. A reading the vocabulary settles on its own is not a guess and reports nothing — periods make ``M.A.`` unambiguously a credential: diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 9be48608..96160213 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -1,6 +1,7 @@ """Stage: assign. -Consumes: pieces + piece_tags (grouped), segments, structure, tokens. +Consumes: pieces + piece_tags (grouped), segments, structure, tokens, +one_case. Produces: tokens with roles set on every main-stream token. Reads: Policy.name_order (#270), is_suffix_lenient on the trailing piece of a two-part comma name, and Policy.script_orders (#271, which @@ -44,10 +45,14 @@ particles_ambiguous token with more pieces following ("Van Johnson", and since #367 "Dr. Van Johnson" too, a title no longer displacing the particle out of that position) -- whatever role name_order assigns. -Emits SUFFIX_OR_NAME at three sites: the trailing roman numeral, each -ambiguous acronym the trailing peel had to resolve, and the bare-suffix +Emits SUFFIX_OR_NAME at FOUR sites: the trailing roman numeral, each +ambiguous acronym the trailing peel had to resolve, the bare-suffix carve-out where an input that is nothing but post-nominal vocabulary -gets its first word made into the name (H4's suffix half, #491). And +gets its first word made into the name (H4's suffix half, #491), and +-- since #289 -- the FAMILY-COMMA path's own read of the first +post-comma piece. Further emitters of the same kind live in +`_segment.py`, `_group.py` and `_post_rules.py`; they are not +assign's and are not counted here. And at the one site that places a LONE name word, GIVEN_OR_FAMILY for the field the convention picked (O5, #449) and TITLE_OR_NAME for the two shapes where the doubt is whether a word is a title instead (H4, @@ -69,7 +74,8 @@ tail_reading, trailing_titles, ) from nameparser._pipeline._state import ( - ParseState, PendingAmbiguity, Structure, WorkToken, _NEVER_FLIPPED, + ParseState, PendingAmbiguity, Structure, WorkToken, + _AMBIGUOUS_CREDENTIAL_TAGS, _NEVER_FLIPPED, ) from nameparser._policy import Policy, Script from nameparser._types import AmbiguityKind, Role @@ -294,7 +300,8 @@ def _assign_main(seg_idx: int, state: ParseState, # wording reads the role back, and which role "not peeled" means # depends on name_order. (The roman-numeral fork needs no such # deferral and is reported here.) - rest, titled_tail, peeled = tail_reading(rest, pieces, ptags, tokens) + rest, titled_tail, peeled = tail_reading(rest, pieces, ptags, tokens, + state.one_case) for piece_idx in titled_tail: _set_roles(tokens, pieces[piece_idx], Role.TITLE) if peeled.numeral is not None: @@ -532,7 +539,57 @@ def assign(state: ParseState) -> ParseState: # its family (the code review). reading = segment_suffix_reading( state.pieces[1], state.piece_tags[1], tokens, - state.policy.lenient_comma_suffixes) + state.policy.lenient_comma_suffixes, state.one_case) + # rules.md#C1's exception, scoped to the ambiguous credential + # class: this is the first report of the comma's OWN decision + # (listing or credential run), where the writing left the + # fork open and the comma stayed quiet by design until now. + # P6's attachment fork already reports on a family-comma path + # from post_rules, since 2.3 ("Berg, Jan vd") -- a different + # fork. Emitted on the family-comma path only -- the structure + # decision reports itself in `segment`, where that branch is + # taken, so this DECISION is never reported twice; a second ambiguous + # token elsewhere in the name is a second fork and reports on + # its own (#289, mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE). + # + # The report tracks the FORK BEING CONSULTED, not the lean -- + # exactly as the trailing slot has always done (`Jack MA` + # reported before #289 too, even where the pick was declined + # for want of words to spare). So membership alone gates it: + # a caseless script or an all-lower spelling still called this + # fork and read it positionally. + # + # The gate reads EITHER tag (`_AMBIGUOUS_CREDENTIAL_TAGS`, the + # same pair `_group`'s chain emitter asks), and the shape one + # is what reaches a by-shape member -- under EITHER 2.4 + # switch, the dotted and the caps alike, since classify writes + # it from both branches. It reaches one with the dotted switch + # OFF as well: classify writes the shape tag whether or not + # the switch admits the token to the class, which is what lets + # a declined fork be REPORTED without being taken ('Smith, + # A.B.' under `unlisted_dotted_suffixes=False` reports and + # keeps its given). An earlier wording said the class reaches + # a by-shape member "once `Policy.unlisted_dotted_suffixes` + # admits it", which is true of the CLASS and false of this + # report. + # + # Read off the FIRST post-comma piece only -- + # `segment_suffix_reading` decides piece by piece, and this is + # the one piece the lean can reach at one word before the + # comma. So `"Smith, MA PhD"` reports ONCE, for 'MA' alone: + # 'PhD' is settled vocabulary and carries neither tag, and + # even a second CLASS member there would not be read here + # (test_assign.py asserts the count). + if state.pieces[1] and len(state.pieces[1][0]) == 1: + i = state.pieces[1][0][0] + if not tokens[i].tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS): + chose = ("a credential" if reading and reading[0] + else "the given name") + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{tokens[i].text!r} after the comma is also an " + f"ordinary name word; read as {chose}", + (i,))) # Segment 1 is read FIRST, ahead of either branch below. It # consumes `reading`, piece tags and text only -- nothing # segment 0's read writes -- and running it first is what puts diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 12498f53..27e6d7a8 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -2,20 +2,31 @@ Consumes: tokens, comma_offsets (with token roles, the two halves of the structural-boundary test the marker pass applies -- see -_tag_marker_runs). +_vocab.tag_marker_runs), and one_case where an earlier stage recorded +it -- segment writes it lazily where a comma form can turn it on +(#289/#516), so this read is the fallback for every other name. Produces: tokens with vocabulary tags added (text/span/role unchanged), -plus ambiguities (SUFFIX_OR_NICKNAME, CONJUNCTION_OR_INITIAL). +plus ambiguities (SUFFIX_OR_NICKNAME, CONJUNCTION_OR_INITIAL) and +one_case -- whether the name's own words are written in one case, +recorded for the later stages that read it (#289/#516) and left alone +where an earlier stage already asked. Reads: every Lexicon vocabulary field except surnames and -honorific_tails, which script_segment consumes upstream; no Policy -FIELD is consulted (is_initial does consult the _policy module's -_NO_INITIALS constant, which is not configuration -- nothing here -varies by Policy value). +honorific_tails, which script_segment consumes upstream; and, since +2.4, Policy.unlisted_dotted_suffixes and Policy.unlisted_caps_suffixes, +which decide whether an UNLISTED dotted or all-caps token joins the +ambiguous credential class by SHAPE (#516). is_initial also consults +the _policy module's _NO_INITIALS constant. It is named apart from the +fields above because it is not CONFIGURATION -- no Lexicon or Policy +carries it and no caller can change it -- and not because it decides +nothing: the tags this stage writes do vary by it ('씨.' is not tagged +`initial`, which is the whole of #320). Tags emitted -- stable (API): "particle", "conjunction", "initial"; namespaced (unstable): "vocab:title", "vocab:given-title", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", "vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", -"vocab:maiden-marker-cont". +"vocab:maiden-marker-cont"; and, in a namespace of its own, +"shape:acronym". "vocab:maiden-marker" tags the HEAD of a maiden marker, which is a whole marker whenever the marker is one word; the continuation tag carries the rest of a PHRASE marker ("z domu"), so a site asking @@ -29,21 +40,28 @@ words-to-spare guard) and its Accepted consequences. The initial veto is assign's job, not classify's: 'V' carries both "vocab:suffix" and "initial". +"shape:acronym" is the one tag in the shape: namespace and it records +WHERE a class claim came from rather than what the vocabulary holds: +an unlisted token the writing makes credential-shaped. It rides +beside "vocab:suffix-ambiguous" where a Policy switch admits the +token to that class, and stands alone where the switch is off, which +is what lets the fork be reported without being taken. """ from __future__ import annotations import dataclasses -from collections.abc import Sequence from nameparser._lexicon import _normalize from nameparser._pipeline._state import ( - ParseState, PendingAmbiguity, WorkToken, comma_bucket, + AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG, ParseState, PendingAmbiguity, + WorkToken, ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( - _longest_marker, is_initial, is_one_case, maiden_marker_head, - maiden_marker_run, period_joined_vocab, suffix_as_written, + caps_shape_candidate, is_initial, is_one_case, period_joined_vocab, + suffix_as_written, tag_marker_runs, ) +from nameparser._pipeline._pieces import own_words @@ -56,19 +74,34 @@ # bare ambiguous acronym is consumed only when the name has words to # spare" def _tags_for(token: WorkToken, n: str, state: ParseState, - marker_tag: str | None, one_case_own: bool) -> frozenset[str]: + marker_tag: str | None, one_case_own: bool, + one_case: bool) -> frozenset[str]: """`n` is _normalize(token.text), folded once by the caller and shared with the marker pass; `marker_tag` is what that pass decided for this token, or None. The marker DECISION is entirely - _tag_marker_runs'; only the writing happens here, so the two tokens - of a phrase are built once rather than replaced twice. + `_vocab.tag_marker_runs`'; only the writing happens here, so the + two tokens of a phrase are built once rather than replaced twice. `one_case_own` is true when the name's OWN words are written in one case AND this token is one of the name's own words -- a maiden clause and any delimited (nickname) content are not, so the fork never reads them either (rules.md#P3): a clause's words are not the name's own words, and appending one must not change how THIS token - reads.""" + reads. + + `one_case` is the bare NAME-level fact alone -- P3's own-words + span is not this question's business. #516's caps branch reads + THIS, not `one_case_own`: a maiden clause's own words are outside + `one_case_own`'s span by construction (`i < clause_at` fails for + every one of them), so a token past the clause cut reads + `one_case_own` as False regardless of whether the WHOLE name is + written in one case -- 'JOHN SMITH NEE' flipped 'NEE' to a + credential reading with the switch on, one case and all, because + `not one_case_own` was true for it purely from being past the + clause cut, never from the name's own writing (#516 review round, + a second reviewer's finding). `single_letter_connective` below + keeps `one_case_own`: that fork is genuinely about the OWN-WORDS + span, and reads a clause's word as no evidence on purpose.""" lex = state.lexicon tags = set(token.tags) if marker_tag is not None: @@ -82,7 +115,7 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, if n in lex.suffix_words: tags.add("vocab:suffix-word") if n in lex.suffix_acronyms_ambiguous: - tags.add("vocab:suffix-ambiguous") + tags.add(AMBIGUOUS_ACRONYM_TAG) if n in lex.particles: tags.add("particle") if n in lex.particles_ambiguous: @@ -122,160 +155,109 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, tags.add("vocab:bound-given") # maiden markers are NOT tagged here: an entry may be a phrase whose # words are not markers on their own, and this function sees one - # token with no neighbours. _tag_marker_runs below does the whole + # token with no neighbours. `_vocab.tag_marker_runs` does the whole # field, single words included, so there is one place that decides # it (mechanisms.md#ONE-PREDICATE-PER-QUESTION). - # v1's period-joined derivation (parse_pieces): a token with a - # period not at the end, ANY of whose period chunks is a title, is - # a title as a whole ('Lt.Gov.', and by the ANY rule 'Mr.Smith'); + # The block a WHOLE-TOKEN vocabulary match skips, and four + # readings inside it, in precedence order. The first two are v1's + # period-joined derivation (parse_pieces): a token with a period + # not at the end, ANY of whose period chunks is a title, is a + # title as a whole ('Lt.Gov.', and by the ANY rule 'Mr.Smith'); # else ANY suffix chunk makes it a suffix ('JD.CPA'). Title wins - # (v1's continue). Skipped when the whole token already matched. + # (v1's continue). The third and fourth are 2.4's by-shape halves + # (#516) and reach only a token NO vocabulary claimed: a dotted + # token of two or more alphabetic chunks, and -- where its switch + # is on -- an unlisted all-caps word. They are an `elif` chain + # with the dotted one first, so a dotted token never reaches the + # caps test and the two shapes stay disjoint. + # + # Both by-shape branches carry SHAPE_ACRONYM_TAG BESIDE the + # membership tag rather than instead of it: `vocab:` records + # membership (the roster above) and the peel reads membership, + # while the shape tag records that the claim came from the + # WRITING. The dotted branch writes it even with its switch off, + # which is what lets a declined fork be reported without being + # taken. `role is None` guards both: a token already carrying a + # role is delimited content, decided by extract's escape and never + # at the trailing slot -- 'Bridge (A.B)' is the control that + # proves the guard load-bearing (without it the nickname reading + # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report below), and + # 'Bridge (1.4)' cannot exercise it, a digit chunk never reaching + # the shape verdict at all (period_joined_vocab's alphabetic + # gate). if "vocab:title" not in tags and "vocab:suffix" not in tags: derived = period_joined_vocab(token.text, lex) if derived == "title": tags.add("vocab:title") elif derived == "suffix": tags.add("vocab:suffix") + elif (derived == "shape" and token.role is None + and n not in lex.suffix_acronyms_ambiguous): + # This branch and `_vocab.ambiguous_class_candidate` ask + # the SAME question twice, of necessity -- `segment` runs + # before `classify` and has no tags to read yet -- kept + # from drifting by `test_classify. + # test_ambiguous_class_candidate_agrees_with_the_tag` + # rather than by this sentence alone. + # + # `n not in suffix_acronyms_ambiguous` is the third guard, + # and it is about a LISTED member rather than a role: a + # caller may list a dotted entry ('a.b'), which the whole- + # token membership test above matches while + # `suffix_as_written`'s period-free acronym lookup ('ab') + # misses, so the chunk view reached here and called the + # word by-shape -- silencing `_pieces.listed_lean`, which + # declines wherever SHAPE_ACRONYM_TAG rides, and costing + # the caller's own listing its case lean ('Jack A.B.' read + # family where 'Jack MA' reads suffix). One frozenset + # lookup on a branch only a dotted token reaches; the + # shipped ambiguous vocabulary carries no periods, so + # nothing default changes. + tags.add(SHAPE_ACRONYM_TAG) + if state.policy.unlisted_dotted_suffixes: + tags.add(AMBIGUOUS_ACRONYM_TAG) + elif (state.policy.unlisted_caps_suffixes and token.role is None + and caps_shape_candidate(token.text, lex, state.policy, + one_case)): + # #516's all-caps half, OPT-IN: an unlisted word written + # in capitals inside a mixed-case name. The policy conjunct + # comes FIRST and stays a plain attribute read -- False by + # default, so `caps_shape_candidate` is never CALLED at the + # default and sharing its body costs the default nothing + # (that is why this half is a call where the dotted branch + # above stays inline: the dotted caller has no such cheap + # first conjunct to hide behind). The predicate's own + # docstring carries the whole-vocabulary roster and what + # each measured entry would have cost unfixed. + tags.add(SHAPE_ACRONYM_TAG) + tags.add(AMBIGUOUS_ACRONYM_TAG) return frozenset(tags) -def _tag_marker_runs(state: ParseState, - folded: Sequence[str]) -> dict[int, str]: - """Which tokens are maiden marker runs: index -> "vocab:maiden-marker" - for a run's head, "vocab:maiden-marker-cont" for the rest. - - Returns the decision rather than rewriting the tokens; classify - writes it into the one pass that builds them, so a marker token is - not replaced twice. `folded` is _normalize per token, computed once - for this pass and the vocabulary tags alike. - - The one sequence pass in this stage, and it has to be one: a marker - entry may be a PHRASE whose words are not markers individually - ('z', 'domu'), so no per-token membership test can find it. - Left to right, longest first at each position, then skip past what - the run claimed -- a second marker cannot start inside the first. - - This is where the tag is DECIDED for the two stages that read it - afterwards. group runs later and asks its questions of these tags - rather than re-deriving the run (the recorded-answer half of - mechanisms.md#ONE-PREDICATE-PER-QUESTION); extract runs EARLIER, - before tokens exist, so it calls the predicate itself over the - clause's whitespace words. - - A tagged run is structurally contiguous, and the test is - one-directional: a role change IS a clause edge, so no run spans - one, but not every clause edge is a role change -- two ADJACENT - clauses of the same role are indistinguishable here, and - 'Jane (z) (domu) Jones' does tag a run across them. Both consumers - refuse that run for reasons of their own (the piece walk never sees - role-bearing tokens at all; the clause drop is scoped to one - clause's span), so no reading depends on it today, and the claim - this pass can honestly make is the weaker one. What it does - guarantee is what _group._marker_run_pieces needs: a run inside the - MAIN stream stays inside one segment. Without it this pass walked - the whole span-sorted stream while group walked one segment -- - _segment keeps only role-less tokens and buckets them by the commas - before them -- so a run half inside a bracketed clause was tagged - whole and consumed as a proper PREFIX of itself, and - 'Anna z (domu) Nowak' read family 'Anna', maiden 'Nowak': the bare - preposition eating the name, which is the exact damage the phrase - entry exists to prevent. Refusing to tag such a run is the fix; - truncating it instead would hand M2 the same wrong prefix one word - shorter. - """ - markers = state.lexicon.maiden_markers - # the lookahead the vocabulary actually needs; 0 for an empty set, - # which skips the pass entirely - cap = _longest_marker(markers) - if not cap: - return {} - tokens = state.tokens - n_tokens = len(tokens) - # Deferred, not computed up front: only the contiguity walk reads - # it, only a phrase vocabulary runs that walk, and only at a token - # that opens an entry -- so a single-word vocabulary, and a - # phrase vocabulary over a name holding no marker, never pay the - # sweep at all. - buckets: list[int] | None = None - tags: dict[int, str] = {} - i = 0 - while i < n_tokens: - # The predicate's own head test first, over the fold the caller - # already has: almost no token opens any entry, and for those - # there is nothing to assemble. Same function maiden_marker_run - # consults, so a token skipped here is one it would refuse. - if not maiden_marker_head(folded[i], markers): - i += 1 - continue - # Bound the lookahead at the first structural boundary, so the - # predicate is asked over the words that could form one run and - # answers longest-first WITHIN them -- a two-word entry refused - # at a clause edge still leaves a one-word entry starting there - # free to match. - limit = 1 - if cap > 1: - if buckets is None: - buckets = [comma_bucket(t.span.start, state.comma_offsets) - for t in tokens] - role, bucket = tokens[i].role, buckets[i] - while (limit < cap and i + limit < n_tokens - and tokens[i + limit].role is role - and buckets[i + limit] == bucket): - limit += 1 - run = maiden_marker_run( - [tokens[k].text for k in range(i, i + limit)], markers) - if not run: - i += 1 - continue - tags[i] = "vocab:maiden-marker" - for k in range(i + 1, i + run): - tags[k] = "vocab:maiden-marker-cont" - i += run - return tags - - def classify(state: ParseState) -> ParseState: # One fold per token, shared by the marker pass and the vocabulary # tags -- the shape suffix_as_written already asks for ("n is # _normalize(text), passed in so callers normalize once"). folded = [_normalize(t.text) for t in state.tokens] - marker_tags = _tag_marker_runs(state, folded) + marker_tags = tag_marker_runs(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, folded) # rules.md#P3 says a maiden marker, taken as one, and the words it - # takes, are not among the name's own words -- so clause_at is the - # smallest index tagged as a marker HEAD, and everything from there - # on is the clause. A plain loop, not a generator handed to min(): - # marker_tags is almost always empty, and its keys arrive in index - # order (_tag_marker_runs walks left to right), so the first head a - # forward walk finds is already the smallest. Filtering on the HEAD - # tag specifically (never "-cont") is what makes that answer right - # independent of _tag_marker_runs's insertion order too: every - # matching entry is a clause start, so a min() over them in any - # order would agree with this walk -- the walk just takes the - # cheaper path given the order this dict happens to arrive in. - clause_at = len(state.tokens) - for i, tag in marker_tags.items(): - # A marker word already carrying a role arrived pre-set by - # extract (WorkToken.role's docstring) -- it is the CLAUSE's - # word, not a bare one opening a new clause, so it must not - # move clause_at: a delimited/maiden clause's own marker - # content is excluded from "own" by its role already, and - # letting it also set clause_at truncates the OWN words that - # follow the clause ("JUAN (NEE JONES) GARCIA Y LOPEZ"'s - # trailing "GARCIA Y LOPEZ" is such own text). - if tag == "vocab:maiden-marker" and state.tokens[i].role is None: - clause_at = i - break - # ONE fact per parse, taken over the name's OWN words (rules.md#P3): - # not a delimited clause's tokens, which arrive with `role` already - # set by extract (WorkToken.role's docstring), and not the maiden - # clause itself, which starts at clause_at. Appending a clause must - # not flip the reading of words that did not change. Not stored on - # ParseState: nothing downstream reads it today, and #289/#516 can - # promote it the way `order` was recorded rather than recomputed. - # is_one_case's own `Sequence` parameter is where the frame-cost - # argument for handing it a built list lives (_vocab.py, #475). - own = [t.text for t in state.tokens[:clause_at] if t.role is None] - one_case = is_one_case(own) + # takes, are not among the name's own words -- so the span and its + # clause cut are _pieces.own_words', shared with the site that + # needs the same answer two stages earlier (#289/#516). The marker + # map goes with it: this stage has already decided which tokens + # are run HEADS, so the helper reads that decision rather than + # walking the texts again, and classify's answer is the one it was + # before the helper existed. + own, clause_at = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, marker_tags) + # ONE fact per parse, and it is recorded now (ParseState.one_case): + # segment writes it first where a comma form could turn on it, and + # a fact two stages decide apart is what recording it prevents + # (decisions.md#S2). + one_case = state.one_case + if one_case is None: + one_case = is_one_case(own) # The fork itself must not read a clause's words either, so the # `one_case and ...` argument below repeats `own`'s membership test # per token, and the fork and its emitter then agree with the case @@ -284,8 +266,8 @@ def classify(state: ParseState) -> ParseState: tokens = tuple( dataclasses.replace( t, tags=_tags_for(t, folded[i], state, marker_tags.get(i), - one_case and i < clause_at - and t.role is None)) + one_case_own=one_case and i < clause_at + and t.role is None, one_case=one_case)) for i, t in enumerate(state.tokens)) # Delimited content whose vocabulary cannot settle it: extract's # escape sends an UNambiguous suffix straight through ("(MBA)" -> @@ -296,7 +278,7 @@ def classify(state: ParseState) -> ParseState: ambiguities = list(state.ambiguities) for i, token in enumerate(tokens): if (token.role is Role.NICKNAME - and "vocab:suffix-ambiguous" in token.tags): + and AMBIGUOUS_ACRONYM_TAG in token.tags): ambiguities.append(PendingAmbiguity( AmbiguityKind.SUFFIX_OR_NICKNAME, f"delimited {token.text!r} is also a post-nominal; read " @@ -340,5 +322,8 @@ def classify(state: ParseState) -> ParseState: f"the name is written in one case, so nothing marks " f"which, and it is read as an initial", (i,))) + # The write rides the replace this stage already makes, so + # recording the fact costs no frame of its own. return dataclasses.replace(state, tokens=tokens, - ambiguities=tuple(ambiguities)) + ambiguities=tuple(ambiguities), + one_case=one_case) diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index f1a62b0d..8d2ce4c0 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1,7 +1,7 @@ """Stage: group. -Consumes: tokens (classified), segments, structure, extracted (the -role + inner span per delimited region, for the #329 pass below -- +Consumes: tokens (classified), segments, structure, one_case, extracted +(the role + inner span per delimited region, for the #329 pass below -- the only stage after tokenize that reads it). Produces: pieces + piece_tags per segment (runs of token indices -- tokens are NEVER joined into strings: the anti-#100 invariant); maiden @@ -49,6 +49,7 @@ ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, + _AMBIGUOUS_CREDENTIAL_TAGS, ) from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import delimiter_cores @@ -179,7 +180,7 @@ def _marker_run_pieces(seen: Sequence[int], pieces: Sequence[Sequence[int]], and no more: a run of ROLE-LESS tokens stays inside one segment. It can also tag a run across two adjacent clauses of the same role, which this walk never sees because a role-bearing token is in no - segment at all -- see _tag_marker_runs, which states the limit. + segment at all -- see _vocab.tag_marker_runs, which states the limit. """ return marker_run_length( tokens[pieces[seen[k]][0]].tags for k in range(m + 1, len(seen))) @@ -188,7 +189,8 @@ def _marker_run_pieces(seen: Sequence[int], pieces: Sequence[Sequence[int]], def _maiden_take(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], - cores: Set[str]) -> tuple[list[int], list[int]] | None: + cores: Set[str], + one_case: bool | None) -> tuple[list[int], list[int]] | None: """The piece indices the marker pass removes, split the way MaidenTake declares them: the MARKER's pieces (one, or several for a phrase entry like 'z domu') and the maiden name's. None when the @@ -236,9 +238,32 @@ def _maiden_take(pieces: Sequence[Sequence[int]], # the fork wants, and 'Jane Smith née V' declines like 'Jane Smith # née PhD' -- nothing after the marker but a suffix, so the marker # stays a word -- as 1.4.0 read it. + # + # `one_case` is passed here and at the re-ask below and changes + # NOTHING, by construction: `numeral_only` answers off + # `peeled.numeral`, and the numeral fork is decided before the peel + # ever reads a lean -- the fact reaches only the bare-acronym fork, + # which this reading discards. Measured anyway, 2026-09-18, because + # "by construction" is the claim this repository gets wrong most + # often: dropping the argument at these TWO sites moves 0 of 9,852 + # parses (1,642 names -- the distinct union of every + # `tools/differential/corpus*.jsonl` entry, every `cases.py` text, + # and `tests/test_variations.TEST_NAMES` with its comma + # permutations -- under six policies: the default, both + # family-first orders, strict commas, and each 2.4 switch flipped; + # recompute recipe in decisions.md#S2). It stays passed rather + # than spelled `None` because `None` is a different statement -- + # "nobody asked" -- and a future numeral fork that DID read the + # writing would then be wrong silently. + # + # The chain-tail measure below (`tail`, and the re-peel after the + # chain) is the opposite, and the same sweep says so: dropping it + # there moves 18 of the 9,852, on 'John van der Berg Ma', 'John de + # Ma' and 'Freiherr von Berg MA' under every one of the six. A + # review round called all three sites inert together; two are. skip = frozenset(range(len(pieces))) - frozenset(seen) trailing = trailing_start(seen[m], pieces, ptags, tokens, skip, - numeral_only=True) + numeral_only=True, one_case=one_case) # The fork reads the piece before the numeral, and the take # REMOVES that piece: afterwards assign sees the piece before the # marker there, and if that is initial-shaped the fork will not @@ -256,7 +281,8 @@ def _maiden_take(pieces: Sequence[Sequence[int]], view_tags = [ptags[i] for i in left] if trailing_start(leading_titles(view, view_tags, tokens), view, view_tags, tokens, - numeral_only=True) == len(view): + numeral_only=True, + one_case=one_case) == len(view): trailing = len(pieces) j = m + run while (j < len(seen) and seen[j] < trailing @@ -300,6 +326,8 @@ def _group_segment(seg: tuple[int, ...], additional: int, cores: Set[str] = frozenset(), given_name_titles: Set[str] = frozenset(), opens_the_name: bool = False, + *, + one_case: bool | None, ) -> tuple[list[Piece], list[set[str]], MaidenTake | None]: pieces: list[Piece] = [[i] for i in seg] ptags: list[set[str]] = [set() for _ in seg] @@ -440,7 +468,7 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # The tokens are not touched here: this function reads them and # returns what it took, and group() records the drop and the roles. taken: MaidenTake | None = None - take = _maiden_take(pieces, ptags, tokens, cores) + take = _maiden_take(pieces, ptags, tokens, cores, one_case) if take is not None: marker_ks, maiden_ks = take taken = ([i for k in marker_ks for i in pieces[k]], @@ -570,7 +598,7 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # the acronym still has the pieces the fork counted (below). name_start = leading_titles(pieces, ptags, tokens) tail = len(pieces) - trailing_start(name_start, pieces, ptags, - tokens) + tokens, one_case=one_case) def chain(tail: int) -> None: k = 0 while k < len(pieces): @@ -643,6 +671,75 @@ def chain(tail: int) -> None: f"name piece; it is also a given name in other " f"names", (i,))) + # rules.md#S2: "A BARE ambiguous acronym is consumed + # only when the name has words to spare — as the second + # of two words it stays the family name — and at the + # slots that report, either reading carries the + # ambiguity flag" + # + # The other half of SUFFIX_OR_NAME's declined branch, + # and it is here for the reason PARTICLE_OR_GIVEN's + # second emitter is: a fork whose branches are taken in + # different stages needs an emitter in each + # (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE). + # `assign` reports from `peel_trailing`'s picks, and a + # pick reaches it only as a LONE piece -- the peel's own + # `len(piece) == 1` test -- so the moment this chain + # takes the acronym into the particle run, the token + # assign would have reported on no longer exists as a + # piece and NOBODY reports. Measured: 'John van der Berg + # Ma', 'John de Ma', 'Dr. John van Smith Ma', 'John van + # Smith Ma Jr.' and 'John Smith Mc Ma' each lost the + # report #289's own lean had just made true of them, + # while 'John Smith Ma' -- the same fork with no + # particle to chain -- kept it (review round). + # + # The DECLINED reading is what this reports, because + # this branch only runs over pieces the peel left + # standing: `tail` is the peel's own answer, the inner + # scan stops at `len(pieces) - tail`, and a piece the + # peel TOOK is behind that bound. Where the chain runs + # again with a smaller tail (the re-peel below), the + # first pass's appends are truncated with the pieces, + # so the surviving report is the surviving reading's. + # + # `j > k + 1` above is this test's floor too: a merge + # that folds a piece into itself chained nothing. + # Written inline against the tags and with no new walk + # -- `pieces[j - 1]` is the last piece the merge is + # about to claim, one index and one frozenset test -- + # so the ordinary chained name ('de la Vega') pays no + # frame for it. + # + # `not prefix(j - 1)` is the other floor, and it names + # WHICH of the two scans above claimed the piece. The + # first extends the PARTICLE run and the second takes + # name words up to the trailing suffix; only the second + # is taking a word the peel had looked at. A word in + # both vocabularies ('do', 'mc', 'vd') ends a particle + # run as a particle, which is P4's reading and P6's + # fork, not this one -- 'anh van do' has read family + # 'van do' silently since 1.4.0 and its case row says + # so. It is tested LAST, and measured: `prefix` is a + # closure over `_is_prefix_piece`, so asking it is TWO + # frames, and asking it ahead of the tag test moved the + # reference name from 412 to 414 -- every chained name + # in the library paying for a question only an + # ambiguous acronym can make interesting. Behind the + # `isdisjoint` (a C call, no frame) almost nothing + # reaches it. + last = pieces[j - 1] + if (j > k + 1 and len(last) == 1 + and not tokens[last[0]].tags.isdisjoint( + _AMBIGUOUS_CREDENTIAL_TAGS) + and not prefix(j - 1)): + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{tokens[last[0]].text!r} is both a post-nominal " + f"and an ordinary name; the particle chain took " + f"it into the name rather than reading it as a " + f"post-nominal", + tuple(last))) merge(k, j, drop={"prefix"}) k += 1 @@ -668,7 +765,7 @@ def chain(tail: int) -> None: chain(tail) left = len(pieces) - trailing_start( leading_titles(pieces, ptags, tokens), pieces, ptags, - tokens) + tokens, one_case=one_case) if left < tail: pieces[:], ptags[:] = kept[0], kept[1] del ambiguities[kept[2]:] @@ -735,13 +832,14 @@ def chain(tail: int) -> None: # unjoined and keeps it joined. Shapes pinned in # test_group.py. rest, chain_took, before = tail_reading( - peel_walk(fk, ptags), pieces, ptags, tokens) + peel_walk(fk, ptags), pieces, ptags, tokens, one_case) view, view_tags = list(pieces), list(ptags) view[fk:fk + 2] = [pieces[fk] + pieces[fk + 1]] view_tags[fk:fk + 2] = [joined_tags(fk, fk + 2, drop={"title"})] view_rest, _, after = tail_reading( - peel_walk(fk, view_tags), view, view_tags, tokens) + peel_walk(fk, view_tags), view, view_tags, tokens, + one_case) same_suffixes = ( [tuple(view[j]) for j in view_rest[after.names:]] == [tuple(pieces[j]) for j in rest[before.names:]]) @@ -834,7 +932,8 @@ def group(state: ParseState) -> ParseState: None if family_comma else ambiguities, seg_cores, state.lexicon.given_name_titles, - opens_the_name=(seg_idx == 0 and not family_comma)) + opens_the_name=(seg_idx == 0 and not family_comma), + one_case=state.one_case) # the marker is dropped and the maiden name's tokens become # MAIDEN (#274); which pieces those are was settled in # _group_segment, before the joins diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 9af23707..ad38208a 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -7,7 +7,11 @@ which stage happens to ask (mechanisms.md#ONE-PREDICATE-PER-QUESTION). _vocab points here from its own side: "Text-level tests used by more than one stage; piece-level ones live in _pieces, the sibling layer -over tokens-plus-tags." +over tokens-plus-tags." own_words (#289/#516) answers over the whole +token STREAM rather than one piece -- pieces do not exist yet at the +stages that call it -- but the question is still piece-shaped, not +word-shaped: it reads token ROLE, which _vocab's text-level tests +never take. Before this module those predicates lived in _group, not because grouping owned them but because assign imported group and could not be @@ -34,8 +38,10 @@ their fixed point for the two stages that must not disagree about the answer. -Layering: imports _state and _vocab only; _group and _assign import -it, and neither of the two it imports imports it back. +Layering: imports _state and _vocab only; FOUR stages import it -- +_segment, _classify, _group and _assign, segment being the one the +#289/#516 own-words span added -- and neither of the two it imports +imports it back. Naming follows _vocab's: inside an already-private module the leading underscore marks module-PRIVATE, so the names other stages call are @@ -46,16 +52,82 @@ """ from __future__ import annotations -import re -from collections.abc import Sequence, Set +from collections.abc import Mapping, Sequence, Set from typing import NamedTuple -from nameparser._pipeline._state import WorkToken +from nameparser._pipeline._state import ( + AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG, WorkToken, +) from nameparser._pipeline._vocab import ( - in_initialless_script, is_trailing_numeral_suffix, + _PERIOD_ABBREV, Lean, ambiguous_lean, in_initialless_script, + is_trailing_numeral_suffix, tag_marker_runs, ) +# rules.md#P3: "both questions this rule asks of a name — how many +# words it has, and whether it is written in one case — are asked of +# the name's OWN words: a maiden marker taken as one, and the words it +# takes (M2), are not among them, and neither is a delimited clause +# (N1, M1)" (history: decisions.md#P3) +def own_words(tokens: Sequence[WorkToken], comma_offsets: Sequence[int], + markers: frozenset[str], + marker_tags: Mapping[int, str] | None = None, + ) -> tuple[list[str], int]: + """The name's OWN word texts and the index the maiden clause + starts at -- one span for the two stages that ask about it + (#289/#516). + + Own words are the role-less tokens before the clause: a delimited + clause's tokens arrive from extract with a role already set, and + everything from a maiden marker on is the clause. Appending a + clause to a name must not change how a word in the name reads. + + `marker_tags` is the map `_vocab.tag_marker_runs` already built, + index -> "vocab:maiden-marker"/"...-cont"; a caller that has it + (classify) hands it over and pays no second walk. A caller that + runs BEFORE those tags exist (segment) omits it, and this + function calls `tag_marker_runs` itself to build the SAME map + classify would -- not an approximation of it, which is what a + from-scratch text walk (this module's earlier `first_marker_head`) + could disagree with on a name where a marker's head opens an entry + but no run completes ('z' of 'z domu'): measured, 'ANNA z Nowak, + MD' flipped the recorded one-case verdict under that approximation + (decisions.md#P3). Sharing the exact function instead makes the + two paths agree by construction, not by corpus luck. + + Takes tokens/comma_offsets/markers rather than a whole ParseState: + both call sites have all three already, and passing them lets this + function sit beside the piece predicates rather than in _vocab + (mechanisms.md#ONE-PREDICATE-PER-QUESTION; the + _post_rules.suffix_entries precedent, AGENTS.md's named exception) + -- it answers with the SPAN, where `tag_marker_runs` answers only + which tokens open a run. + + `marker_tags`' keys must arrive in index order for the walk below + to find the SMALLEST head in one pass: `tag_marker_runs` walks its + tokens left to right, so the first head it records is already the + smallest, and a caller building its own map must preserve that + order too. + + A plain tuple, not a NamedTuple: measured 2026-09-17, wrapping + this in a `NamedTuple` (this module's `Peel` is one) cost the + reference name one more frame (413.00 vs the 412.00 band this + commit must hold) -- a NamedTuple's `__new__` is itself a call, + where a bare tuple literal is not. `Peel` can afford the frame + because assign builds one only where the trailing peel actually + ran; `own_words` returns on every parse. + """ + if marker_tags is None: + marker_tags = tag_marker_runs(tokens, comma_offsets, markers) + clause_at = len(tokens) + for i, tag in marker_tags.items(): + if tag == "vocab:maiden-marker" and tokens[i].role is None: + clause_at = i + break + return ([t.text for t in tokens[:clause_at] if t.role is None], + clause_at) + + # rules.md#H3: "successive title words at the start of the part # carrying the given name chain into one title; a title word # elsewhere in the name does not" @@ -66,13 +138,16 @@ def is_title_piece(piece: Sequence[int], ptags: Set[str], return len(piece) == 1 and "vocab:title" in tokens[piece[0]].tags -# Ported verbatim from v1 (nameparser/config/regexes.py -# "period_abbreviation") -- layering forbids the config import; keep -# in sync by hand (tests/v2/test_regex_sync.py). Out of assign since -# #424 and in the piece layer since #439: the test is assign's, and group's -# leading-particle scan and trailing-run walk must start where assign -# starts. -_PERIOD_ABBREV = re.compile(r'^[^\W\d_]{2,}\.$') +# _PERIOD_ABBREV: imported from _vocab, not redefined here (#289/#516, +# quality-review finding) -- _vocab.name_word_count needed the SAME +# shape test is_leading_title asks (_vocab.is_title_shaped), and +# layering only allows the move in that direction (_pieces may import +# _vocab; _vocab may not import _pieces). tests/v2/test_regex_sync.py +# still reaches it as `_pieces._PERIOD_ABBREV` -- an import binds the +# same name here, so the sync test's target did not move. Out of +# assign since #424 and in the piece layer since #439: the test is +# assign's, and group's leading-particle scan and trailing-run walk +# must start where assign starts. # rules.md#H2: "an abbreviation opening the part of the name that @@ -86,17 +161,24 @@ def is_leading_title(piece: Sequence[int], ptags: Set[str], if len(piece) != 1: return False text = tokens[piece[0]].text - # The shape reads a Latin convention: a period marks an - # abbreviation. Scripts with no initials have no period - # abbreviations either (_policy._NO_INITIALS, the #320 veto - # is_initial carries), so a CJK word wearing a period is a name - # word, not a title -- a lone '田中.' is the family name (#323). - # _PERIOD_ABBREV stays ASCII-period only: a word wearing '。' never - # matched it, and the veto is what makes the ASCII spelling agree. - # ASCII text can carry no _NO_INITIALS character (every range sits - # above U+3000), so the C-level test declines before the regex - # search runs -- four frames per unlisted-abbreviation opener per - # parse, is_leading_title running four times per piece. + # INLINED rather than calling _vocab.is_title_shaped, which asks + # the exact same question (#289/#516, quality-review finding: the + # two must not drift, and did once -- name_word_count's own + # vocabulary-only title test read 'Xyz.' as a name word where this + # predicate reads it as a title, and the disagreement flipped a + # comma structure `Dr. Smith, Ed`'s LISTED spelling did not). + # THIS is the one home for the number, `is_title_shaped` pointing + # here rather than restating it: routing this hot path through the + # shared function costs one frame per call (`is_leading_title` + # runs on every leading piece of every parse, unlike + # name_word_count's comma-only path), moving the reference name + # from 412/449 to 417/454 -- five calls on `Dr. Juan de la Vega + # III`, recomputable with `uv run python + # tools/perf/call_count.py`. Kept as two spellings of ONE test + # instead -- if you touch one, touch both, and + # `test_is_title_shaped_and_is_leading_title_agree` (this module's + # own test file) checks it over the union of both predicates' + # example tables rather than leaving it to a sentence. return (bool(_PERIOD_ABBREV.match(text)) and (text.isascii() or not in_initialless_script(text))) @@ -207,11 +289,20 @@ def segment_suffix_reading(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], lenient: bool, + one_case: bool | None, ) -> tuple[bool, ...] | None: """How each piece of a no-name segment reads: True a suffix, False a title. None when the segment holds a name word and so is not a credential run at all. + `one_case` admits #289's credential lean: an ALL-CAPS member of + the ambiguous set inside a mixed-case name is a credential in this + slot even with one word before the comma, because the writing is + evidence the count does not have ('Smith, MA' -> family 'Smith', + suffix 'MA'). Only the LEAN reaches here: a token admitted to the + class by SHAPE takes the count instead, which is decided at the + comma and not in this walk ('Smith, A.B.' -> given 'A.B.'). + ONE answer for two readers, both in _assign.py -- the no-name gate and the router -- because they must agree piece for piece. #429 shipped the inverse of its own fix by deriving that agreement twice @@ -260,6 +351,11 @@ def segment_suffix_reading(pieces: Sequence[Sequence[int]], after_suffix = bool(out) and out[-1] if is_suffix_piece(piece, tags, tokens): out.append(True) + elif (len(piece) == 1 + and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags + and listed_lean(tokens[piece[0]], one_case) + == "credential"): + out.append(True) elif (lenient and after_suffix and _numeral_behind_the_initial_veto(piece, tokens)): out.append(True) @@ -310,7 +406,8 @@ def peel_walk(start: int, ptags: Sequence[Set[str]], def trailing_start(start: int, pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], skip: Set[int] = frozenset(), - numeral_only: bool = False) -> int: + numeral_only: bool = False, + *, one_case: bool | None) -> int: """Where assign's trailing suffix run begins, read over the pieces as they stand from `start`: the index of the first piece the S2 peel takes, or len(pieces) when it takes none (#424). What P2's @@ -329,21 +426,51 @@ def trailing_start(start: int, pieces: Sequence[Sequence[int]], with the piece the take leaves there; the acronym is left to assign.""" rest = peel_walk(start, ptags, skip) - peeled = peel_trailing(rest, pieces, ptags, tokens) + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case) if numeral_only: return rest[-1] if peeled.numeral is not None else len(pieces) return rest[peeled.names] if peeled.names < len(rest) else len(pieces) +# #289/#516: the "listed member, not by-shape" test both +# peel_trailing and segment_suffix_reading ask before reading the +# lean -- shared here so the two cannot drift on what counts +# (quality-review finding: it was spelled twice, once per site, +# before this). Both callers test "vocab:suffix-ambiguous" in tags +# INLINE, before calling this, rather than leaving that cheap check to +# this function's own body: measured, a caller whose `elif` reaches +# this on every piece (segment_suffix_reading's does, one per +# family-comma segment 1, member or not) pays one frame for the call +# regardless of what is inside it, and the inline pre-check is what +# keeps a non-member piece ("Smith, John"'s "John") from ever making +# the call at all. +def listed_lean(token: WorkToken, one_case: bool | None) -> Lean | None: + """`ambiguous_lean` for a LISTED bare-ambiguous token, or None if + the token is not tagged a listed member, is admitted by SHAPE + instead (`SHAPE_ACRONYM_TAG`, a switch's doing, not the writing's), + or there is no case fact to ask at all.""" + if (one_case is None or AMBIGUOUS_ACRONYM_TAG not in token.tags + or SHAPE_ACRONYM_TAG in token.tags): + return None + return ambiguous_lean(token.text, one_case) + + def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], - tokens: Sequence[WorkToken]) -> Peel: + tokens: Sequence[WorkToken], + one_case: bool | None) -> Peel: """The S2 trailing peel over `rest`, a peel_walk list. In the piece layer rather than in assign because group's bound-given reserve asks the same question of the view the join would leave (#425): one walk, so the reserve and the assignment cannot drift. Pure -- the ambiguities are returned for assign to report, in the order it always reported - them.""" + them. + + `one_case` is ParseState.one_case, the recorded fact: None means + nobody asked, which is every caller that has no state to ask with, + and reads as "no lean" -- rules.md#S2's count alone, the behavior + of every release before this one. + """ picks: list[tuple[int, ...]] = [] numeral: tuple[int, ...] | None = None k = len(rest) @@ -372,12 +499,43 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], # left ("Smith PhD" -> suffix, a classified fix), because there # the vocabulary is not in doubt. bare_ambiguous = (len(piece) == 1 - and "vocab:suffix-ambiguous" in tokens[piece[0]].tags) + and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags) + # #516, switch off: the writing still makes this token + # credential-SHAPED, and the parser is choosing the name + # reading over that one -- the fork the caller asked to be + # told about. Reported here, unconsumed, rather than folded + # into `bare_ambiguous` above: with the switch off the token + # never carries "vocab:suffix-ambiguous" (classify's own + # gate), so `bare_ambiguous` is already False and this is the + # ONLY place the report can be recorded. Switch ON, the token + # carries BOTH tags, so `not bare_ambiguous` is what stands + # this branch down and lets the consuming branch below take + # it; the two are not exclusive. + if (not bare_ambiguous and k >= 2 and len(piece) == 1 + and SHAPE_ACRONYM_TAG in tokens[piece[0]].tags): + picks.append(tuple(piece)) + break + # #289: written case is evidence the count does not have, and + # it overrides the count in BOTH directions -- an all-caps + # member of a mixed-case name is taken with nothing to spare + # ("Jack MA"), a Title-case one is declined with plenty + # ("John Smith Ma"). The lean is the LISTED set's alone: a + # token admitted to this class by SHAPE carries no writing + # convention to read, so it takes the count (decisions.md#S2). + # # k < 2 means it is the only piece left, which is not the fork - # this reports. + # this reports -- and not a floor the lean moves: the walk + # starts after the leading title run, so one piece behind a + # title ("Mr MA") is exactly this case and must stay a name. + # The lean is computed only past this floor -- membership, + # then the floor, then the count-or-lean, in that order, with + # nothing computed a step earlier could discard. if bare_ambiguous and k >= 2: picks.append(tuple(piece)) - if k >= 3: # peeling still leaves given + family + lean = listed_lean(tokens[piece[0]], one_case) + # peeling still leaves given + family, or the writing says + # to peel anyway + if lean == "credential" or (lean is None and k >= 3): k -= 1 continue break @@ -452,6 +610,7 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], + one_case: bool | None, ) -> tuple[list[int], tuple[int, ...], Peel]: """The S2 peel and the H5 chain read together to a FIXED POINT: peel, chain, splice the chained pieces out, peel again over what @@ -492,7 +651,7 @@ def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], """ titled: list[int] = [] while True: - peeled = peel_trailing(rest, pieces, ptags, tokens) + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case) kept = trailing_titles(rest[:peeled.names], pieces, ptags, tokens) if kept == peeled.names: diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index 8d3fd72b..f5f60cc9 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -1,6 +1,9 @@ """Stage: script_segment (#271, #272, #308, #312). -Consumes: tokens, segments, structure, interpunct_offsets, segmenter. +Consumes: tokens, segments, structure, interpunct_offsets, segmenter, +one_case (where segment recorded it; None everywhere else, which is +what this stage's suffix-run predicate read for every name before +2.4). Produces: tokens, by two independent splits into sub-slices -- a listed honorific peeled off the END of the name's last non-post-nominal token, in whichever of the name's runs that falls @@ -487,9 +490,21 @@ def _peel_honorific_tail(state: ParseState) -> ParseState: runs = state.segments[:1] if state.structure is Structure.FAMILY_COMMA: second = [state.tokens[j].text for j in state.segments[1]] - # a site here is asked about, not used: the offset it carries is - # >= 1 by the cap, so a site is always truthy and None never is - if not (is_wholly_suffix(second, state.lexicon, state.policy) + # `one_case` is passed for the reason the field exists: the + # credential lean is part of what "wholly suffix" MEANS since + # #289, and a stage that asks the question without it gets a + # different answer from `segment`, which asked it with the fact + # in hand one stage earlier. Left out, 'Kim김민준씨, MA' read + # 'MA' as name material, declined nothing, and scanned the + # second run -- where the site is 'MA' itself, which carries no + # listed tail -- so the person's own 씨 went unpeeled (family + # 'Kim김민준씨') while 'Kim김민준씨, PhD' peeled it, one name in + # two spellings parsed two ways (review round, #289/#516). + # `segment` records the fact only where a comma form could turn + # ON it, so this read is None for every other name and the + # predicate then behaves exactly as it did before 2.4. + if not (is_wholly_suffix(second, state.lexicon, state.policy, + one_case=state.one_case) and _peel_site(state, state.segments[0], tails)): runs = state.segments[:2] site = _peel_site(state, [j for seg in runs for j in seg], tails) diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index 5e7ae560..8ab71c6d 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -1,15 +1,36 @@ """Stage: segment. -Consumes: tokens (role-None main stream), comma_offsets. +Consumes: tokens (role-None main stream), comma_offsets, one_case +(where an earlier stage recorded it). Produces: segments (runs of main-token indices; interior segments may be EMPTY -- doubled commas keep their structural position), structure, -COMMA_STRUCTURE ambiguities for unrecognized extra segments. +one_case where the comma form asked for it, COMMA_STRUCTURE +ambiguities for unrecognized extra segments, and SUFFIX_OR_NAME where +the comma FLIPPED the structure for a member of the ambiguous +credential class. The flip and nothing else: where the structure did +not move, the word's reading is still open and `assign` takes it on +the family-comma path, so it reports there. Reads: Lexicon suffix vocabulary and Policy, both through _vocab.is_wholly_suffix -- the suffix-comma decision is definitionally vocabulary-dependent (decisions.md#C1), and the predicate owns the rest (Policy.lenient_comma_suffixes picks the lenient or strict token test; Policy.extra_suffix_delimiters gives v1 -suffix_delimiter parity, a delimiter-core token being transparent). +suffix_delimiter parity, a delimiter-core token being transparent); +Lexicon.maiden_markers DIRECTLY, for the own-words span the lazy case +gate takes (_pieces.own_words); Lexicon title and suffix vocabulary +plus Policy.lenient_comma_suffixes again through _vocab. +name_word_count, which counts NAME words for the class's own comma +rule; and, since 2.4, Policy.unlisted_dotted_suffixes through +_vocab.ambiguous_class_candidate, and Policy.unlisted_caps_suffixes +DIRECTLY as this stage's own gate before the run test calls +_vocab.caps_shape_candidate -- which reads every Lexicon vocabulary +field in turn (_lexicon._VOCAB_FIELDS) to decide that an all-caps +word is UNLISTED. `is_wholly_suffix` deliberately sees neither +by-shape half, dotted or caps (#516). An unlisted dotted or all-caps +token joins the ambiguous credential class by SHAPE at this stage's +own candidate tests the same way a listed member does; the caps half +additionally needs `one_case` to decide membership at all, which this +stage's own lazy gate supplies. Implements rules C1 and C2 of docs/design/rules.md, cited at the decision site below; history in decisions.md#C1. @@ -18,10 +39,14 @@ import dataclasses +from nameparser._pipeline._pieces import own_words from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, comma_bucket, ) -from nameparser._pipeline._vocab import is_wholly_suffix +from nameparser._pipeline._vocab import ( + ambiguous_class_candidate, ambiguous_class_member, caps_shape_candidate, + is_one_case, is_wholly_suffix, name_word_count, +) from nameparser._types import AmbiguityKind @@ -57,34 +82,231 @@ def segment(state: ParseState) -> ParseState: return dataclasses.replace(state, segments=segs, structure=Structure.NO_COMMA) - def suffixy(seg: tuple[int, ...]) -> bool: + # The case fact, asked LAZILY: only a comma form can turn on it + # here, and only where the part after the first comma is a single + # token -- the shape the ambiguous class comes in. A comma-less + # name never reaches this and pays nothing; classify asks for + # itself later where this did not (decisions.md#S2, and #429's + # precedent for paying a predicate twice rather than plumbing a + # field two sites would not otherwise share). + one_case = state.one_case + + def case_class() -> bool: + nonlocal one_case + if one_case is None: + # `own, _` rather than `[0]`: the second element is the + # maiden clause's start index, which this stage has no use + # for, and saying so by name is what stops a reader having + # to go and look up what a bare subscript dropped. + own, _ = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers) + one_case = is_one_case(own) + return one_case + + def texts(seg: tuple[int, ...]) -> list[str]: + return [state.tokens[i].text for i in seg] + + # Inlined rather than built on `texts` (measured, #289/#516's + # eager-gate fix round): every comma parse calls `suffixy` at + # least once, and a `texts(seg)` indirection costs a SECOND frame + # on top of the comprehension's own -- on 3.11 a list comprehension + # IS a frame (PEP 709 inlines it only from 3.12 on; see + # tools/perf/call_count.py's docstring), so wrapping it in another + # call doubles the cost every comma name pays, member or not. + # `texts` still serves the two call sites that are not on this + # path (name_word_count's pre-comma texts, and a flagged tail + # segment's joined display). + # + # `case` is ParseState.one_case: None is the case-FREE reading, + # which is what every caller on this path wants, and the tail + # walk below passes `case_class()` for the leaning one. One + # closure with a default rather than two whose bodies differ only + # in that argument. + def suffixy(seg: tuple[int, ...], case: bool | None = None) -> bool: return is_wholly_suffix([state.tokens[i].text for i in seg], - state.lexicon, state.policy) + state.lexicon, state.policy, one_case=case) + + def class_run(seg: tuple[int, ...]) -> bool: + # Every token of the run joins the ambiguous credential class + # BY SHAPE -- a candidate that is not a LISTED member, which is + # the one half `is_wholly_suffix` cannot see (its own docstring + # says so, and the blindness is what keeps a by-shape token out + # of C1's legacy token-count disjunct). A tail segment is + # consumed as suffix either way, so what this decides is only + # whether the parse says it did not RECOGNIZE the segment -- and + # a run the parse itself reads as a credential run by shape is + # recognized. Without it, the narrow roman retirement + # (rules.md#S3) moved 'R.A.I.', 'X.Y.I.' and 'J.u.n.i.o.r.' out + # of the vocabulary verdict and into the shape class, and every + # one of them gained a COMMA_STRUCTURE flag in a third segment + # that 2.3 did not raise -- a report about the parser's own new + # reading rather than about the name (review round, #289/#516). + # + # The LISTED half is excluded rather than folded in: a listed + # member reaches this reading through the LEAN (the leaning + # `suffixy` call below), and that is the whole of what quiets + # it -- 'STEVEN HARDMAN, MD, DO, DDS' is written in one case, + # leans nothing, and keeps its flag, the recorded negative + # control for the lean's own effect here. Admitting membership + # alone would silence it and leave the control measuring + # nothing. + # + # Policy-sensitive by construction: with + # `unlisted_dotted_suffixes` off the token is name material, is + # no candidate, and the flag stands. + return all(ambiguous_class_candidate(state.tokens[i].text, + state.lexicon, state.policy) + and not ambiguous_class_member(state.tokens[i].text, + state.lexicon) + for i in seg) # rules.md#C1: "the name reads as trailing suffixes when the part # after the first comma is entirely suffix words and more than one # word precedes the comma; otherwise it reads as the listing form" # (v1 parity: only parts[1] decides, parser.py:1318; history: # decisions.md#C1) + # + # And for the AMBIGUOUS class the count is of NAME words, not of + # words: 'Smith Jr., MA' is two tokens and one name, and the token + # count hands its family to `given` (#289/#516). One rule for the + # whole class: the listed and dotted halves are asked only of a + # single-token part, a member of either being always exactly one + # token (a bare word, or one glued acronym), while the caps half + # may come as a RUN ('LEED AP', below). + # + # Membership is tested CASE-FREE first (`ambiguous_class_candidate`, + # the listed set OR -- since 2.4 -- a by-shape member Policy + # admits, #516): the structure decision below is itself + # case-independent (item 5's count decides "whatever case the name + # is written in"), so the case fact is worth forcing only once a + # genuine candidate is found. Passing `case_class()` as an + # ARGUMENT instead ran the `own_words` -> `tag_marker_runs` walk + # for every comma name whose post-comma part is one token -- + # `"Smith, John"`, `"John Smith, Jr."`, neither able to reach the + # class at all (measured regression). + candidate = (len(groups[1]) == 1 + and ambiguous_class_candidate( + state.tokens[groups[1][0]].text, state.lexicon, + state.policy)) + # The all-caps half (Policy.unlisted_caps_suffixes, #516) is the + # FIRST shape this class can wear across more than one token -- + # 'LEED AP' is two separate all-caps words, not one glued acronym + # -- and the one membership test in this class that NEEDS the case + # fact to answer membership at all: 'XYZ' is only credential-shaped + # where the name contrasts it. Gated on the switch (default off, so + # a non-candidate comma name never enters this branch) and tried + # only where the single-token test above already declined. + # + # `caps_shape_candidate` directly, not the un-narrowed + # `ambiguous_class_candidate`: the run is a property of the CAPS + # class ALONE (#516 review round, F2), the other two halves being + # single-token by construction, so `all()` over more than one of + # THEM asks a question the design never posed. Measured, the + # un-narrowed call per token moved `'John Smith, Ed Ma'`, `'John + # Smith, ma do'` and `'John Smith, X.Y.Z. A.B.'` to a credential + # run neither the spec nor any case row wants. + # + # `one_case=False` asks the case-free question -- "if this name + # turned out mixed, would EVERY token in the run join the CAPS + # shape" -- the same trick the single-token test above uses, + # generalized over `all()`. Every other conjunct of + # `caps_shape_candidate` is case-independent, so for a run already + # confirmed shape-eligible the fact itself is the only unknown + # left, and the verdict is `case_class() is False` directly rather + # than a second walk that could only reach the same answer (a + # quality-review finding: the walk was provably redundant). + if (not candidate and state.policy.unlisted_caps_suffixes and groups[1] + and all(caps_shape_candidate(state.tokens[i].text, + state.lexicon, state.policy, + one_case=False) + for i in groups[1])): + candidate = case_class() is False + # Computed only where `candidate` is true, alongside `case_class()` + # -- the same lazy gate: a non-candidate comma name never counts + # its pre-comma words either. Hoisted to a local because the + # report below quotes the exact count rather than a hardcoded + # "two" ('John Q. Public, MA' has three). + pre_comma_names = None + if candidate: + # forced here, downstream of the structure decision that does + # not need it, because assign's post-comma slot and its report do + case_class() + pre_comma_names = name_word_count(texts(groups[0]), state.lexicon, + state.policy) + structure = ( + Structure.SUFFIX_COMMA + if ((suffixy(groups[1]) and len(groups[0]) > 1) + or (pre_comma_names is not None and pre_comma_names >= 2)) + else Structure.FAMILY_COMMA) + ambiguities = list(state.ambiguities) + if candidate and structure is Structure.SUFFIX_COMMA: + # The first report of the comma's OWN structure call in the + # library, and it is emitted for the branch taken HERE only -- + # the flip. Where the structure did not move, that token's + # reading is still open and `assign` takes it on the + # family-comma path, so it reports there; this DECISION is + # never reported twice, and a second ambiguous token elsewhere + # is a second fork reporting on its own + # (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE -- emitted + # where the branch is taken, and this branch is taken here). + # The neighbouring reports are other forks: C2's structural + # flag below says what the parse could not RECOGNIZE, and P6's + # attachment fork ("Berg, Jan vd", post_rules, since 2.3) is + # the attachment. rules.md#C1's comma-quiet policy gains its + # exception for this class and no other. + # + # `disp`/the index tuple cover the WHOLE post-comma part -- + # the same text as before for the single-token listed and + # dotted halves (a join of one element is that element), while + # the caps half's run ('LEED AP') is the first time this class + # reaches the comma form as more than one token (#516). The + # single-token case skips the join: measured, a generator + # expression is its own frame on 3.11 regardless of element + # count (unlike a list comprehension, which PEP 709 inlines + # only from 3.12), so the join alone cost every REPORTING + # comma name (`John Smith, MA`, `John Smith, A.B.`, `Davis + # Royce, Ed`) +2 frames at the DEFAULT policy, a path this + # switch must not touch at all (#516 review round, F4). + disp = (state.tokens[groups[1][0]].text if len(groups[1]) == 1 + else " ".join(state.tokens[i].text for i in groups[1])) + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{disp!r} after the comma is also an " + f"ordinary name word; the part before the comma holds " + f"{pre_comma_names} name words, so it is read as a " + f"credential run", + groups[1])) # rules.md#C2: "a non-empty extra part that is not entirely suffix # words is flagged as a structural ambiguity rather than rejected" # -- parts[2:] are consumed as suffixes unconditionally either # way, so a non-suffix tail segment gets the COMMA_STRUCTURE - # flag, not a structure veto - structure = (Structure.SUFFIX_COMMA - if suffixy(groups[1]) and len(groups[0]) > 1 - else Structure.FAMILY_COMMA) - ambiguities = list(state.ambiguities) + # flag, not a structure veto. The lean reaches this reading too: + # a tail of leaning credentials is a credential run, which is the + # one place this design quiets a report rather than adding one. for seg in groups[2:]: # empty segments are consumed silently (v1 skips them without - # comment); only non-empty non-suffix tails get flagged - if seg and not suffixy(seg): - texts = " ".join(state.tokens[i].text for i in seg) + # comment); only non-empty non-suffix tails get flagged. + # The case-FREE `suffixy(seg)` first: by construction the + # leaning call can only ADD a disjunct to it + # (`is_wholly_suffix`'s credential-lean branch), never remove + # one -- so a seg that already reads wholly suffix case-free + # reads so case-aware too, and the case fact is worth forcing + # only when the case-free answer was False (measured + # regression: the leaning call forced the fact for every tail + # segment, suffix or not). + # `class_run` last, for the same lazy reason and one step + # further out: it is the only one of the three that walks the + # by-shape class, and it is asked only of a run BOTH suffix + # readings have already declined. + if (seg and not suffixy(seg) and not suffixy(seg, case_class()) + and not class_run(seg)): + texts_joined = " ".join(texts(seg)) ambiguities.append(PendingAmbiguity( AmbiguityKind.COMMA_STRUCTURE, - f"segment {texts!r} beyond the recognized comma " + f"segment {texts_joined!r} beyond the recognized comma " f"structures; consumed as suffix best-effort", tuple(seg))) return dataclasses.replace(state, segments=tuple(groups), structure=structure, - ambiguities=tuple(ambiguities)) + ambiguities=tuple(ambiguities), + one_case=one_case) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 5aba83d4..870a7ef1 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -64,6 +64,44 @@ class WorkToken: #: cannot reach the other way). _NEVER_FLIPPED = frozenset({"vocab:bound-given", "initial"}) +#: The by-shape half of #289/#516's ambiguous credential class: a +#: token classify admits to `vocab:suffix-ambiguous`'s READING by +#: SHAPE rather than by the listed vocabulary +#: (`Policy.unlisted_dotted_suffixes` is the first emitter; +#: `Policy.unlisted_caps_suffixes` is the second, and classify writes +#: the tag from both branches). One constant, not a string literal at +#: each site, because the readers that must tell a by-shape member +#: apart from a listed one -- `_pieces.peel_trailing` and +#: `_pieces.listed_lean`, which `segment_suffix_reading` asks through, +#: so the reading it decides is second-hand -- cannot afford to spell +#: it several ways and have one of them typo silently past the +#: others. The two sites that want EITHER half read +#: `_AMBIGUOUS_CREDENTIAL_TAGS` below rather than this constant. +SHAPE_ACRONYM_TAG = "shape:acronym" + +#: The MEMBERSHIP half of the same class: classify's tag for a token +#: the ambiguous credential vocabulary claims, by listing +#: (`Lexicon.suffix_acronyms_ambiguous`) or -- where a Policy switch +#: admits the by-shape half -- beside `SHAPE_ACRONYM_TAG`. Beside that +#: constant and for its reason: the string was spelled out at eight +#: sites across four modules, each of them a place for a typo to pass +#: silently, since a tag that is never written is simply a tag no +#: reader ever finds. +AMBIGUOUS_ACRONYM_TAG = "vocab:suffix-ambiguous" + +#: EITHER way a token joins the ambiguous credential class -- the +#: vocabulary's claim and the writing's. The two emitters that report +#: a fork the peel called and declined ask exactly this: `_group`'s +#: prefix chain (a listed member the case lean read as a name, 'John +#: van der Berg Ma'; a by-shape member the count left standing, +#: 'Freiherr von Berg X.Y.I.') and `_assign`'s family-comma slot. One +#: constant beside the two above and for their reason -- the pair was +#: spelled two ways, a frozenset here and an `or` of two `in` tests +#: there -- and a frozenset so each test is one `isdisjoint`, a C call +#: with no Python frame, on a branch every chained name reaches. +_AMBIGUOUS_CREDENTIAL_TAGS = frozenset( + {AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG}) + class Structure(Enum): """segment's comma-structure decision.""" @@ -101,12 +139,14 @@ class ParseState: extract_delimited -> extracted/masked; tokenize -> tokens (span- sorted)/comma_offsets/interpunct_offsets (the 间隔号 offsets the order and segmentation decisions consult, #298; the nakaguro - separators record NOTHING); segment -> segments/structure; + separators record NOTHING); segment -> segments/structure/one_case + (lazily, only where a comma form could turn it on -- #289/#516); script_segment -> tokens and segments again (the one stage that changes the token COUNT: an unspaced CJK token splits into n+1 pieces, still as sub-slices of the original, and every later index - in the segment runs shifts by n); classify -> token tags; group -> - pieces/piece_tags/dropped AND maiden token roles; + in the segment runs shifts by n); classify -> token tags AND + one_case; group -> pieces/piece_tags/dropped AND maiden token + roles; assign -> the remaining token roles AND `order`, the effective order it read them under; post_rules -> roles again, and the ambiguity P6's attachment reports. @@ -153,4 +193,29 @@ class ParseState: #: script_orders entry -- which is why the test for it builds its #: own (test_post_rules.py). order: tuple[Role, Role, Role] | None = None + #: Whether the name's OWN words are written wholly in one case -- + #: all upper or all lower alike -- and so carry no case EVIDENCE + #: about any word in them (rules.md#P3's own-words span, + #: _pieces.own_words). None means NOT ASKED YET: the fact is + #: computed by whichever of segment and classify needs it first, + #: segment only when a comma form could turn on it, so a reader + #: between the two stages sees None and must not guess. + #: Recorded rather than recomputed, the way `order` above is: the + #: trailing suffix slot, the post-comma slot, the tail-segment + #: reading, the prefix chain's own tail measure (_group) and the + #: glued-honorific peel's decline of a post-comma run + #: (_script_segment) all consult it and must not disagree + #: (#289/#516, decisions.md#S2). Five stages read it in all -- + #: segment, script_segment, classify, group and assign -- plus the + #: two predicate layers they read it through (_pieces, _vocab). + #: A fact segment records SURVIVES script_segment, the one stage + #: that changes the token count, and survives it unrecomputed + #: because splitting a token cannot change the answer: the verdict + #: is `is_one_case` over the name's own words JOINED, and a split + #: only moves a space into a string whose upper/lower comparison + #: ignores spaces entirely -- '김민준씨' and '김민준 씨' fold alike. + #: So the field is carried through rather than invalidated, and + #: script_segment reads it (its suffix-run predicate takes the + #: lean) rather than asking again. + one_case: bool | None = None ambiguities: tuple[PendingAmbiguity, ...] = () diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 56bd50b6..a1877501 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -1,8 +1,9 @@ """Shared vocabulary predicates for pipeline stages. Text-level tests used by more than one stage; piece-level ones live -in _pieces, the sibling layer over tokens-plus-tags. All take normalized-or-raw text -explicitly -- no state. +in _pieces, the sibling layer over tokens-plus-tags. All take +normalized-or-raw text explicitly and no state, with the departures +named below. is_wholly_suffix departs from that shape twice, deliberately. It is RUN-level rather than text-level, because the question it answers is @@ -21,7 +22,21 @@ test can answer, and the vocabulary reaches it as a plain frozenset field like every other predicate here. -Layering: imports _lexicon, _types, and _policy only. +tag_marker_runs is the third departure, and the reason this module also +imports _state (WorkToken, comma_bucket): deciding which tokens open a +maiden-marker RUN needs each token's role and span, not its text alone, +because a run must not cross a role change or a comma. It moved here +from classify (#289/#516) so _pieces.own_words -- which may import +only _state and _vocab (tests/v2/test_layering.py) -- can call the +SAME function classify does rather than approximating it with a +text-only walk. Two callers building a map from the same tokens with +the same function cannot disagree, which a from-scratch approximation +could (measured: 'ANNA z Nowak, MD' flips the one-case verdict when +the approximation and the real run-completion test disagree on +whether 'z' opens a run that completes). + +Layering: imports _lexicon, _policy, and _pipeline._state (WorkToken, +comma_bucket -- read-only, never a whole ParseState). """ from __future__ import annotations @@ -29,10 +44,14 @@ import re import unicodedata from collections.abc import Callable, Iterable, Sequence +from typing import Literal -from nameparser._lexicon import FULL_STOPS, Lexicon, _normalize +from nameparser._lexicon import ( + FULL_STOPS, Lexicon, _VOCAB_FIELDS, _normalize, +) from nameparser._policy import (Policy, Script, _JA_SCRIPTS, _NO_INITIALS, _SCRIPT_RANGES, _script_matcher) +from nameparser._pipeline._state import WorkToken, comma_bucket # Ported verbatim from v1 (nameparser/config/regexes.py "initial") minus # its empty-string alternative -- WorkToken text is never empty. Kept in @@ -86,7 +105,7 @@ # contains-any, not whole=True: the shape half has already admitted the # trailing period, so the text reaching here is '씨.' rather than '씨' # and a wholly-of match would be False for every case this exists for. -# The second caller, _pieces.is_leading_title, admits two or more +# The second caller, is_title_shaped below, admits two or more # characters, so contains-any there means one CJK character anywhere # vetoes the whole word -- 'Kim김.' is refused as a title along with # '田中.' -- and that is deliberate: a word carrying a script with no @@ -94,6 +113,62 @@ in_initialless_script = _script_matcher(*_NO_INITIALS, whole=False) +# H2's own shape, text-level: an unlisted period-marked abbreviation, +# two or more letters, one trailing period. Moved here from _pieces +# (#289/#516, quality-review finding) so _vocab.name_word_count can +# ask the SAME question _pieces.is_leading_title asks -- measured +# divergence before the move: 'Dr. Smith, Ed' (LISTED title 'Dr.') and +# 'Xyz. Smith, Ed' (UNLISTED, H2-shaped) both read family 'Dr. +# Smith'/'Xyz. Smith', given 'Ed' at the peel, because is_leading_title +# reads H2's shape and a bare `folded in lexicon.titles` lookup does +# not -- but name_word_count's OWN count disagreed: 'Xyz.' counted as +# a name word where 'Dr.' did not, so 'Xyz. Smith, Ed' alone flipped +# the comma structure to a credential run +# (mechanisms.md#ONE-PREDICATE-PER-QUESTION). +# +# Ported verbatim from v1 (nameparser/config/regexes.py +# "period_abbreviation") -- layering forbids the config import; keep +# in sync by hand (tests/v2/test_regex_sync.py, which reaches this +# object through `_pieces._PERIOD_ABBREV`, an IMPORT of this one, not +# a second definition -- the sync test's target name did not move). +_PERIOD_ABBREV = re.compile(r'^[^\W\d_]{2,}\.$') + + +# rules.md#H2: "an abbreviation opening the part of the name that +# carries the given name — the whole name, or the part after a +# family comma — reads as a title even when unlisted" +# (history: decisions.md#H2) +def is_title_shaped(text: str) -> bool: + """Whether TEXT wears H2's shape alone -- vocabulary-free, the + LISTED half being a separate lookup at each caller's own site + (is_title_piece/lexicon.titles). `name_word_count` below is the + only CALLER; `_pieces.is_leading_title` asks the same question + but keeps this body inline, its path being hot (every leading + piece of every parse) where this one is cold (comma names only) + -- the measured figure lives at that inline copy and is not + restated here, two homes for one measurement being how they come + to disagree. Touch one, touch both: + `test_pieces.test_is_title_shaped_and_is_leading_title_agree` + runs both over the union of the two example tables, so the + agreement is checked rather than asserted in prose + (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + + The shape reads a Latin convention: a period marks an + abbreviation. Scripts with no initials have no period + abbreviations either (_policy._NO_INITIALS, the #320 veto + is_initial carries), so a CJK word wearing a period is a name + word, not a title -- a lone '田中.' is the family name (#323). + `_PERIOD_ABBREV` stays ASCII-period only: a word wearing '。' never + matches it, and the veto is what makes the ASCII spelling agree. + ASCII text can carry no _NO_INITIALS character (every range sits + above U+3000), so the C-level test declines before the regex + search runs -- four frames per unlisted-abbreviation opener per + parse, this running four times per piece. + """ + return (bool(_PERIOD_ABBREV.match(text)) + and (text.isascii() or not in_initialless_script(text))) + + def is_initial_shaped(text: str) -> bool: """v1's is_an_initial verbatim: the SHAPE half alone -- one word character plus a period, or a bare ASCII capital. @@ -177,6 +252,59 @@ def is_one_case(texts: Sequence[str]) -> bool: return joined in (joined.upper(), joined.lower()) +#: Which way the WRITING leans for a member of the ambiguous +#: credential class; None is no lean at all. Named so the two +#: predicates that answer it -- `ambiguous_lean` here and +#: `_pieces.listed_lean`, which wraps it -- carry the same three-value +#: type rather than a bare `str`. Under mypy's `strict_equality` that +#: makes a misspelled comparison (`== "credentail"`) an error at the +#: two call sites that branch on the answer, where a `str` return +#: leaves it silently False forever -- which is what the value is FOR: +#: `_pieces.peel_trailing` peels or declines on it. +#: `period_joined_vocab`'s own verdict is spelled inline for the same +#: reason; it has one caller shape and no wrapper to keep in step. +Lean = Literal["credential", "name"] + + +# #289's credential lean: rules.md#S2 records the words-to-spare count +# today; this predicate is the WRITING evidence that decides ahead of +# it, in a mixed-case name only (history: decisions.md#S2). +def ambiguous_lean(text: str, one_case: bool) -> Lean | None: + """Which way the WRITING leans for a member of the ambiguous + credential set: "credential", "name", or None for no lean at all. + + None is the fall-through to rules.md#S2's count, and it is the + answer for three inputs. A name written wholly in one case says + nothing about any word in it. A member written wholly in lower + case says nothing either -- lower is how most of a mixed-case + name is written, so it is not a contrast. And a CASELESS token + ('씨', '毛') can be written against nothing, so neither test + fires and the count decides, which is what keeps this rule out of + a caseless script by construction. + + `one_case` is the recorded fact (ParseState.one_case), taken over + the name's OWN words -- handed in rather than recomputed, because + three sites read this and a fact two of them derived apart is the + bug the field exists to prevent + (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + + The caller decides MEMBERSHIP: this answers only about the + writing. Periods do not disturb either test, '.' having no case, + so 'MA.' leans as 'MA' does -- the lean reads CASE, not periods, + and the dotted spelling is the vocabulary's own question + (suffix_as_written). + """ + if one_case: + return None + if text.upper() == text.lower(): # caseless: no contrast to read + return None + if text.isupper(): + return "credential" + if text.islower(): + return None + return "name" + + _DOTTED = re.compile(r"(?:[^\W\d_]\.)+") @@ -262,28 +390,290 @@ def splits_into_suffixes(text: str, cores: frozenset[str], # rules.md#S3: "a word with interior periods reads as a suffix when -# any of its period-separated chunks is suffix vocabulary" -def period_joined_vocab(text: str, lexicon: Lexicon) -> str | None: +# any of its period-separated chunks is suffix vocabulary — except +# where every chunk the vocabulary matches is a single ASCII +# character, the roman numerals and the lone digit the vocabulary +# lists, which are about generations rather than credentials" +def period_joined_vocab( + text: str, lexicon: Lexicon, +) -> Literal["title", "suffix", "shape"] | None: """v1's parse_pieces derivation for interior-period tokens ('Lt.Gov.', 'Msc.Ed.', and by the ANY rule 'Mr.Smith'): ANY title chunk makes the token a title (checked first, v1's continue); else ANY suffix chunk makes it a suffix. Chunk-level suffix membership is v1's is_suffix: bare ambiguous acronyms COUNT ('Msc.Ed.' derives via 'ed') -- the ambiguous period-gate applies to whole - tokens only. Returns "title", "suffix", or None.""" + tokens only. Returns "title", "suffix", "shape", or None. + + "shape" is the third verdict and it is the absence of a claim + worth acting on (#516): two or more chunks, EVERY one of them + alphabetic, and nothing the vocabulary matches except -- possibly + -- chunks that are a SINGLE ASCII CHARACTER. That exception is the + roman-numeral accident retired narrowly: measured 2026-09-15 the + one-character suffix vocabulary is {'2', 'i', 'v'} plus the glued + CJK honorific tails, so 'John Smith R.A.I.' was reading as a + generational suffix off the chunk 'i'. CHARACTER rather than + LETTER because '2' is a digit and is in the set; ASCII because + '씨' is the single character that must KEEP its claim ('J.씨'). A + multi-character match still reads as it always did, so 'Msc.Ed.', + 'JD.CPA' and 'Lt.Gov.' are untouched -- the WIDE retirement, where + any chunk match yields to the shape, was measured and rejected: it + costs 'Doe, John Msc.Ed.' a real credential and re-routes the + honorific peel of '김민준씨, J.씨' for nothing this design wants + (decisions.md#S2). + + The ALPHABETIC requirement is a SEPARATE, narrower gate on the + shape verdict alone (#516 review round, decided by the + orchestrator): a bare digit chunk is not an acronym letter by any + reading, so 'Smith, 1.4' and 'John Smith 1.4' do not join the + class by shape -- the only PROTECTED digit control this design + had was the delimited 'Bridge (1.4)', and 'Smith, 1.4' and its + no-comma twin were the gap that control did not cover, since + delimited content is excluded by a different mechanism (extract's + escape) and digits reach THIS detector, never that one. + '.isalpha()' is asked of EVERY chunk, not just the matched ones, + so a mixed token like 'X.Y.2.' is not shape-admitted either -- one + non-letter chunk is enough to say the writing is not spelling an + acronym. Chunks the vocabulary itself matches are already + alphabetic in the shipped lexicon, so this narrows only the + previously-unclaimed "shape" answer, never the "suffix" one. + + The INITIALLESS-SCRIPT guard is a second, independent narrowing of + the same verdict, on the same reasoning `is_title_shaped` already + gives its own period-abbreviation inference (#323): a script with + no period abbreviations at all has nothing for interior periods to + ABBREVIATE, so a CJK word glued into period-separated single + characters is not spelling an acronym either -- 'John Smith + 田.中.' and '김 민준 이.박.' stayed family at every release before + this gate existed, and would otherwise have joined the ambiguous + class by shape for the first time (measured regression, #516 + review round). + + What a "shape" verdict MEANS is the caller's question, not this + one's: classify writes the tag, and Policy.unlisted_dotted_suffixes + decides whether it is class membership. + """ if not _PERIOD_NOT_AT_END.match(text): return None chunks = [_normalize(c) for c in text.split(".") if c] if any(c in lexicon.titles for c in chunks): return "title" - if any(c in lexicon.suffix_acronyms or c in lexicon.suffix_words - for c in chunks): + matched = [c for c in chunks + if c in lexicon.suffix_acronyms or c in lexicon.suffix_words] + if matched and not all(len(c) == 1 and c.isascii() for c in matched): return "suffix" + if (len(chunks) >= 2 and all(c.isalpha() for c in chunks) + and (text.isascii() or not in_initialless_script(text))): + return "shape" return None +def ambiguous_class_member(text: str, lexicon: Lexicon) -> bool: + """Whether TEXT is a member of the LISTED ambiguous credential + class, ignoring case and policy entirely (#289/#516). + + The case-INDEPENDENT half of the comma form's own candidate test + (`ambiguous_class_candidate`, below, which ALSO admits a by-shape + member where Policy allows it) and of `is_wholly_suffix`'s + credential-lean disjunct: membership by vocabulary never needs the + case fact, only the lean does. That split is what lets the comma + form's lazy gate ask membership FIRST and pay for `one_case` only + where this says yes (measured regression, #289/#516: forcing the + fact first ran `own_words` -> `tag_marker_runs` for every + single-token-after-the-comma name, none of them able to reach the + class at all). + + Membership is the listed set, bare: a whole-token vocabulary match + is not in this class at all, being settled ('M.A.', 'Ph.D.', + 'A.B.C.'). + + Cheaper than `suffix_as_written(n, text, lexicon) or ...` would be + here, and provably the same answer for UNDOTTED text: the Lexicon + invariant that an ambiguous acronym is never also a suffix WORD + (`__post_init__`'s gate_bypassed check) and is never counted + without periods once it IS one (`suffix_as_written`'s own + exclusion) together kill that predicate's two disjuncts once text + is known to hold no period. One frame (`_normalize`) on the common + path that never reaches the class, where the general predicate + cost at least two (mechanisms.md#ONE-PREDICATE-PER-QUESTION's cost + clause). + + The '.' gate here is DELIBERATELY STRICTER than S2's own + dotted-form test (`_dotted`, a period after EACH letter): '.' + anywhere excludes membership, so a single TRAILING period ('MA.', + 'Ed.') is excluded here even though the LEAN still reads it as + the bare acronym's case ('Smith, MA.' -> suffix 'MA.', measured). + Not a contradiction: the two questions this answers -- the + comma-form CANDIDATE test below and the credential-lean disjunct + -- are never asked of those shapes, the TAG path + (`vocab:suffix-ambiguous`, read directly by the trailing peel and + the post-comma slot) and H2's leading-title shape test already + carrying them where they need to go. + """ + if "." in text: + return False + return _normalize(text) in lexicon.suffix_acronyms_ambiguous + + +# #516's all-caps half, ONE PREDICATE for the shape test and its +# WHOLE-VOCABULARY exclusion, shared by the three sites that each +# needed the identical question answered (classify's tag emission, +# `_segment.py`'s multi-token run test, and this module's own unit +# tests), where it had been spelled three times over (quality-review +# finding). The usual objection to sharing -- a call costing every +# default-policy parse a frame it cannot use -- does not apply: every +# caller's own first conjunct is `policy.unlisted_caps_suffixes`, +# False by default, so neither this call nor the loop inside it is +# ever reached at the default (confirmed against the 412/449 frame +# band and the default comma harness). +def caps_shape_candidate(text: str, lexicon: Lexicon, policy: Policy, + one_case: bool | None) -> bool: + """Whether TEXT is an UNLISTED all-caps credential candidate: two + or more alphabetic characters, no period (a period fails + `isalpha()` outright, which is what excludes the dotted spelling + here for free -- no separate '.' test needed), written in a name + `one_case` says is mixed (`one_case is False`; `None`, "not + established", declines exactly as if the name were one case). + + UNLISTED means in NO wordlist at all, not merely "no whole-token + suffix vocabulary", and the roster is `_lexicon._VOCAB_FIELDS` + itself rather than a list written out here -- checked directly + against the lexicon rather than through a caller's tags (this + function's own callers have none to read, `segment` running before + `classify`). A hand-written roster is a second place to remember, + and it had already gone wrong: it named eleven of the thirteen + fields, leaving `surnames` and `honorific_tails` out, so + `Lexicon.default().add(surnames={"dupont"})` still read + `Jean Pierre DUPONT` as suffix `DUPONT` -- a caller listing a word + as a SURNAME and getting it read as a credential is this switch's + own worst failure, arriving through the one wordlist that says + "this is a family name" (review round, #289/#516; + `honorific_tails` was already excluded transitively, being a + subset of `suffix_words` by Lexicon invariant, and joins the + roster for completeness rather than for a behavior change). + Measured, #516 review rounds: particles, ambiguous particles, + conjunctions, bound-given heads, a maiden marker ('NEE'/'GEB') and + a title all join the shape by capitalization alone if their field + is left unchecked. + + `suffix_acronyms_ambiguous` is in the roster, which also makes + this predicate stand in for `ambiguous_class_member` wherever a + caller needs "and not already a LISTED member" (undotted text's + only path into that function is the identical membership test) -- + `_segment.py`'s run test relies on exactly that rather than + calling both. + """ + if not (policy.unlisted_caps_suffixes and one_case is False + and len(text) >= 2 and text.isalpha() and text.isupper()): + return False + n = _normalize(text) + for field in _VOCAB_FIELDS: + if n in getattr(lexicon, field): + return False + return True + + +# The comma form's own candidate test (rules.md#C1, decisions.md#S2). +def ambiguous_class_candidate(text: str, lexicon: Lexicon, + policy: Policy) -> bool: + """Whether TEXT is a CANDIDATE for the ambiguous credential class + at the comma form's own structure decision (`_segment.py`): the + LISTED half (`ambiguous_class_member`, case-free) OR, where Policy + admits it, the SHAPE an unlisted dotted token wears + (`period_joined_vocab`'s third verdict, #516). + + Case-free throughout, and the CAPS half of the class is + deliberately not here: its real site is `_segment.py`'s + multi-token run test, which calls `caps_shape_candidate` directly + because the run is a property of that shape alone. A second route + through here existed briefly, behind an optional `one_case` + parameter no production caller ever passed -- `segment` has + nothing to hand in at its single-token test -- so the branch + answered False for every name the library parsed and only the + unit tests reached it (review-round finding, #289/#516). A dead + second route is a place for the two to disagree, not a + convenience. + + A period anywhere is the gate for even ASKING the shape question, + checked before the shape's own two calls: `period_joined_vocab` + already declines a period-free text, but a Python-level call is + not free and a comma name's post-comma part usually has no period + ("Smith, John") -- measured regression, #516 review round, fixed + by hoisting the same cheap substring test `ambiguous_class_member` + already makes for its own reason. Where a period IS present, + `suffix_as_written` runs only after `period_joined_vocab` says + "shape": a dotted whole-token match with no single-chunk + vocabulary hit of its own ('A.B.C.', via 'abc') would otherwise + read "shape" from this function's chunk-level view alone, + oblivious to the WHOLE-token match `suffix_as_written` already + settled -- the same precedence classify's own tag order gives it + (`vocab:suffix` is set before `period_joined_vocab` is even + consulted). + + A LISTED member spelled with its periods is excluded from the + shape branch by the same test classify's own shape branch makes: + a caller who puts a dotted entry in `suffix_acronyms_ambiguous` + has said the word is a listed member of this class, and reading + it by shape instead loses the case lean the listing asks for + (review-round finding, #289/#516 -- `Jack A.B.` with 'a.b' listed). + It cannot change the answer for the shipped vocabulary, whose + ambiguous entries carry no period at all. + + This function and classify's tag emission (`_tags_for`'s + `derived == "shape"` branch) still ask the SAME question twice, of + necessity: `segment` runs before `classify` and has no tags to + read yet, so the two stages cannot share the call. Kept from + drifting by + `test_classify.test_ambiguous_class_candidate_agrees_with_the_tag`, + which asks both of the same texts, rather than by a sentence + alone. + """ + if ambiguous_class_member(text, lexicon): + return True + if "." in text: + # `_normalize` stays behind the shape verdict, where it always + # was: it is a call, and a dotted post-comma token that is not + # acronym-shaped at all ('Jr.') must not pay for it. + if not (policy.unlisted_dotted_suffixes + and period_joined_vocab(text, lexicon) == "shape"): + return False + n = _normalize(text) + return (n not in lexicon.suffix_acronyms_ambiguous + and not suffix_as_written(n, text, lexicon)) + return False + + +def name_word_count(texts: Sequence[str], lexicon: Lexicon, + policy: Policy) -> int: + """How many of these texts are NAME words -- not suffix + vocabulary, not title vocabulary. + + rules.md#C1's count for the ambiguous class, and it is of names + rather than of words because 'Smith Jr., MA' is two tokens and one + name: counting tokens there flips the structure and hands the + family to `given`, which no reading of that string wants + (decisions.md#S2). The suffix half asks the POLICY-selected + predicate, the same one is_wholly_suffix asks, so the two agree + about what a suffix word is; the title half asks BOTH the listed + lookup and H2's shape test (`is_title_shaped`) -- a bare + `lexicon.titles` lookup here read 'Xyz.' as a name word where the + leading peel reads it as a title, and that divergence is recorded + once, at `is_title_shaped` itself + (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + """ + predicate = (is_suffix_lenient if policy.lenient_comma_suffixes + else is_suffix_strict) + n = 0 + for text in texts: + if (predicate(text, lexicon) or _normalize(text) in lexicon.titles + or is_title_shaped(text)): + continue + n += 1 + return n + + def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, - policy: Policy) -> bool: + policy: Policy, one_case: bool | None = None) -> bool: """Every token in a RUN counts as a suffix -- segment's suffix-comma test, lifted out of it so the peel can ask the same question (#319). @@ -305,6 +695,33 @@ def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, An adjacent Ph./D. pair counts as ONE unit (v1's fix_phd extracted the credential pre-parse, so 'Smith, Ph. D.' read as suffix-comma); keep in sync with group's _PH/_D merge. + + `one_case` is ParseState.one_case, and it admits the LEAN: a bare + ambiguous acronym written in capitals inside a mixed-case name is + a credential here, so 'Steven Hardman, MD, DO, DDS' reads its + third segment as the credential run it is (#289). None -- the + default, and what every caller with no state to ask has -- reads + as no lean and is this predicate's behavior in every release + before 2.4. + + Neither Policy.unlisted_dotted_suffixes NOR + Policy.unlisted_caps_suffixes reaches this predicate: admitting a + by-shape member here unconditionally (an earlier version of this + docstring described exactly that, for the dotted half alone) + bypassed both the lean AND the NAME-word count, and combined with + C1's own legacy TOKEN-count disjunct in `_segment.py` + (`suffixy(groups[1]) and len(groups[0]) > 1`) it flipped + 'Smith Jr., A.B.' to given 'Smith', suffix 'Jr., A.B.' with a + self-contradicting report ("holds 1 name words, so it is read as + a credential run") -- proved by mutation testing to be otherwise + unreached: nothing but this predicate's own two unit tests + depended on it, and 'John Smith, A.B.' still flips correctly + through `pre_comma_names >= 2` alone (#516 review round). Neither + by-shape class, dotted or caps, reaches the comma form through + this predicate at all -- only through `_vocab. + ambiguous_class_candidate`, which segment's structure decision and + report both already consult (the caps half as a RUN test over it, + #516's second review round). """ if not texts: return False @@ -318,10 +735,13 @@ def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, def counts_as_suffix(text: str) -> bool: if text in cores: return True + if (one_case is not None + and ambiguous_class_member(text, lexicon) + and ambiguous_lean(text, one_case) == "credential"): + return True return (predicate(text, lexicon) or period_joined_vocab(text, lexicon) == "suffix" - or (bool(cores) - and splits_into_suffixes(text, cores, lexicon))) + or (bool(cores) and splits_into_suffixes(text, cores, lexicon))) merged = list(texts) k = 0 @@ -435,6 +855,117 @@ def maiden_marker_run(words: Sequence[str], markers: frozenset[str]) -> int: return 0 +def tag_marker_runs(tokens: Sequence[WorkToken], comma_offsets: Sequence[int], + markers: frozenset[str], + folded: Sequence[str] | None = None) -> dict[int, str]: + """Which tokens are maiden marker runs: index -> "vocab:maiden-marker" + for a run's head, "vocab:maiden-marker-cont" for the rest. + + Returns the decision rather than rewriting the tokens: a caller + that writes tags into the one pass that builds them (classify) + consults this map rather than deciding a run twice, so a marker + token is never replaced twice; a caller that never writes tags at + all (own_words) reads the same map for its span instead. + + Moved here from classify (#289/#516) so a caller outside classify + -- _pieces.own_words, when it has no marker map yet -- can build + the SAME map classify would, rather than approximating it with a + text-only head test. `_pieces.py` may import only _state and + _vocab (tests/v2/test_layering.py), so the one shared function + both sites call has to live here, in the layer both can reach. + + `folded` is _normalize per token; classify has already built it + for the vocabulary-tag pass and hands it over so this pass costs + no second fold, and a caller with only tokens (own_words' self- + built path) omits it and pays the fold here instead -- paid only + on that path, never on classify's, so the reference name's frame + count is unchanged (#289/#516). + + The one sequence pass in this stage, and it has to be one: a marker + entry may be a PHRASE whose words are not markers individually + ('z', 'domu'), so no per-token membership test can find it. + Left to right, longest first at each position, then skip past what + the run claimed -- a second marker cannot start inside the first. + + This is where the tag is DECIDED for the stages that read it + afterwards. group runs later and asks its questions of these tags + rather than re-deriving the run (the recorded-answer half of + mechanisms.md#ONE-PREDICATE-PER-QUESTION); extract runs EARLIER, + before tokens exist, so it calls the predicate itself over the + clause's whitespace words. + + A tagged run is structurally contiguous, and the test is + one-directional: a role change IS a clause edge, so no run spans + one, but not every clause edge is a role change -- two ADJACENT + clauses of the same role are indistinguishable here, and + 'Jane (z) (domu) Jones' does tag a run across them. Both consumers + refuse that run for reasons of their own (the piece walk never sees + role-bearing tokens at all; the clause drop is scoped to one + clause's span), so no reading depends on it today, and the claim + this pass can honestly make is the weaker one. What it does + guarantee is what _group._marker_run_pieces needs: a run inside the + MAIN stream stays inside one segment. Without it this pass walked + the whole span-sorted stream while group walked one segment -- + _segment keeps only role-less tokens and buckets them by the commas + before them -- so a run half inside a bracketed clause was tagged + whole and consumed as a proper PREFIX of itself, and + 'Anna z (domu) Nowak' read family 'Anna', maiden 'Nowak': the bare + preposition eating the name, which is the exact damage the phrase + entry exists to prevent. Refusing to tag such a run is the fix; + truncating it instead would hand M2 the same wrong prefix one word + shorter. + """ + # the lookahead the vocabulary actually needs; 0 for an empty set, + # which skips the pass entirely + cap = _longest_marker(markers) + if not cap: + return {} + if folded is None: + folded = [_normalize(t.text) for t in tokens] + n_tokens = len(tokens) + # Deferred, not computed up front: only the contiguity walk reads + # it, only a phrase vocabulary runs that walk, and only at a token + # that opens an entry -- so a single-word vocabulary, and a + # phrase vocabulary over a name holding no marker, never pay the + # sweep at all. + buckets: list[int] | None = None + tags: dict[int, str] = {} + i = 0 + while i < n_tokens: + # The predicate's own head test first, over the fold the caller + # already has: almost no token opens any entry, and for those + # there is nothing to assemble. Same function maiden_marker_run + # consults, so a token skipped here is one it would refuse. + if not maiden_marker_head(folded[i], markers): + i += 1 + continue + # Bound the lookahead at the first structural boundary, so the + # predicate is asked over the words that could form one run and + # answers longest-first WITHIN them -- a two-word entry refused + # at a clause edge still leaves a one-word entry starting there + # free to match. + limit = 1 + if cap > 1: + if buckets is None: + buckets = [comma_bucket(t.span.start, comma_offsets) + for t in tokens] + role, bucket = tokens[i].role, buckets[i] + while (limit < cap and i + limit < n_tokens + and tokens[i + limit].role is role + and buckets[i + limit] == bucket): + limit += 1 + run = maiden_marker_run( + [tokens[k].text for k in range(i, i + limit)], markers) + if not run: + i += 1 + continue + tags[i] = "vocab:maiden-marker" + for k in range(i + 1, i + run): + tags[k] = "vocab:maiden-marker-cont" + i += run + return tags + + def _normalized_for_script(text: str) -> str | None: """The guard AND the two normalizations single_script and effective_script's license path both need, single-sourced so they diff --git a/nameparser/_policy.py b/nameparser/_policy.py index 6f4fbf3f..8aabc580 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -662,6 +662,59 @@ class Policy: #: they appear in no token, field, or rendered view; the original #: string keeps them. strip_bidi: bool = True # =False replaces v1's opt-out CONSTANTS.regexes.bidi = False + # -- fields added after 2.3 ------------------------------------------ + # Policy is not kw_only, so a field's POSITION is API: a caller + # writing Policy(GIVEN_FIRST, ..., True, False) binds by position. + # The two 2.4 switches below first landed beside + # lenient_comma_suffixes, which re-bound every positional argument + # from `strip_emoji` on; they are appended here instead so 2.3's + # eleven positions keep their meaning + # (tests/v2/test_policy.py::test_the_2_3_positional_fields_did_not_move). + # A field added in a later cycle goes at the END for the same + # reason, whatever it is about. + #: Reads an UNLISTED token of two or more period-separated chunks + #: as a credential where the position allows it: "John Smith + #: X.Y.Z." gives suffix ``X.Y.Z.`` and "Jack X.Y.Z." keeps family + #: ``X.Y.Z.``, the same words-to-spare rule a listed ambiguous + #: acronym takes, and either reading is reported. Case is + #: irrelevant here -- the periods are the signal, so + #: "john smith x.y.z." reads as the mixed-case spelling does. + #: Whole-token vocabulary still wins ("M.A.", "Ph.D.", "A.B.C."), + #: and a single trailing period is not this shape ("John Smith + #: Xyz." keeps family ``Xyz.``). ``False`` reads such a token as + #: name material everywhere, as 2.3 did for a token no chunk + #: claimed; the roman-chunk retirement (rules.md#S3) is not + #: behind this switch, and still reports the fork. + unlisted_dotted_suffixes: bool = True + #: Reads an UNLISTED all-caps word of two or more letters, with no + #: period in it, in a name written in more than one case as a + #: credential where the position allows it: with this on, + #: "John Smith XYZ" gives suffix ``XYZ`` and "Smith, XYZ" still + #: gives given ``XYZ``, the same words-to-spare rule the rest of + #: the class takes. A listed member keeps its own case lean + #: regardless of this switch ("Jack MA" still gives suffix ``MA`` + #: on or off), and the roman-numeral fork still claims a bare + #: numeral first either way ("Jack VI" is unaffected by this + #: switch, on or off). OFF BY DEFAULT, and the asymmetry with + #: ``unlisted_dotted_suffixes`` is deliberate: an all-caps surname + #: is a real writing convention that shape cannot separate from a + #: credential -- "Jean Pierre DUPONT" gives family ``Pierre``, + #: suffix ``DUPONT`` with this on, and a swallowed family name is + #: the worse failure. The two-word "Jean DUPONT" and "Minjun KIM" + #: read as family names at the default and KEEP that family with + #: this on too (one word before the credential is never enough, + #: the same words-to-spare rule above) -- but a genuine candidate + #: this switch does not move still gains the fork's report: it + #: was a real fork the parser considered and declined, and that + #: is reported even where the reading did not change. Off, + #: nothing changes and nothing is reported. A digit anywhere + #: disqualifies the token and a single capital stays an initial. + #: ``isupper()`` is script-agnostic, so this is the same + #: convention and the same reason for being off in ANY script + #: that has a case contrast at all, not just Latin -- an all-caps + #: Cyrillic surname ("Иван ИВАНОВ") or an accented Latin one + #: ("Jean ÉCOLE") joins this class exactly as an ASCII one does. + unlisted_caps_suffixes: bool = False # in the class body so @dataclass(slots=True) keeps them __getstate__ = _guarded_getstate @@ -752,8 +805,17 @@ def __post_init__(self) -> None: # Truthy strings ("no", "false") would silently invert the # caller's intent downstream; bools are the one field kind the # coercing checks above can't cover. - for flag in ("middle_as_family", "lenient_comma_suffixes", - "strip_emoji", "strip_bidi"): + # + # The roster is DERIVED from the dataclass rather than written + # out: a hand-written list is a second place to remember, and + # the twin loop in tests/v2/test_policy.py proved it -- it still + # named 2.3's four flags after 2.4 added two, so the new + # switches shipped with no "must be a bool" coverage while the + # library validated them. `bool` is the whole test: every + # bool-annotated Policy field is a flag, and the annotation is + # a plain string here because of `from __future__ import + # annotations` (AGENTS.md's guard-the-whole-family rule). + for flag in _BOOL_FIELDS: value = getattr(self, flag) if not isinstance(value, bool): raise TypeError( @@ -790,6 +852,18 @@ def patched(self, patch: PolicyPatch) -> Policy: return apply_patch(self, patch) +#: Every bool-valued Policy field, read off the dataclass rather than +#: listed: `__post_init__`'s bool check sweeps this, so a flag added to +#: the class above is validated the day it lands and cannot ship +#: unchecked the way `unlisted_dotted_suffixes` and +#: `unlisted_caps_suffixes` did. `from __future__ import annotations` +#: makes every annotation a string, so the comparison is against the +#: SPELLING "bool" -- which is also what a reader of the class body +#: sees, and a field annotated any other way is not a plain flag. +_BOOL_FIELDS: tuple[str, ...] = tuple( + f.name for f in dataclasses.fields(Policy) if f.type == "bool") + + class _Unset(Enum): UNSET = auto() @@ -833,6 +907,12 @@ class PolicyPatch: lenient_comma_suffixes: bool | _Unset = UNSET strip_emoji: bool | _Unset = UNSET strip_bidi: bool | _Unset = UNSET + # Appended, in Policy's order and for Policy's reason -- this class + # is positional too, and the parity test holds the two field + # sequences equal, so a field inserted on one side would have to be + # inserted on the other and both would re-bind together. + unlisted_dotted_suffixes: bool | _Unset = UNSET + unlisted_caps_suffixes: bool | _Unset = UNSET # in the class body so @dataclass(slots=True) keeps them __getstate__ = _guarded_getstate @@ -882,9 +962,11 @@ def __post_init__(self) -> None: f"{f.name} must be an iterable, got {value!r}{hint}" ) from None object.__setattr__(self, f.name, frozenset(value)) - # middle_as_family, lenient_comma_suffixes, strip_emoji, and - # strip_bidi are scalar (compose="override") fields and - # DELIBERATELY get no type check here, unlike name_order and + # middle_as_family, lenient_comma_suffixes, + # unlisted_dotted_suffixes, unlisted_caps_suffixes, strip_emoji, + # and strip_bidi are scalar (compose="override") fields and + # DELIBERATELY get no + # type check here, unlike name_order and # the union fields above: a PolicyPatch(strip_emoji="off") is # constructible, and only raises once apply_patch runs # Policy.__post_init__'s bool check. This is the one place the diff --git a/nameparser/_types.py b/nameparser/_types.py index 3d076a1e..07b44d3b 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -371,12 +371,23 @@ class AmbiguityKind(StrEnum): existing values never change meaning. A kind names a FORK THE PARSE HAD TO CALL, not a word that could be - read two ways: the same token elsewhere in a name may present no - choice at all and is then reported by nothing. Reporting is also + read two ways, except where a member says otherwise: the same token + elsewhere in a name may present no choice at all and is then + reported by nothing. COMMA_STRUCTURE and UNBALANCED_DELIMITER are + the two that say otherwise -- each reports a shape the parse could + not recognize rather than a fork it chose between, and each says so + on its own member below. Reporting is also partial -- a kind listed here is not necessarily emitted everywhere - its fork occurs (the comma paths stay quiet by design), and coverage - grows over releases. A non-empty tuple is a signal to act on; an - empty one is not a guarantee of certainty.""" + its fork occurs (the comma's structure decision reports no + reading by design, except where it decides a member of the + ambiguous credential class, where since 2.4 that decision is + reported either way -- #289; an attachment decided AFTER a + family comma is a separate fork and has reported on its own + since 2.3, e.g. "Berg, Jan vd"; the two structural kinds above + were never covered by either silence), and coverage grows over + releases. A non-empty tuple is + a signal to act on; an empty one is not a guarantee of + certainty.""" #: Reserved: the name's field order itself is uncertain (e.g. a #: two-word name under a non-default name_order). Not yet emitted; @@ -404,6 +415,16 @@ class AmbiguityKind(StrEnum): #: vocabulary ("Rinpoche", "QC MP"): with no name word beside it #: the first post-nominal is read as the name, because something #: has to be one, and only that word reports. + #: WHERE it is emitted is narrower than where the doubt exists, + #: and this is the boundary rather than an omission to be read + #: past. The emitters cover the trailing slot of a name, the + #: post-comma slot of a family-comma listing, and the extra + #: segments beyond it. They do NOT cover the trailing slot of the + #: GIVEN part after a family comma: "Doe, John MA" reads middle + #: ``MA`` and "Doe, John X.Y.Z." middle ``X.Y.Z.``, both silently, + #: as every 2.x release has; 1.4.0 read "Doe, John MA" as a + #: suffix. The fork is real there and nothing reports it; + #: decisions.md#S2 records it as open. SUFFIX_OR_NAME = "suffix-or-name" #: An input the title peel eats down to one last word which is #: itself title vocabulary still has to name somebody, so that @@ -497,6 +518,12 @@ class AmbiguityKind(StrEnum): #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so #: the tokens are the one the stray character ended up inside. + #: NOT a fork the parse called: it reports a shape the parse could + #: not recognize, which is the carve-out this enum's own docstring + #: names for this member and for COMMA_STRUCTURE. That docstring + #: said "each says so on its own member below" while only + #: COMMA_STRUCTURE's did; this sentence is the other half + #: (2026-09-18). #: Two cases leave that tuple empty: a character that lands in no #: token at all (inside a masked region), and an input with no #: alphanumeric content anywhere, which parses to an empty name -- diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index 6ce0be9d..e8945ece 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -122,11 +122,24 @@ so only EDGE periods come off and interior ones survive -- "Junior." matches here, "J.u.n.o.r." does not and stays name text ("John J.u.n.o.r." parses a family name, on both APIs). The example is deliberately not "J.u.n.i.o.r.", -which fails this lookup too and is a suffix anyway: an interior-period token -that no whole-token set claims goes to ``period_joined_vocab``, which splits -it on its periods and, no chunk being a title, calls the whole thing a -suffix if ANY chunk is suffix vocabulary -- and the chunk "i" is the Roman -numeral listed above. +which fails this lookup too: an interior-period token that no whole-token +set claims goes to ``period_joined_vocab``, which splits it on its periods +and, no chunk being a title, calls the whole thing a suffix if ANY chunk is +suffix vocabulary -- except where every chunk it matches is a SINGLE ASCII +CHARACTER, which since 2.4 retires rather than claims (decisions.md#S2): the +one-character suffix vocabulary is the Roman numerals and the digit "2" -- +ASCII is load-bearing there, since the seven single-character CJK +honorifics in THIS set (様, 殿, 氏, 군, 님, 씨, 양 -- three of them also +glued-honorific heads below) must KEEP their chunk claim ("J.씨" still +derives from 씨) -- and a word +built of single ASCII letters is not about generations. So "J.u.n.i.o.r." +no longer reaches this set's "i" entry as a generational claim; it is +read by POSITION instead, as an unlisted multi-chunk word +(``Policy.unlisted_dotted_suffixes``, default on), and with words to +spare it is still a suffix (rules.md#S3). A multi-character chunk match +is untouched -- "Msc.Ed." still derives from ``ed`` and reads as a +suffix outright, and "Lt.Gov." still derives a TITLE from its own +multi-character chunk match. So membership here is not the last word on a dotted token; the sentence is about this set's lookup alone. :data:`SUFFIX_ACRONYMS` is the set matched with every period removed, so it alone covers a multi-dot spelling: both @@ -135,8 +148,9 @@ DISJOINT (see the guard block at the bottom): a post-nominal belongs to one of them or the other, and which one holds it is what decides whether its multi-dot spelling reaches a whole-token lookup at all -- -``period_joined_vocab`` may still claim the token chunk by chunk, as the -"J.u.n.i.o.r." example above shows. +``period_joined_vocab`` may still claim the token chunk by chunk, as +"Msc.Ed." and "Lt.Gov." above show -- or read it by position instead, +as "J.u.n.i.o.r." now does. """ GLUED_HONORIFICS = frozenset({ diff --git a/pyproject.toml b/pyproject.toml index 2a95ea83..9959e2a1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -82,6 +82,18 @@ disallow_any_generics = true warn_return_any = true strict_equality = true +# strict_equality for the WHOLE package rather than the core modules +# above. A comparison between non-overlapping types is always False and +# is always a bug, and the facade layer -- which the block above does +# not name -- is exactly where a v1/v2 translation makes one likeliest. +# Measured 2026-09-18: clean over nameparser/ as it stands, so this +# costs nothing today and catches the next one. tests/ keep the default, +# where the check has real hits (a test comparing an off-contract value +# to a real one is doing its job). +[[tool.mypy.overrides]] +module = ["nameparser", "nameparser.*"] +strict_equality = true + [[tool.mypy.overrides]] module = ["tests.v2.*"] check_untyped_defs = true diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 5b6d131b..9f064d26 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -360,40 +360,47 @@ def _check_cjk_shape_purity(self) -> None: {"given": "John", "family": "Smith", "suffix": "Esq"}, notes="the suffix_words branch of the delimited-content " "escape (v1 parity, pinned live 2026-07-17)"), - Case("suffix_acronym_multidot_spelling_is_a_name_word", + Case("the_removed_esq_spelling_returns_as_a_credential", "John Smith E.S.Q.", - {"given": "John", "middle": "Smith", "family": "E.S.Q."}, - classification="fix(suffix-acronym-collisions)", - notes="a BEHAVIOR CHANGE rather than a fix -- a deliberate " - "2.x parity break, every release from 1.4.0 read " - "suffix 'E.S.Q.' here. 'esq' left SUFFIX_ACRONYMS " - "2026-09-08: Esquire is a contraction, not an " - "initialism, the entry arrived in the 2019 bulk " - "post-nominal import (af5bdab, #93), and the multi-dot " - "spelling was its only unique coverage. Same criterion " - "as the rai/cha rows above -- " - "decisions.md#suffix-acronym-collisions -- asked of the " - "machinery instead of a surname: does the entry " - "describe the WORD or the set's normalization. The " - "classification is a slug and not an issue number " - "because no issue asked for it; the bundle that " - "carried it is #489/#316"), + {"given": "John", "family": "Smith", "suffix": "E.S.Q."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="1.4.0 RESTORED a SECOND time, by a different route " + "than it lost it. 'esq' left SUFFIX_ACRONYMS 2026-09-08: " + "Esquire is a contraction, not an initialism, the entry " + "arrived in the 2019 bulk post-nominal import " + "(af5bdab, #93), and the multi-dot spelling was its " + "only unique coverage -- so for one bundle this read " + "family 'E.S.Q.' (a deliberate 2.x parity break, " + "decisions.md#suffix-acronym-collisions; same criterion " + "as the rai/cha rows above, asked of the machinery " + "instead of a surname: does the entry describe the " + "WORD or the set's normalization; the bundle that " + "carried the removal is #489/#316). #516's by-shape " + "class reads it again now, with no wordlist entry at " + "all: an unlisted dotted token of three single-letter " + "chunks joins the ambiguous class by SHAPE and the " + "words-to-spare count reads it as the credential it is", + shape=1), Case("suffix_acronym_multidot_after_a_family_comma", "Smith, E.S.Q.", {"given": "E.S.Q.", "family": "Smith"}, classification="parity", - notes="the other path the same removal moves, and the one " - "that RESTORES v1: with 'esq' in SUFFIX_ACRONYMS the " - "multi-dot spelling was a suffix piece, so the " - "post-comma segment held no name word and read suffix " - "'E.S.Q.' (2.0.0 through 2.2.0). Out of the set, it is " - "an ordinary name word and the walk's first non-title " - "piece is ALWAYS the given -- which is what 1.4.0 read " - "here, first 'E.S.Q.' / last 'Smith' (measured " - "2026-09-09), so this row is parity where its no-comma " - "sibling above is a parity BREAK. Same entry, opposite " - "directions, because v1 read the two paths " - "differently"), + ambiguities=("suffix-or-name",), + notes="the other path the same removal moved, once -- with " + "'esq' in SUFFIX_ACRONYMS the multi-dot spelling was a " + "suffix piece, so the post-comma segment held no name " + "word and read suffix 'E.S.Q.' (2.0.0 through 2.2.0); " + "out of the set it briefly restored 1.4.0 instead, " + "first 'E.S.Q.' / last 'Smith' (measured 2026-09-09). " + "#516's by-shape class now reads the SAME single " + "pre-comma word as a candidate for the ambiguous class " + "-- one word is never enough to flip the structure " + "(rules.md#C1), so the FIELDS stay at the 1.4.0 " + "reading, but the class is now considered, and the " + "consideration reports: the parity note above is about " + "the reading, not about whether a fork was called", + shape=2), Case("suffix_word_esq_still_reads_as_a_suffix", "John Smith Esq", {"given": "John", "family": "Smith", "suffix": "Esq"}, notes="the other half of the row above, and what the removal " @@ -425,18 +432,35 @@ def _check_cjk_shape_purity(self) -> None: "initials() too, for names whose roles are identical -- " "but only under the policies it runs, and middle_as_family " "is not one, so this row stays pinned here"), + # MOVED by #289, not deleted: 'MA' is written in capitals inside a + # mixed-case name, so it now leans CREDENTIAL and is taken with no + # words to spare -- the reserve/count this row used to pin no + # longer decides here, the written case does (decisions.md#S2). Case("ambiguous_surname_acronyms", "Jack MA", - {"given": "Jack", "family": "MA"}, - ambiguities=("suffix-or-name",), - notes="'ma'/'do' joined suffix_acronyms_ambiguous: with only " - "two pieces, 'one of them is a credential' is the less " - "likely reading, so the ambiguous acronym stays the " - "family name (v1 parity via its reserve_last)"), + {"given": "Jack", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name", "given-or-family"), + notes="the defect #289 reports: an all-caps member of the " + "ambiguous set inside a mixed-case name is written the " + "way a credential is written, so it is taken as one " + "even though peeling it leaves no family -- what " + "'Jack MD' has always done with an unambiguous one. " + "'Jack' is then the only name word left, which is also " + "what turns on GIVEN_OR_FAMILY. 1.4.0 read LAST 'MA' " + "here (family 'MA'), so this row BREAKS 1.4.0 parity " + "deliberately, in the direction the East Asian surname " + "wants (decisions.md#S2) -- corrected 2026-09-17, F3 " + "review finding: an earlier version of this note " + "wrongly claimed the opposite"), Case("ambiguous_surname_acronym_with_suffix", "Jack MA Jr", - {"given": "Jack", "family": "MA", "suffix": "Jr"}, - ambiguities=("suffix-or-name",), - notes="'Jr' peels first; 'MA' would then be the only piece " - "left beside the given name, so it stays family"), + {"given": "Jack", "suffix": "MA Jr"}, + classification="fix(#289)", + ambiguities=("suffix-or-name", "given-or-family"), + notes="'Jr' peels first as unambiguous vocabulary; 'MA' is " + "then the last piece, and the caps lean takes it with " + "no words to spare rather than leaving it the family " + "the old reserve kept (decisions.md#S2) -- both suffix " + "words render as one run"), Case("ambiguous_acronym_is_a_suffix_when_a_family_name_remains", "John Smith MA", {"given": "John", "family": "Smith", "suffix": "MA"}, @@ -490,16 +514,22 @@ def _check_cjk_shape_purity(self) -> None: "is the same shape and moves with it"), Case("removed_credential_loses_the_dotted_spelling_too", "John Smith C.H.A.", - {"given": "John", "middle": "Smith", "family": "C.H.A."}, - classification="fix(#342)", - notes="the removal reaches the DOTTED spelling through S3's " - "period fold, which strips the periods to reach the " - "entry -- so with 'cha' gone this reads family. The " - "contrast is 'John Smith R.A.I.', which still reads " - "suffix and is an accident of 'i' being a SUFFIX_WORDS " - "numeral rather than a survival of the vocabulary, " - "deliberately unpinned " - "(decisions.md#suffix-acronym-collisions)"), + {"given": "John", "family": "Smith", "suffix": "C.H.A."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="for one bundle the removal reached the DOTTED spelling " + "through S3's period fold, which strips the periods to " + "reach the entry -- so with 'cha' gone this read family " + "(the contrast was 'John Smith R.A.I.', which kept " + "reading suffix as an accident of 'i' being a " + "SUFFIX_WORDS numeral rather than a survival of the " + "vocabulary, decisions.md#suffix-acronym-collisions). " + "#516's by-shape class now reads BOTH the same way, by " + "POSITION rather than by any chunk claim -- three " + "single-letter chunks, none of them vocabulary, still " + "join the ambiguous class by shape and the words-to-" + "spare count reads this one a credential too", + shape=1), # -- #342: 'ba' is the other half of the same decision. BA is a # common credential and Ba a real surname (Vietnamese; Senegalese # Fula), which is the ma/Ma shape exactly, so it takes the @@ -516,18 +546,23 @@ def _check_cjk_shape_purity(self) -> None: "through 2.2.0 read suffix 'Ba' with no family name; " "1.4.0 read family 'Ba' unflagged, so this row " "restores 1.4.0's roles and adds the flag"), + # MOVED by #289, not deleted: 'BA' is written in capitals inside a + # mixed-case name, so it now leans CREDENTIAL and the post-comma + # slot takes it with one word to spare where the count alone would + # not -- back to what 2.0.0 through 2.2.0 read, though for a + # different reason (decisions.md#S2, Derek's own #289 comment: + # positive evidence outranks position here). Case("comma_ambiguous_acronym_ba", "Smith, BA", - {"given": "BA", "family": "Smith"}, - classification="parity", + {"family": "Smith", "suffix": "BA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), notes="the marking's comma cost, and it is 'Smith, Ed' " "arriving for a second word: with 'ba' ambiguous, S2 " "declines the post-comma credential and C1 reads it as " "the given name. 2.0.0 through 2.2.0 read family " - "'Smith', suffix 'BA'; 1.4.0 read given 'BA', which is " - "what this restores and why it classifies parity. The " - "row is here because the release note names the text -- " - "comma_ambiguous_acronym pins the fork and stays green " - "when 'ba' leaves the ambiguous subset"), + "'Smith', suffix 'BA'; 1.4.0 read given 'BA'. #289 moves " + "this row again: the caps lean now reads 'Smith, BA' the " + "way a business form is written"), Case("ba_is_a_suffix_when_a_family_name_remains", "John Smith BA", {"given": "John", "family": "Smith", "suffix": "BA"}, ambiguities=("suffix-or-name",), @@ -1198,6 +1233,1209 @@ def _check_cjk_shape_purity(self) -> None: "shape tag asserts the ARRANGEMENT, not a script " "(tools/differential/shapes.py)", shape=1), + # #289: a bare ambiguous credential acronym is read by the + # EVIDENCE the writing carries, and a name written in more than + # one case carries some. These rows pin the FORK, not the wordlist + # (mechanisms.md#VOCABULARY-EXERCISES-FORKS): the same two letters + # in the same slot, spelled five ways, with the one-case controls + # that must not move beside them. + Case("caps_ambiguous_leans_credential", "Jack MA", + {"given": "Jack", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name", "given-or-family"), + notes="the defect. An all-caps member of the ambiguous set " + "inside a mixed-case name is written the way a " + "credential is written, so it is taken as one even " + "though peeling it leaves no family -- which is what " + "'Jack MD' has always done with an unambiguous one. " + "1.4.0 read last 'MA' here, so this breaks v1 parity " + "deliberately", + shape=1), + Case("caps_ambiguous_leans_credential_with_a_period", "Jack MA.", + {"given": "Jack", "suffix": "MA."}, + classification="fix(#289)", + ambiguities=("suffix-or-name", "given-or-family"), + notes="the lean reads CASE, not periods: a single trailing " + "period is the abbreviation shape any word can wear " + "(rules.md#S2) and changes nothing here, where the " + "capitals have already spoken", + shape=1), + Case("title_case_ambiguous_leans_surname_two_words", "Jack Ma", + {"given": "Jack", "family": "Ma"}, + ambiguities=("suffix-or-name",), + notes="the contrast that makes the row above a fork rather " + "than a wordlist entry: Title case is how a surname is " + "written, and this row is unchanged in every release", + shape=1), + Case("one_case_upper_ambiguous_takes_the_count", "JACK MA", + {"given": "JACK", "family": "MA"}, + ambiguities=("suffix-or-name",), + notes="the one-case control for 'Jack MA': all-caps against " + "all-caps is no contrast at all, so nothing leans and " + "rules.md#S2's words-to-spare count decides alone -- " + "two pieces, so the acronym stays the family name", + shape=1), + Case("one_case_lower_ambiguous_takes_the_count", "jack ma", + {"given": "jack", "family": "ma"}, + ambiguities=("suffix-or-name",), + notes="the lowercase spelling of the row above; both cases " + "read alike, which is the point of asking about " + "evidence rather than about capitals", + shape=1), + Case("title_case_ambiguous_leans_surname_with_words_to_spare", + "John Smith Ma", + {"given": "John", "middle": "Smith", "family": "Ma"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the lean's other direction, and the one that costs a " + "credential rather than a surname: with words to spare " + "the count read 'Ma' as a suffix, and a Title-case " + "word beside 'John Smith' is how a surname is written. " + "1.4.0 read suffix 'Ma', so this is a deliberate " + "parity break in the direction the East Asian surname " + "wants (decisions.md#S2)", + shape=1), + Case("one_case_upper_ambiguous_with_words_to_spare", + "JOHN SMITH MA", + {"given": "JOHN", "family": "SMITH", "suffix": "MA"}, + ambiguities=("suffix-or-name",), + notes="the one-case control for the row above: no evidence, " + "so the count decides and three pieces make the " + "acronym a credential, exactly as before 2.4", + shape=1), + Case("mixed_case_caps_ambiguous_with_words_to_spare", + "john smith MA", + {"given": "john", "family": "smith", "suffix": "MA"}, + ambiguities=("suffix-or-name",), + notes="the lean and the count AGREE here, which is why the " + "row does not move: capitals against lowercase lean " + "credential, and three pieces read credential anyway", + shape=1), + Case("one_case_ambiguous_surname_is_untouched", "ANH DO", + {"given": "ANH", "family": "DO"}, + ambiguities=("suffix-or-name",), + notes="the name this whole design must not break: an " + "all-caps Vietnamese surname in an all-caps name. One " + "case, so nothing leans, and the count keeps the " + "family -- the reading 2.0 shipped and 1.4.0 had", + shape=1), + Case("one_case_ambiguous_surname_behind_a_particle", + "anh van do", + {"given": "anh", "family": "van do"}, + notes="its lowercase, particle-bearing twin: the particle " + "chain takes 'do' as part of the surname before any of " + "this is asked", + shape=1), + Case("a_title_and_one_ambiguous_word_is_still_a_name", "Mr MA", + {"title": "Mr", "family": "MA"}, + notes="the floor the lean does not move, measured " + "2026-09-15: the trailing peel's walk starts after the " + "leading title run, so one piece is all it sees and " + "rules.md#S2's two-piece floor refuses it. A lean that " + "reached below the floor would read this as a title " + "with a credential and no name at all -- and nothing " + "is reported, because no fork was taken", + shape=1), + Case("the_lean_reaches_the_post_comma_slot", "Smith, MA", + {"family": "Smith", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="Derek's own reading of #289: positive evidence " + "outranks position, so the credential lean fires with " + "ONE word before the comma where the words-to-spare " + "count would not. The first report of the comma's OWN " + "decision rides with it -- C2's structural flag already " + "reports on the comma path, but reports what the parse " + "could not recognize, not a fork it called " + "(rules.md#C1's exception, scoped to this class; P6's " + "attachment fork has reported on a family-comma path " + "since 2.3, e.g. 'Berg, Jan vd')", + shape=2), + Case("the_surname_lean_keeps_the_post_comma_given", "Smith, Ma", + {"given": "Ma", "family": "Smith"}, + ambiguities=("suffix-or-name",), + notes="the contrast: Title case leans surname, so the word " + "stays the given name exactly as it did -- and the " + "call is reported all the same, both directions of one " + "fork being worth telling the caller about", + shape=2), + Case("the_comma_count_is_of_name_words", "Smith Jr., MA", + {"family": "Smith", "suffix": "Jr., MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="rules.md#C1's count for this class is of NAME words, " + "not of words: two tokens and one name here, so the " + "structure does not flip and the family survives. " + "Counting tokens read it as given 'Smith' with suffix " + "'Jr., MA' -- no reading of this string puts the " + "family anywhere but where this row does " + "(decisions.md#S2)", + shape=2), + Case("the_by_shape_class_never_reaches_is_wholly_suffix", + "Smith Jr., A.B.", + {"given": "A.B.", "family": "Smith", "suffix": "Jr."}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="FULL PARITY on the 1.4.0 wheel -- first 'A.B.', last " + "'Smith', suffix 'Jr.'; only the report is new, like " + "its report-only siblings above. Also the control that " + "pins the review-round fix: an EARLIER version of " + "`is_wholly_suffix` admitted a by-shape member " + "unconditionally, and combined with C1's own legacy " + "TOKEN-count disjunct in `_segment.py` that flipped " + "this to given 'Smith' with a self-contradicting " + "'holds 1 name words' report. `ambiguous_class_candidate` " + "is the only reader the by-shape class has at the " + "comma form, and it declines here exactly as the " + "listed class does above -- one name word, so `A.B.` " + "stays the given and `Jr.` its own suffix", + shape=2), + Case("two_name_words_read_the_listed_acronym_as_a_credential", + "John Smith, MA", + {"given": "John", "family": "Smith", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="1.4.0 RESTORED: v1 read first John, last Smith, " + "suffix MA and 2.0 through 2.3 read given 'MA' with " + "the whole name in `family`. The comma structure moves " + "with the class, so this is the SUFFIX_COMMA form " + "again", + shape=3), + Case("the_comma_count_reaches_the_listed_set_title_case", + "John Smith, Ed", + {"given": "John", "family": "Smith", "suffix": "Ed"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="one rule for the whole ambiguous class rather than " + "two: where case is silent -- 'Ed' leans SURNAME here, " + "and the lean does not reach the structure -- the " + "count of name words before the comma decides, and two " + "of them make it a credential. 1.4.0 read suffix 'Ed' " + "too, so this restores parity rather than opening " + "distance from it (decisions.md#S2, 2026-09-15)", + shape=3), + Case("the_comma_count_reaches_the_listed_set_lower", + "john smith, ma", + {"given": "john", "family": "smith", "suffix": "ma"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the one-case spelling of the row above: no evidence " + "either way, so the count decides and 1.4.0's suffix " + "reading comes back", + shape=3), + Case("the_comma_count_reaches_the_listed_set_upper", + "JOHN SMITH, MA", + {"given": "JOHN", "family": "SMITH", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the all-caps spelling, and the reason the count has " + "to reach the listed set at all: an all-caps name " + "carries no lean, and leaving it to the lean alone " + "would read this one way and its mixed-case twin " + "another. 1.4.0 parity restored", + shape=3), + Case("two_name_words_before_the_comma_at_two_words", + "Davis Royce, Ed", + {"given": "Davis", "family": "Royce", "suffix": "Ed"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the two-word twin of 'Royce, Ed' below, and the case " + "row item 5 of decisions.md#S2 turns on -- no corpus " + "name has this shape, so this row carries the whole " + "weight of that decision", + shape=3), + Case("one_name_word_before_the_comma_is_never_enough", + "Royce, Ed", + {"given": "Ed", "family": "Royce"}, + ambiguities=("suffix-or-name",), + notes="the control: one name word before the comma leaves " + "nothing to spare, 'Ed' leans surname anyway, and the " + "row is unchanged in every release. The pair above and " + "below it is what makes the count visible", + shape=2), + Case("a_declined_ambiguous_pick_stops_the_walk", + "abdul Smith Jr Ma", + {"given": "abdul Smith", "middle": "Jr", "family": "Ma"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="an ACCEPTED cost, pinned rather than repaired " + "(decisions.md#S2): the surname lean breaks the peel " + "AT 'Ma', so the unambiguous 'Jr' in front of it is " + "never reached and becomes a name word. The walk stops " + "at the declined pick rather than continuing past it, " + "and a name-leaning acronym blocking a genuine suffix " + "behind it is the shape that costs", + shape=1), + Case("a_tail_segment_of_leaning_credentials_is_a_run", + "Steven Hardman, MD, DO, DDS", + {"given": "Steven", "family": "Hardman", "suffix": "MD, DO, DDS"}, + classification="fix(#289)", + notes="the one place this design QUIETS a report rather than " + "adding one: 'DO' is a listed ambiguous acronym " + "written in capitals inside a mixed-case name, so the " + "third comma segment is wholly suffix and rules.md#C2 " + "stops flagging it. The FIELDS do not move -- a " + "quieted flag is invisible to a reader sweeping for " + "new reports, which is why it has a row of its own", + shape=3), + # Quality-review finding, 2026-09-17: "one name never reports + # twice" was wrong as a blanket claim -- what is true is that one + # DECISION never reports twice. Two DIFFERENT ambiguous acronyms + # at two different forks each report once: the trailing peel over + # the pre-comma part ('MA', credential lean, no words to spare) + # and the comma structure's own flip ('Ed', two pre-comma name + # words -- 'Smith' and 'MA' both count, since a bare undotted + # ambiguous acronym is not suffix vocabulary to `name_word_count` + # either). + Case("two_different_forks_each_report_once", "Smith MA, Ed", + {"given": "Smith", "suffix": "MA, Ed"}, + classification="fix(#289)", + ambiguities=("suffix-or-name", "suffix-or-name", + "given-or-family"), + notes="'Smith MA' reads two name words, flipping the comma " + "structure so 'Ed' joins the credential run; peeling " + "'Smith MA' positionally then peels 'MA' too (caps " + "lean, no words to spare), leaving 'Smith' the only " + "name word, which is also what turns on " + "GIVEN_OR_FAMILY. Three reports, none of them the same " + "fork reporting twice"), + Case("a_caseless_script_wrote_no_contrast", "毛泽东, MA", + {"given": "MA", "family": "毛泽东"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the LEAN is inert by construction: is_one_case answers " + "True for text that has only one case, so nothing leans " + "and the count decides -- one word before the comma, so " + "the acronym stays the given name, as in every release. " + "But the REPORT is not lean-gated (F4/F5 review finding, " + "2026-09-17): it tracks the fork being consulted, " + "exactly as the trailing slot always has, so this comma " + "form is read positionally and still reports which way " + "it went -- an earlier round wrongly excluded this row", + tolerated=True), + # MEASURED, not the plan's prediction: unlike its single-token + # Chinese twin above, '마틴 킹' is TWO whitespace tokens, so item + # 5's name-word count -- uniform across the whole ambiguous class, + # case silent or not -- sees two name words before the comma and + # flips the structure exactly as 'JOHN SMITH, MA' does. The comma + # form then reads segment 0 the way a NO_COMMA name would (the + # (a-lazy) mechanism's own tradeoff: the flip is decided before + # script_segment, so the Hangul surname split runs over '마틴 킹' + # positionally rather than over the untouched pre-comma text) -- + # the same shape 'Smith 김민준씨, MA' shows in the spec's decision + # table. Not a caseless-script exemption: item 5 answers "where " + # case is silent" by the SAME count the rest of item 5 uses, and a + # two-word Hangul name is silent about case but not about how many + # name words it has. + Case("a_caseless_script_wrote_no_contrast_hangul", "마틴 킹, MA", + {"given": "틴", "middle": "킹", "family": "마", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="MEASURED 2026-09-17, not the spec's older prediction " + "(given MA, family '마틴 킹', unchanged): dated before " + "item 5's name-word-count extension was decided " + "(2026-09-15) reached the listed set in ANY case. Two " + "name words before the comma flip the structure here " + "exactly as they do for 'JOHN SMITH, MA', and the flip " + "precedes script_segment's Hangul surname split, which " + "then runs over '마틴 킹' as if no comma had stood " + "there at all", + tolerated=True), + # F2 review finding, 2026-09-17: the SAME mechanism reaches Japanese + # too, by the same two-token count -- '田中 太郎' is two whitespace + # tokens, not one, so it is not '毛泽东, MA's twin (a single token) + # but '마틴 킹, MA's. 1.4.0 read both '마틴 킹, MA' and + # '田中 太郎, MA' as suffix MA (measured on the wheel), so the + # suffix half of this move is a PARITY RESTORATION, not a fresh + # deviation -- only the pre-comma surname split (given '太郎', + # family '田中') is #271/#272's pre-existing positional behavior, + # unrelated to #289. + Case("a_caseless_script_wrote_no_contrast_japanese", "田中 太郎, MA", + {"given": "太郎", "family": "田中", "suffix": "MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="MEASURED 2026-09-17: two name words before the comma " + "flip the structure exactly as '마틴 킹, MA' and " + "'JOHN SMITH, MA' do, and the flip again precedes " + "script_segment, so the family-first Han split runs " + "over '田中 太郎' positionally. Docs/design's older " + "'a caseless script is inert by construction' bullet " + "is true of the LEAN alone; item 5's name-word count " + "is orthogonal to case and reaches this row too", + tolerated=True), + # #516's dotted half: an UNLISTED token of two or more + # period-separated chunks joins the ambiguous credential class BY + # SHAPE, and the words-to-spare count then reads it. These rows + # pin the FORK -- the same token in the two positions the count + # separates, with the chunk-claim names that must NOT move beside + # them (mechanisms.md#VOCABULARY-EXERCISES-FORKS). + Case("unlisted_dotted_reads_by_position_with_words_to_spare", + "John Smith X.Y.Z.", + {"given": "John", "family": "Smith", "suffix": "X.Y.Z."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the periods are the signal and the position decides: " + "three pieces, so the unlisted acronym is the " + "credential. 1.4.0 read last 'X.Y.Z.', so this breaks " + "parity deliberately -- the parking-lot entry of " + "decisions.md#suffix-acronym-collisions is what it " + "closes", + shape=1), + Case("unlisted_dotted_is_case_blind", "john smith x.y.z.", + {"given": "john", "family": "smith", "suffix": "x.y.z."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the control that separates this switch from #289's " + "lean: case is irrelevant here, the PERIODS being the " + "signal, so the all-lower spelling reads as the " + "mixed-case one does", + shape=1), + Case("unlisted_dotted_without_words_to_spare_is_the_family", + "Jack X.Y.Z.", + {"given": "Jack", "family": "X.Y.Z."}, + ambiguities=("suffix-or-name",), + notes="the other half of the count, and the row is unchanged " + "in its FIELDS: two pieces, so the shape loses to the " + "surname reading. What is new is the report -- the " + "fork was always there and was called silently", + shape=1), + Case("unlisted_dotted_of_any_chunk_length", "John Smith B.Tech.", + {"given": "John", "family": "Smith", "suffix": "B.Tech."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="chunk LENGTH is not the test: two chunks, one of them " + "four letters, and the Indian degree reads as the " + "credential it is", + shape=1), + Case("the_s3_boundary_example_moves", "John Smith Q.W.E.R.T.", + {"given": "John", "family": "Smith", "suffix": "Q.W.E.R.T."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="rules.md#S3's own boundary example, which said this " + "shape stays a name. It moves, and the rule moves with " + "it", + shape=1), + Case("the_roman_chunk_accident_retires_narrowly", "Jack X.Y.I.", + {"given": "Jack", "family": "X.Y.I."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="1.4.0 RESTORED. The chunk 'i' is a roman numeral in " + "the suffix vocabulary, so this read as a generational " + "suffix -- by accident, the fork being about " + "generations and the word being nothing of the kind. " + "Retired where every matched chunk is a single ASCII " + "character -- the roster this retirement is scoped to " + "is asserted, not just described, in " + "test_vocab.test_period_joined_vocab_retires_the_single_" + "character_chunk", + shape=1), + Case("a_retired_chunk_claim_still_reads_by_position", + "John Smith R.A.I.", + {"given": "John", "family": "Smith", "suffix": "R.A.I."}, + ambiguities=("suffix-or-name",), + notes="the pair to the row above, and the reason the " + "retirement is safe: with words to spare the shape " + "reads the same token the accident read, so the FIELDS " + "do not move and only the cause does. The report is " + "what makes the new cause visible", + shape=1), + Case("a_multi_letter_roman_chunk_also_reads_by_position", + "John Smith J.u.n.i.o.r.", + {"given": "John", "family": "Smith", "suffix": "J.u.n.i.o.r."}, + ambiguities=("suffix-or-name",), + notes="the six-chunk twin of the row above: no chunk here is " + "vocabulary at all (this is not 'junior' the suffix " + "word, glued letter by letter), so the shape read this " + "one before the retirement existed and reads it the " + "same way after -- the FIELDS never moved, only the " + "report is new", + shape=1), + Case("the_comma_structure_moves_with_the_shape_class", + "John Smith, A.B.", + {"given": "John", "family": "Smith", "suffix": "A.B."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="two NAME words before the comma read the part after " + "it as the credential run, for the by-shape half " + "exactly as for the listed one (rules.md#C1)", + shape=3), + Case("one_word_before_the_comma_keeps_the_given", "Smith, A.B.", + {"given": "A.B.", "family": "Smith"}, + ambiguities=("suffix-or-name",), + notes="its control: one name word before the comma is never " + "enough, so the dotted token stays the given name and " + "the structure does not move. Unchanged in every " + "release; the report is new", + shape=2), + Case("whole_token_vocabulary_wins_over_the_shape", + "Smith, A.B.C.", + {"family": "Smith", "suffix": "A.B.C."}, + notes="'abc' IS a suffix acronym, so the whole-token lookup " + "settles this before any shape reading and the count " + "never runs. The boundary between #516's shape class " + "and the vocabulary it does not touch", + shape=2), + Case("a_leading_dotted_run_is_untouched", "X.Y.Z. Smith", + {"given": "X.Y.Z.", "family": "Smith"}, + notes="rules.md#S2 already says a suffix never opens the " + "string, and the shape class is read only by the " + "trailing peel and the post-comma slot -- so every " + "leading dotted run, 'J.R.R. Tolkien' included, is " + "untouched by this switch", + shape=1), + Case("a_leading_dotted_run_of_three_chunks_is_untouched", + "J.R.R. Tolkien", + {"given": "J.R.R.", "family": "Tolkien"}, + notes="the three-chunk twin of the row above: 'j', 'r' and " + "'r' claim nothing either, and the leading slot never " + "asks the shape class regardless", + shape=1), + Case("a_multi_character_chunk_claim_survives", "Doe, John Msc.Ed.", + {"given": "John", "family": "Doe", "suffix": "Msc.Ed."}, + notes="the narrow retirement's protected control: the chunk " + "'ed' is two characters, so the chunk rule still fires " + "and this real credential is not handed to the shape " + "class. The WIDE retirement -- any chunk match yielding " + "to the shape -- would have read 'Msc.Ed.' as a middle " + "name, which is what rejected it (decisions.md#S2)", + shape=2), + Case("a_glued_honorific_chunk_is_not_ascii", "J.씨", + {"given": "J.", "suffix": "씨"}, + notes="the reason the retirement says ASCII and not just " + "single-character: '씨' is one character and MUST keep " + "its chunk claim, the honorific peel resting on it", + tolerated=True), + Case("a_delimited_dotted_token_keeps_its_clause_reading", + "Bridge (1.4)", + {"family": "Bridge", "nickname": "1.4"}, + notes="delimited content is decided by the clause escape " + "rather than at the trailing slot -- so the shape " + "class is not offered to a token that already carries " + "a role, and this row gains neither a reading nor a " + "report. Cannot exercise the `token.role is None` " + "guard on its own, though: a digit chunk never reaches " + "the shape verdict at all (period_joined_vocab's own " + "alphabetic gate) -- the row beside it is the guard's " + "real control", + shape=1), + Case("the_role_guard_is_the_real_delimited_control", "Bridge (A.B)", + {"family": "Bridge", "nickname": "A.B"}, + notes="unlike the digit row above, 'A.B' WOULD reach the " + "shape verdict were it not for classify's `token.role " + "is None` guard -- without it this nickname gains " + "SHAPE_ACRONYM_TAG and, with the switch on, " + "'vocab:suffix-ambiguous', and classify's own nickname " + "check misreports SUFFIX_OR_NICKNAME on a token the " + "clause escape already decided (measured regression, " + "#516 review round). This row is what actually pins " + "the guard load-bearing", + shape=1), + Case("a_bare_digit_chunk_is_never_an_acronym_by_shape", + "John Smith 1.4", + {"given": "John", "middle": "Smith", "family": "1.4"}, + notes="#516 review round: the shape gate requires every chunk " + "to be ALPHABETIC, not merely unclaimed -- '1' and '4' " + "are chunks nothing spells as letters, so this stays " + "name material exactly as 'Bridge (1.4)' does, and " + "unlike the delimited control this one was never inside " + "a clause at all. Unchanged from dfb3170, and the gap " + "the delimited control alone did not cover", + shape=1), + Case("a_bare_digit_chunk_after_the_comma_is_never_an_acronym", + "John Smith, 1.4", + {"given": "1.4", "family": "John Smith"}, + notes="the comma-form twin of the row above: one word before " + "the comma is never enough regardless, but the point " + "this row pins is that the digit chunk never even " + "becomes a CANDIDATE for the class -- no report, " + "unchanged from dfb3170", + shape=2), + Case("a_leading_shape_token_never_reports_only_the_trailing_one", + "J.A. K.D.", + {"given": "J.A.", "family": "K.D."}, + ambiguities=("suffix-or-name",), + notes="'K.D.' is a considered pick -- the peel records it " + "even though k == 2 declines to take it, so the FIELDS " + "stay the two-piece positional read while the fork is " + "reported, exactly as a declined LISTED pick always " + "has been. 'J.A.' gets no such consideration: the peel " + "walks from the END, so a LEADING shape-admitted token " + "never reaches it and this reports exactly once, for " + "'K.D.' alone", + shape=1), + Case("an_initialless_script_glued_into_periods_is_not_this_shape", + "John Smith 田.中.", + {"given": "John", "middle": "Smith", "family": "田.中."}, + notes="the CJK control for #516's shape verdict, on the same " + "reasoning is_title_shaped already gives H2's own " + "period-abbreviation inference (#323): a script with " + "no period abbreviations at all has nothing for " + "interior periods to abbreviate, so this stays name " + "material exactly as it did at dfb3170 -- no reading " + "and no report", + tolerated=True), + # The whole-PR review round, 2026-09-18. Three groups: the fork + # the prefix chain swallowed, the honorific peel that stopped + # asking for the lean, and the tail segment that started flagging + # its own new reading. Every row here is a REPORT moving, not a + # role -- which is exactly why none of them had a row before. + Case("the_chain_reports_the_acronym_it_takes", + "John van der Berg Ma", + {"given": "John", "family": "van der Berg Ma"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="rules.md#S2 says either reading carries the flag, and " + "this reading carried none. assign reports from the " + "peel's picks and a pick reaches it only as a LONE " + "piece, so once the prefix chain merged 'Ma' into the " + "family piece the token assign would have reported on " + "no longer existed. The roles are unchanged and were " + "never in doubt -- 'Ma' is Title-case in a mixed-case " + "name, so #289's lean declines it; what the review " + "round restored is the report, emitted at the chain's " + "own merge (mechanisms.md#AMBIGUITY-AT-THE-DECISION-" + "SITE)", + shape=1), + Case("the_chain_reports_at_one_particle_too", "John de Ma", + {"given": "John", "family": "de Ma"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the shortest spelling of the row above: one particle, " + "one chained acronym. Kept beside it because the chain " + "reaches the merge by a different route here -- 'de' " + "takes the single following piece rather than a run -- " + "and the emitter's `j > k + 1` floor is what both have " + "to clear", + shape=1), + Case("the_chain_reports_the_by_shape_half_too", + "John van der Berg X.Y.Z.", + {"given": "John", "family": "van der Berg X.Y.Z."}, + policy=Policy(unlisted_dotted_suffixes=False), + ambiguities=("suffix-or-name",), + classification="fix(#516)", + notes="the BY-SHAPE half of the chain emitter's two-tag " + "test, unpinned until the verification round " + "(2026-09-18): with the listed tag alone, all 8624 " + "tests still passed. The switch is what makes the " + "half reachable -- at the default, classify writes " + "BOTH tags, so the listed tag answers for this name " + "too and the shape tag's absence is invisible. Off, " + "only the shape tag is written, and dropping it from " + "`_AMBIGUOUS_CREDENTIAL_TAGS` silences this row, " + "'Freiherr von Berg X.Y.I.' and 'John van Berg A.B.' " + "at once"), + Case("a_particle_that_is_also_an_acronym_reports_nothing", + "anh van mc", + {"given": "anh", "family": "van mc"}, + notes="the control the emitter above must NOT claim, and the " + "reason it asks `prefix(j - 1)` last. 'mc' is a " + "particle as well as suffix vocabulary, so the chain's " + "PARTICLE run takes it -- P4's reading and P6's fork, " + "not S2's -- and 1.4.0 read it this way in silence. " + "'anh van do' is the shipped twin (its own row above) " + "and this is its unambiguous-vocabulary sibling, which " + "the peel never even considers", + shape=1), + Case("the_glued_honorific_peels_behind_a_leaning_credential", + "Kim김민준씨, MA", + {"family": "Kim김민준", "suffix": "씨, MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="script_segment asked `is_wholly_suffix` of the " + "post-comma run WITHOUT the case fact segment had just " + "recorded, so 'MA' read as name material, the run was " + "scanned, its only peel site was 'MA' itself -- which " + "ends in no listed tail -- and the person's own 씨 went " + "unpeeled (family 'Kim김민준씨'). 'Kim김민준씨, PhD' " + "peeled all along, so one name in two credential " + "spellings parsed two ways. Passing the fact is the " + "whole fix (review round, 2026-09-18)", + tolerated=True), + Case("the_glued_honorific_peel_reads_the_lean_not_the_word", + "Jo김민준씨, DO", + {"family": "Jo김민준", "suffix": "씨, DO"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="a second member of the ambiguous set behind the same " + "comma, so the row above pins the mechanism rather " + "than the word 'MA' " + "(mechanisms.md#VOCABULARY-EXERCISES-FORKS is about " + "not doing this per entry; 'DO' is here because it is " + "ALSO particle vocabulary, which nothing else on this " + "path exercises)", + tolerated=True), + Case("the_glued_honorific_peel_behind_a_title_and_a_lean", + "Dr. 김민준씨, MA", + {"title": "Dr.", "family": "김민준", "suffix": "씨, MA"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="the title is what makes this name MIXED-case at all " + "-- Hangul is caseless, so '김민준씨, MA' on its own is " + "one case, leans nothing and keeps today's reading " + "(given 'MA', family '김민준씨', unpeeled). Two rows, " + "one difference, and the difference is the case " + "contrast rather than the honorific", + tolerated=True), + Case("a_tail_segment_of_by_shape_credentials_is_not_flagged", + "John Smith, MD, R.A.I.", + {"given": "John", "family": "Smith", "suffix": "MD, R.A.I."}, + classification="fix(#516)", + notes="rules.md#S3's narrow roman retirement moved 'R.A.I.' " + "out of the vocabulary verdict and into the shape " + "class, and segment's tail-segment test reads " + "`is_wholly_suffix`, which is blind to that class by " + "design -- so a third segment the parser itself reads " + "as a credential run gained a COMMA_STRUCTURE flag 2.3 " + "did not raise. A report about the parser's own new " + "reading rather than about the name (review round, " + "2026-09-18). Roles unchanged at every policy: a tail " + "segment is consumed as suffix either way", + shape=3), + Case("the_by_shape_tail_segment_is_flagged_with_the_switch_off", + "John Smith, MD, R.A.I.", + {"given": "John", "family": "Smith", "suffix": "MD, R.A.I."}, + policy=Policy(unlisted_dotted_suffixes=False), + ambiguities=("comma-structure",), + notes="the negative control for the row above, and the " + "reason the test is policy-sensitive rather than a " + "blanket quiet: with the switch off 'R.A.I.' is name " + "material, is no candidate for the class, and the " + "segment genuinely is beyond the recognized comma " + "structures. Same fields, opposite report", + classification="fix(#516)"), + Case("a_listed_tail_segment_keeps_its_flag_where_nothing_leans", + "John Smith, MD, Ma", + {"given": "John", "family": "Smith", "suffix": "MD, Ma"}, + ambiguities=("comma-structure",), + notes="the other boundary of the same test: the quiet is the " + "BY-SHAPE half's alone. A LISTED member reaches this " + "reading through the case lean " + "('Steven Hardman, MD, DO, DDS', " + "test_segment.py's own negative control), and 'Ma' " + "leans the other way, so the flag stands exactly as it " + "did at 1f78bef. Folding membership into the test " + "instead would have silenced the lean's own control", + shape=3), + Case("the_caps_shape_never_reaches_a_tail_segment", + "John Smith, MD, XYZ", + {"given": "John", "family": "Smith", "suffix": "MD, XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("comma-structure",), + classification="fix(#516)", + notes="the SECOND way C2's quiet is narrow, and the half its " + "examples did not pin until the verification round " + "(2026-09-18). The shape a tail segment is recognized " + "by is the DOTTED one alone, so turning the caps " + "switch on does not quiet 'XYZ' here -- the flag is " + "the same one the default policy raises. Folding " + "`caps_shape_candidate` into the class run would make " + "this row fail"), + # #516's all-caps half, and it is OPT-IN. The rows come in pairs: + # the same name under the DEFAULT policy, where nothing moves and + # nothing is reported, and under Policy(unlisted_caps_suffixes= + # True), where the shape reads. The French and Korean names are + # why the default is off (mechanisms.md#VOCABULARY-EXERCISES-FORKS + # -- each pair pins the switch, not the words). A row that sets + # the non-default policy carries no `shape=` tag: the contract + # corpus (build_shapes_corpus.py) keys only on (shape, text), with + # no policy of its own, so admitting one of these texts under a + # shape id would have compare.py diff it against the released + # wheels under the DEFAULT policy -- the wrong question for a row + # whose point IS the non-default policy. + Case("caps_surname_is_a_family_name_by_default", "Jean DUPONT", + {"given": "Jean", "family": "DUPONT"}, + notes="the writing convention the default protects: French " + "records write the surname in capitals, and shape " + "cannot tell that from a credential. 1.4.0's reading, " + "2.3's reading, and the reading at this default", + shape=1), + Case("caps_surname_is_swallowed_with_the_switch_on", + "Jean Pierre DUPONT", + {"given": "Jean", "family": "Pierre", "suffix": "DUPONT"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the cost of the switch, pinned so nobody turns it on " + "without meeting it: three name words, so the count " + "reads the capitalised surname as a credential and the " + "family becomes 'Pierre'. This row is the argument for " + "the default being off (decisions.md#S2)"), + # #516 review round (quality-review finding): the TWO-word shape + # ('Jean DUPONT') is the row `_policy.py`'s own docstring needed + # and did not have -- turning the switch on reports a genuine + # candidate even where one word before the credential declines + # the structure flip, matching 'the_caps_comma_count_declines_at_ + # one_word' (the comma form's own twin of this exact guarantee). + Case("caps_surname_reports_but_does_not_move_at_two_words", + "Jean DUPONT", + {"given": "Jean", "family": "DUPONT"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="one name word is never enough to spend the credential " + "reading, so the family stays 'DUPONT' -- but the fork " + "was genuinely considered and declined, and reports so " + "even though nothing moved"), + Case("caps_surname_default_reading_at_three_words", + "Jean Pierre DUPONT", + {"given": "Jean", "middle": "Pierre", "family": "DUPONT"}, + notes="the same name at the default, which is the half a " + "reader of the row above needs: nothing moves and " + "nothing is reported, so a caller who never sets the " + "switch never meets that cost", + shape=1), + Case("caps_korean_surname_is_a_family_name_by_default", + "Minjun KIM", + {"given": "Minjun", "family": "KIM"}, + notes="the second convention the default protects: Korean " + "records write the family name in capitals to mark " + "which of two words it is, which is the opposite of a " + "credential", + shape=1), + Case("unlisted_caps_reads_by_position_with_the_switch_on", + "John Smith XYZ", + {"given": "John", "family": "Smith", "suffix": "XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="what the switch buys: the shape reads, and the " + "words-to-spare count decides it exactly as it decides " + "the listed set"), + Case("unlisted_caps_is_silent_at_the_default", "John Smith XYZ", + {"given": "John", "middle": "Smith", "family": "XYZ"}, + notes="the same name at the default: name material, and NO " + "report -- the one place this design emits nothing " + "where a fork could be said to exist, because the " + "reading was never on offer (#516)", + shape=1), + Case("the_caps_comma_count_needs_two_name_words", + "John Smith, XYZ", + {"given": "John", "family": "Smith", "suffix": "XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the comma structure moves with this half too, on the " + "same NAME-word count"), + Case("the_caps_comma_count_declines_at_one_word", "Smith, XYZ", + {"given": "XYZ", "family": "Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="its control, switch and all: one name word before the " + "comma is never enough, so the capitalised word stays " + "the given name"), + Case("one_case_input_never_reaches_the_caps_switch", + "JOHN SMITH XYZ", + {"given": "JOHN", "middle": "SMITH", "family": "XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the one-case control, and it needs the switch ON to " + "mean anything: capitals against capitals are no " + "contrast, so the shape never fires and the row is " + "unchanged with the switch either way"), + Case("suffix_vocabulary_never_reaches_the_caps_switch", + "John Smith MC", + {"given": "John", "family": "Smith", "suffix": "MC"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="UNLISTED is the load-bearing word: 'mc' is suffix " + "vocabulary, so the whole-token lookup claims it before " + "any shape reading and this row reads the same with " + "the switch on or off"), + Case("the_caps_comma_count_reaches_a_multi_word_run", + "John Smith, LEED AP", + {"given": "John", "family": "Smith", "suffix": "LEED AP"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the first time this class reaches the comma form as " + "MORE than one token: 'LEED' and 'AP' are two separate " + "all-caps words, and every token in the run must be a " + "candidate for the run itself to be one -- rules.md#C1's " + "`deviates: #291` line comes true ONLY under this " + "switch. At the DEFAULT this exact text still reads " + "given 'LEED', middle 'AP', family 'John Smith' -- the " + "deviation stands there unchanged -- so nothing in " + "this arc may remove rules.md#C1's `deviates: #291` " + "marker on this switch's account; the switch only " + "narrows what makes the deviation true"), + Case("the_caps_comma_multi_word_run_declines_at_one_word", + "Smith, LEED AP", + {"given": "LEED", "middle": "AP", "family": "Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="the one-pre-comma-word twin of the row above: the run " + "is still a candidate (so the fork still reports, via " + "assign's family-comma emitter reading the first " + "post-comma piece's tag) but one name word before the " + "comma is never enough to flip the structure, item 5's " + "count doing its job on real data"), + # #516 review round: LISTED members must keep #289's lean with + # the switch on -- the caps branch must never ride SHAPE_ACRONYM_ + # TAG beside a listed member's own membership tag, which is what + # silenced `listed_lean` for these before the fix (F1, decided by + # the reviewer; decisions.md#S2). Both rows are byte-identical to + # their DEFAULT-policy siblings above (`caps_ambiguous_leans_ + # credential`, `the_lean_reaches_the_post_comma_slot`) and carry + # no `shape=` tag for the same reason every other policy-on row + # here does not: the contract corpus keys on (shape, text) with no + # policy of its own, so tagging one would have the gate diff it + # against the released wheels under the DEFAULT policy -- the + # wrong question for a row whose point IS the non-default policy + # (`Case.shape`'s own docstring). + Case("caps_switch_does_not_silence_the_listed_lean", "Jack MA", + {"given": "Jack", "suffix": "MA"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name", "given-or-family"), + notes="the switch must not touch a LISTED member's own " + "#289 lean: identical to the default reading"), + Case("caps_switch_does_not_silence_the_comma_lean", "Smith, MA", + {"family": "Smith", "suffix": "MA"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="the comma-form twin of the row above, same guarantee"), + # #516 review round: UNLISTED means in no wordlist at all, not + # merely "no whole-token suffix vocabulary" -- a capitalized + # particle or particle phrase must not join the class either + # (F1b, decided by the reviewer; decisions.md#S2's amendment). + Case("caps_switch_does_not_claim_a_capitalized_particle", + "John Smith DE", + {"given": "John", "middle": "Smith", "family": "DE"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="'de' is a particle, not merely absent from suffix " + "vocabulary -- identical to the default reading with " + "the switch on"), + Case("caps_switch_does_not_claim_a_capitalized_particle_phrase", + "John Smith, DE LA", + {"family": "John Smith DE LA"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the multi-token twin: 'DE' and 'LA' are both " + "particles, so the RUN test (#516's own 'LEED AP' " + "shape) must decline them too -- identical to the " + "default reading"), + # #516 review round, F5: the spec's stated mechanism for why + # 'Jack VI' does not move was wrong -- 'VI' DOES reach the caps + # predicate and carries both tags with the switch on. It stays + # unchanged because `_pieces.is_trailing_numeral_suffix` (the + # roman-numeral fork) claims it downstream, before the lean/count + # this switch adds is ever consulted -- unrelated to this switch, + # and the reason these controls read identically on or off. + Case("caps_switch_does_not_move_a_roman_numeral", "Jack VI", + {"given": "Jack", "suffix": "VI"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name", "given-or-family"), + notes="the roman-numeral fork claims 'VI' before this " + "switch's lean/count is consulted -- identical to the " + "default reading, on or off"), + Case("caps_switch_does_not_move_a_roman_numeral_with_words_to_spare", + "John Smith VI", + {"given": "John", "family": "Smith", "suffix": "VI"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="the words-to-spare twin of the row above, same " + "mechanism, same guarantee"), + Case("caps_switch_does_not_move_a_title_floor_control", "Mr XXX", + {"title": "Mr", "family": "XXX"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="one piece behind a title never reaches the peel's " + "`k >= 2` floor -- identical to the default reading"), + Case("caps_switch_does_not_reach_delimited_content", + "Andrew Perkins (XYZ)", + {"given": "Andrew", "family": "Perkins", "nickname": "XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="delimited content is decided by the clause escape, " + "never at the trailing slot -- identical to the " + "default reading"), + # #516 review round, F2: the multi-token run test is a property of + # the CAPS class alone -- a run of pure LISTED members must keep + # its EXISTING reading (`_pieces.segment_suffix_reading`'s own + # per-piece walk, #289, unrelated to this switch and to its + # structure-flip candidate test) rather than being swept into the + # caps run test's `all()`. Identical to the default reading. + Case("caps_switch_run_test_declines_a_pure_listed_run", + "John Smith, Ed Ma", + {"given": "Ed", "middle": "Ma", "family": "John Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="'Ed' and 'Ma' are both LISTED ambiguous members, " + "Title-case (leans NAME, #289) -- the caps run test " + "must not admit a run the listed class already reads " + "on its own"), + # #516 review round (second finding): the caps branch read + # `one_case_own` -- true only for a token INSIDE the maiden + # clause's own-words span -- where it needed the bare NAME-level + # fact. 'NEE' opens the clause, so it sits OUTSIDE that span by + # construction and read as False regardless of the whole name's + # case, wrongly joining the shape class in a wholly one-case name. + # `maiden_markers` is also the one wordlist the ten-list exclusion + # first shipped without. + Case("caps_switch_does_not_claim_a_one_case_maiden_marker", + "JOHN SMITH NEE", + {"given": "JOHN", "middle": "SMITH", "family": "NEE"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="one case, and a maiden marker with no clause to open " + "(nothing follows it) -- identical to the default " + "reading either way"), + Case("caps_switch_does_not_claim_a_mixed_case_maiden_marker", + "John Smith NEE", + {"given": "John", "middle": "Smith", "family": "NEE"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the mixed-case twin: 'NEE' is unlisted by the caps " + "shape test's OWN membership check too, so this one " + "was already declining before this fix -- pinned " + "beside its one-case sibling for the same guarantee"), + # The 2026-09-18 review round's unpinned branches: live code paths + # that no row named, found by reading the diff rather than by a + # failure. Nothing here moved; every value is measured. + Case("caps_switch_reads_the_name_level_case_past_a_clause", + "née JONES XYZ", + {"given": "née", "middle": "JONES", "family": "XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the positive half of the two NEE rows above, and the " + "row that fails if classify's caps branch is reverted " + "to `one_case_own`: a marker OPENING the name leaves " + "own_words EMPTY, so the name-level fact is one-case " + "and nothing in the clause can join the shape class. " + "Under `one_case_own` every token past the cut reads " + "as mixed and 'XYZ' becomes a credential. " + "test_classify.py monkeypatches the revert and asserts " + "this very reading breaks"), + Case("caps_one_case_comma_declines_a_single_token", + "JOHN SMITH, XYZ", + {"given": "XYZ", "family": "JOHN SMITH"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the comma control for the one-case gate: two name " + "words before the comma would flip the structure for a " + "LISTED member, but the caps class needs a case " + "contrast to be a member at all, and a wholly " + "upper-case name has none. No report either -- there " + "was no fork to call"), + Case("caps_one_case_comma_declines_a_run", + "JOHN SMITH, LEED AP", + {"given": "LEED", "middle": "AP", "family": "JOHN SMITH"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the multi-token twin: segment's run test asks " + "`caps_shape_candidate` of every token and then reads " + "the case fact ONCE, so a one-case name declines the " + "whole run rather than per token"), + Case("caps_run_needs_every_token_not_any", + "John Smith, LEED BA", + {"given": "LEED", "middle": "BA", "family": "John Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + ambiguities=("suffix-or-name",), + notes="pins `all()` rather than `any()`: 'BA' is a LISTED " + "ambiguous acronym, so the caps shape test excludes it " + "and the run is no caps run -- the structure stays the " + "listing form. The report is assign's post-comma one, " + "fired on 'LEED' alone, which carries the shape tag " + "from classify whatever segment made of the run"), + Case("the_comma_count_counts_names_not_words_behind_a_title", + "Mr Smith, Ma", + {"given": "Ma", "family": "Mr Smith"}, + ambiguities=("suffix-or-name",), + notes="MEASURED. name_word_count's TITLE arm: 'Mr' is title " + "vocabulary, so the part before the comma holds ONE " + "name word and the count declines the flip. Without " + "that arm two tokens would read as two names and hand " + "the family to `given`"), + Case("the_comma_count_counts_names_not_words_behind_a_suffix", + "Smith Jr, Ma", + {"given": "Ma", "family": "Smith", "suffix": "Jr"}, + ambiguities=("suffix-or-name",), + notes="the SUFFIX arm of the same count, and the shape " + "decisions.md#S2 names: 'Smith Jr., MA' is two tokens " + "and one name. The Title-case spelling declines where " + "the all-caps one flips, which is the pair rules.md#C1 " + "states"), + Case("strict_comma_reads_the_dotted_numeral_as_a_name_word", + "Smith V., Ma", + {"given": "Smith", "family": "V.", "suffix": "Ma"}, + policy=Policy(lenient_comma_suffixes=False), + ambiguities=("suffix-or-name",), + notes="MEASURED, and a known rough edge recorded rather than " + "repaired (decisions.md#S2, 2026-09-18). Under strict " + "the initial-shaped 'V.' fails the suffix test, so " + "name_word_count sees TWO name words before the comma, " + "flips the structure, and the positional read of " + "'Smith V.' gives given 'Smith', family 'V.'. " + "Consistent with master: 'Smith V., PhD' reads the " + "same way under the same knob, so this is the knob's " + "own reading of 'V.' and not the credential class's"), + Case("strict_comma_reads_the_bare_numeral_into_the_run", + "Smith V, Ma", + {"given": "Smith", "suffix": "V, Ma"}, + policy=Policy(lenient_comma_suffixes=False), + ambiguities=("suffix-or-name", "suffix-or-name", + "given-or-family"), + notes="the period is the whole difference from the row " + "above: bare 'V' is still initial-shaped and still " + "counts as a name word here, so the structure flips " + "the same way -- but the trailing peel then takes it " + "as the roman numeral, leaving 'Smith' the only name " + "word and reporting all three forks"), + Case("caps_run_declines_a_bound_given_head", "John Smith, ABDUL AP", + {"given": "ABDUL AP", "family": "John Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the exclusion end to end rather than at the " + "predicate: 'ABDUL' is bound-given vocabulary, so the " + "run is no caps run and the part after the comma is " + "the given name it would be at the default"), + Case("caps_run_declines_a_conjunction", "John Smith, AND AP", + {"given": "AND AP", "family": "John Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="the same end to end for `conjunctions`, the row " + "test_classify.py's predicate table names as the one " + "that genuinely exercises that arm ('Y' declines at " + "the two-character gate first)"), + Case("caps_switch_leaves_a_capitalized_title_a_title", + "John Smith, MR", + {"title": "MR", "given": "John", "family": "Smith"}, + policy=Policy(unlisted_caps_suffixes=True), + notes="`titles` end to end: the post-comma part holds no " + "name word, so C1's no-name-word clause keeps the " + "pre-comma positional read and 'MR' is the title it " + "is at the default"), + Case("a_period_final_delimited_clause_joins_the_shape_class", + "Andrew Perkins (X.Y.Z.)", + {"given": "Andrew", "family": "Perkins", "suffix": "X.Y.Z."}, + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="MEASURED, and it NARROWS the note on " + "caps_switch_does_not_reach_delimited_content: " + "`_extract._suffix_shaped` releases period-final " + "delimited content with role None, so it is ordinary " + "trailing material by the time classify runs and the " + "shape class claims it -- matching '(M.D)', which the " + "escape already sent to `suffix`. What the " + "`token.role is None` guard keeps out is content the " + "escape did NOT release ('Bridge (A.B)', a nickname), " + "which is a different population from 'delimited " + "content' whole"), + Case("a_leading_period_is_not_the_dotted_shape", "John Smith .XY", + {"given": "John", "middle": "Smith", "family": ".XY"}, + notes="`period_joined_vocab` splits on interior periods, so " + "a LEADING one leaves a single chunk and there is no " + "acronym shape to read -- name material, and no fork " + "was called, so nothing is reported"), + Case("a_caseless_name_word_leaves_the_count_to_decide", "毛泽东 MA", + {"given": "毛泽东", "family": "MA"}, + ambiguities=("suffix-or-name",), + notes="the comma-less twin of the 毛泽东, MA row: one name " + "word before the acronym, so the peel's two-piece " + "floor declines it whatever the writing says, and the " + "fork is reported all the same. Han is caseless, so " + "the NAME is mixed-case (the all-caps 'MA' contrasts " + "with nothing that has a case) -- the lean reads " + "'credential' and the floor is what refuses it", + tolerated=True), + Case("a_digit_chunk_moves_the_trailing_slot_silently", + "John Smith 1.4.2", + {"given": "John", "middle": "Smith", "family": "1.4.2"}, + classification="fix(#516)", + notes="ACCEPTED and SILENT, recorded at decisions.md#S2. " + "rules.md#S3's narrow retirement drops a dotted token " + "whose every matched chunk is a single ASCII " + "character, and the vocabulary's lone digit '2' is one " + "of them -- so the chunk claim goes, and the shape " + "class cannot take it either, its own gate wanting " + "every chunk ALPHABETIC. A version string read as a " + "credential was the same accident the retirement " + "removes, so it moves suffix -> family at every " + "policy with no report"), + Case("a_digit_chunk_moves_the_post_comma_slot_silently", + "Smith, 1.4.2", + {"given": "1.4.2", "family": "Smith"}, + classification="fix(#516)", + notes="the comma twin: no candidate, so no flip and no " + "report -- the part after the comma is simply the " + "given name, where 2.3 read family 'Smith', suffix " + "'1.4.2'"), + Case("a_digit_chunk_moves_the_two_word_comma_slot_silently", + "John Smith, 1.4.2", + {"given": "1.4.2", "family": "John Smith"}, + classification="fix(#516)", + notes="and with TWO name words before the comma, where a " + "real class member would flip the structure: '1.4.2' " + "is no member, so the listing form stands and the " + "whole pre-comma run is the family. The widest of the " + "three silent moves"), + Case("the_comma_flip_is_read_under_the_declared_order", + "John Smith, MA", + {"given": "Smith", "family": "John", "suffix": "MA"}, + policy=Policy(name_order=FAMILY_FIRST), + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="MEASURED. The count flips the structure whatever the " + "order -- it counts NAME words, which no order changes " + "-- and the pre-comma run is then read positionally, " + "so the declared order decides which of 'John Smith' " + "is the family. The report's wording quotes the count, " + "not a role, so it reads the same under every order"), + Case("two_caps_credentials_peel_as_a_run", + "John MA XYZ", + {"given": "John", "suffix": "MA XYZ"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("given-or-family", "suffix-or-name", + "suffix-or-name"), + notes="MEASURED, and the widest reading the switch reaches: " + "the peel walks from the END, so 'XYZ' goes first (two " + "pieces still stand behind it), and 'MA' then has " + "'John' alone behind it -- but its LISTED lean says " + "credential, which needs no words to spare. The name " + "loses its family name entirely and says so three " + "times. At the default it reads given John, middle MA, " + "family XYZ, silently"), + Case("the_caps_shape_is_script_agnostic_cyrillic", + "Иван Петр ИВАНОВ", + {"given": "Иван", "family": "Петр", "suffix": "ИВАНОВ"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the switch's docstring claims `isupper()` is " + "script-agnostic, so the convention and the reason for " + "the default are the same in any script with a case " + "contrast. Measured rather than asserted in prose: a " + "Cyrillic all-caps surname joins the class exactly as " + "'Jean Pierre DUPONT' does, and is swallowed the same " + "way"), + Case("the_dotted_switch_off_still_reports_the_post_comma_fork", + "Smith, A.B.", + {"given": "A.B.", "family": "Smith"}, + policy=Policy(unlisted_dotted_suffixes=False), + ambiguities=("suffix-or-name",), + notes="MEASURED. With the switch off the token never joins " + "the class, so the STRUCTURE cannot flip -- but " + "classify still writes SHAPE_ACRONYM_TAG (the fork was " + "real and the parser declined it), and assign's " + "post-comma report reads that tag directly. Exactly " + "ONE report: the shape tag is on one piece and the " + "reading is read off the first post-comma piece alone", + classification="fix(#516)"), + Case("the_dotted_switch_off_leaves_a_lone_token_to_the_convention", + "A.B.", {"given": "A.B."}, + policy=Policy(unlisted_dotted_suffixes=False), + ambiguities=("given-or-family",), + notes="one piece, so the peel's two-piece floor refuses the " + "class before the switch is consulted at all, and what " + "is left is O5's lone-name-word convention. GIVEN_OR_" + "FAMILY and nothing else -- pinned because the shape " + "tag is present and must NOT produce a second report " + "where no fork was taken"), + Case("the_dotted_switch_off_leaves_a_titled_token_a_name", + "Dr. A.B.", {"title": "Dr.", "family": "A.B."}, + policy=Policy(unlisted_dotted_suffixes=False), + notes="the title floor: the peel's walk starts after the " + "leading title run, so one piece is all it sees and " + "the fork is never consulted -- no report at all, the " + "same floor 'Mr MA' pins for the listed half"), + Case("the_caps_shape_is_script_agnostic_accented", + "Jean Pierre ÉCOLE", + {"given": "Jean", "family": "Pierre", "suffix": "ÉCOLE"}, + policy=Policy(unlisted_caps_suffixes=True), + classification="fix(#516)", + ambiguities=("suffix-or-name",), + notes="the other half of the same claim: a non-ASCII LATIN " + "letter. `isalpha()`/`isupper()` are Unicode-wide, so " + "an accented capital is admitted -- the docstring's " + "'Jean ÉCOLE' example, given the third word it needs " + "to have words to spare"), Case("catalan_i_is_not_connective_vocabulary_upper", "JOSEP CAROD I ROVIRA", {"given": "JOSEP", "middle": "CAROD I", "family": "ROVIRA"}, @@ -1392,8 +2630,16 @@ def _check_cjk_shape_purity(self) -> None: {"given": "Juan", "family": "de la Vega", "suffix": "III"}), Case("interior_periods_block_vocab", "Smith, J.R.", {"given": "J.R.", "family": "Smith"}, + ambiguities=("suffix-or-name",), notes="v1's lc() keeps interior periods: 'J.R.' is not the " - "title 'jr' (pinned live 2026-07-17)"), + "suffix word 'jr' (pinned live 2026-07-17), so this " + "FIELD reading is unchanged clear back to 1.4.0. #516: " + "two unlisted single-letter chunks join the ambiguous " + "class by shape all the same, the same fork " + "'Smith, A.B.' reports -- the class is considered and " + "declined (one word before the comma is never enough), " + "and the consideration is what reports, exactly as the " + "trailing peel's own declined picks always have"), Case("dotted_acronym_suffix", "John Smith M.D.", {"given": "John", "family": "Smith", "suffix": "M.D."}, notes="suffix-ACRONYM membership alone strips periods (v1 " @@ -1431,9 +2677,18 @@ def _check_cjk_shape_purity(self) -> None: Case("period_joined_titles", "Lt.Gov. John Doe", {"title": "Lt.Gov.", "given": "John", "family": "Doe"}, notes="v1 derived-title rule: ANY period chunk being a title " - "makes the token a title (pinned live 2026-07-16)"), + "makes the token a title (pinned live 2026-07-16). " + "Title wins before #516's shape verdict is even asked, " + "so this stays a protected control for the switch, not " + "just for the chunk rule", + shape=1), Case("period_joined_suffixes", "John Doe JD.CPA", - {"given": "John", "family": "Doe", "suffix": "JD.CPA"}), + {"given": "John", "family": "Doe", "suffix": "JD.CPA"}, + notes="the multi-character chunk match 'jd'/'cpa' still wins " + "over #516's shape verdict, which is asked only where " + "classify's own fall-through has not already tagged " + "'vocab:suffix' -- unchanged either side of the switch", + shape=1), Case("period_joined_any_rule", "Mr.Smith", {"title": "Mr.Smith"}, notes="the ANY rule is deliberate v1 parity: one title chunk " @@ -3176,8 +4431,36 @@ def _check_cjk_shape_purity(self) -> None: ambiguities=("suffix-or-name",), notes="'ed' is an ambiguous acronym; bare form is a name (C1), " "and the parse reports which reading it took"), + # MOVED by #289, not deleted (roles unchanged, a report gained): + # 'Ed' is Title-case in a mixed-case name, so it leans SURNAME and + # the post-comma slot declines it exactly as before -- but the + # decision is now reported, both directions of the fork being + # worth telling the caller about (decisions.md#S2). Case("comma_ambiguous_acronym", "Smith, Ed", - {"given": "Ed", "family": "Smith"}), + {"given": "Ed", "family": "Smith"}, + ambiguities=("suffix-or-name",)), + # Quality-review finding, 2026-09-17 (item 5): name_word_count's + # title half used a bare `folded in lexicon.titles` lookup, not + # is_leading_title's H2 shape test -- so an UNLISTED period-marked + # opener counted as a NAME word where a LISTED one did not, and + # this pair split on that alone: 'Dr. Smith, Ed' (LISTED 'Dr.') + # read family 'Dr. Smith' unflipped, but 'Xyz. Smith, Ed' + # (UNLISTED, H2-shaped) flipped the comma structure and read title + # 'Xyz.', family 'Smith', suffix 'Ed' -- a different STRUCTURE for + # the same shape, from a title-vocabulary difference the count had + # no business seeing. Fixed by sharing H2's shape test + # (`_vocab.is_title_shaped`, asked in this count and inlined in + # `_pieces.is_leading_title`, two spellings of one predicate -- + # sharing the function costs a frame on the hot leading-peel path, + # measured, so they stay separate call sites instead). + Case("comma_count_reads_h2_s_shape_test_too", "Xyz. Smith, Ed", + {"given": "Ed", "family": "Xyz. Smith"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="MOVED: title 'Xyz.', family 'Smith', suffix 'Ed' before " + "this fix (measured at e545803) -- an unlisted period-" + "marked opener now reads the way a listed title does at " + "this count, exactly as 'Dr. Smith, Ed' beside it"), Case("ambiguous_acronym_with_suffix", "John Ed III", {"given": "John", "family": "Ed", "suffix": "III"}, ambiguities=("suffix-or-name",)), @@ -3460,13 +4743,18 @@ def _check_cjk_shape_purity(self) -> None: "form trailing"), Case("audit_perioded_sa_is_the_postnominal", "Smith, S.A.", {"family": "Smith", "suffix": "S.A."}), - Case("audit_bare_do_after_comma_is_a_name", "Smith, DO", - {"given": "DO", "family": "Smith"}, - classification="fix(#296)", - notes="'do' left TITLES but was already AMBIGUOUS, so the bare " - "spelling is neither title nor suffix and falls to the " - "given position -- the period gate handles the real " - "collision, which is that 'Do' is a name"), + # MOVED by #289, not deleted: 'DO' is written in capitals inside a + # mixed-case name, so it now leans CREDENTIAL and the post-comma + # slot takes it with one word to spare rather than falling to the + # given position (decisions.md#S2). + Case("audit_bare_caps_do_after_comma_is_a_credential", "Smith, DO", + {"family": "Smith", "suffix": "DO"}, + classification="fix(#289)", + ambiguities=("suffix-or-name",), + notes="'do' left TITLES but was already AMBIGUOUS; the period " + "gate handles the real collision, which is that 'Do' is " + "a name -- but the bare, all-caps spelling is now read " + "the way a credential is written"), Case("audit_perioded_do_after_comma_is_a_suffix", "Smith, D.O.", {"family": "Smith", "suffix": "D.O."}), # -- the TRAILING half of the same two removals. `dr` and `sra` are @@ -3912,9 +5200,13 @@ def _check_cjk_shape_purity(self) -> None: "the reports come from the single peel over the " "spliced pieces. Collecting both peels' picks instead " "reported the same coin flip twice"), + # MOVED by #289, not deleted: spliced, this is 'John MA' plus a + # title, and 'MA' is written in capitals inside a mixed-case name + # -- the caps lean now takes it with no words to spare where the + # old reserve kept it the family (decisions.md#S2). Case("title_word_trailing_run_is_read_to_a_fixed_point", "John Prof. MA Prof.", - {"title": "Prof. Prof.", "given": "John", "family": "MA"}, + {"title": "Prof. Prof.", "family": "John", "suffix": "MA"}, ambiguities=("suffix-or-name",), classification="fix(#316)", notes="transparency for a SECOND title, which peel and chain " "reading to a FIXED POINT buys and one re-peel did " @@ -3928,17 +5220,22 @@ def _check_cjk_shape_purity(self) -> None: "first 'John' / middle 'Prof. MA' / last 'Prof.' " "(measured 2026-09-09), so the row is #316's parity " "break either way and the round decided only which " - "reading it is"), + "reading it is. #289 moves it again: the caps lean " + "takes 'MA' as a credential rather than the family " + "(decisions.md#S2)"), Case("title_word_trailing_keeps_the_bare_acronym_reserve", "John Prof. MA", - {"title": "Prof.", "given": "John", "family": "MA"}, + {"title": "Prof.", "family": "John", "suffix": "MA"}, ambiguities=("suffix-or-name",), classification="fix(#316)", notes="the same principle where the peel's answer DEPENDS on " - "the spliced list: S2's reserve keeps a bare ambiguous " - "acronym the family of a two-word name, so this is " - "'John MA' plus a title and reads exactly as 'Prof. " - "John MA' does. Laying a second peel's roles over the " - "first peel's read family 'John', suffix 'MA'"), + "the spliced list: this is 'John MA' plus a title and " + "reads exactly as 'Prof. John MA' does. Laying a " + "second peel's roles over the first peel's read family " + "'John', suffix 'MA'. #289 MOVES this row: 'MA' is " + "written in capitals inside a mixed-case name, so the " + "caps lean now takes it with no words to spare, where " + "S2's reserve alone used to keep it the family " + "(decisions.md#S2)"), Case("title_word_trailing_between_two_numerals", "John Smith V Prof. VI", {"title": "Prof.", "given": "John", "middle": "Smith V", @@ -3955,7 +5252,8 @@ def _check_cjk_shape_purity(self) -> None: classification="parity", notes="negative control, and the doctrine: the leading slot " "has a SHAPE rule that outranks vocabulary (H2), the " - "trailing slot reads vocabulary only"), + "trailing slot reads vocabulary only", + shape=1), Case("title_word_trailing_bare_is_a_name_word", "John Smith Sir", {"given": "John", "middle": "Smith", "family": "Sir"}, classification="parity", diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 13b03773..bb0fa7af 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -48,6 +48,111 @@ def _by_role(state: ParseState, role: Role) -> str: return " ".join(t.text for t in state.tokens if t.role is role) +_ROLE_NAMES = (("title", Role.TITLE), ("given", Role.GIVEN), + ("middle", Role.MIDDLE), ("family", Role.FAMILY), + ("suffix", Role.SUFFIX), ("nickname", Role.NICKNAME), + ("maiden", Role.MAIDEN)) + + +def _fields(text: str, policy: Policy | None = None) -> dict[str, str]: + """The non-empty role fields, by name -- #289's real ambiguous + vocabulary (`ba do ed jd ma`) is Lexicon.default()'s, not the + synthetic `_LEX` most of this module's tests share, so this reads + the default lexicon rather than take one as a parameter every + caller would otherwise have to pass.""" + out = _assigned(text, policy, Lexicon.default()) + return {name: v for name, role in _ROLE_NAMES + if (v := _by_role(out, role))} + + +def test_the_lean_reaches_the_post_comma_given_slot() -> None: + # #289, Derek's own comment on the issue: positive evidence + # outranks position here, so the credential lean fires with ONE + # word before the comma where the count would not. + assert _fields("Smith, MA") == {"family": "Smith", "suffix": "MA"} + assert _fields("Smith, BA") == {"family": "Smith", "suffix": "BA"} + assert _fields("Smith, MA PhD") == {"family": "Smith", + "suffix": "MA PhD"} + # the other two spellings keep today's reading: a surname lean, and + # no lean at all + assert _fields("Smith, Ma") == {"given": "Ma", "family": "Smith"} + assert _fields("Smith, ma") == {"given": "ma", "family": "Smith"} + # a caseless script wrote no contrast, so nothing leans + assert _fields("毛泽东, MA") == {"given": "MA", "family": "毛泽东"} + + +def test_the_comma_path_reports_its_ambiguous_reading_once() -> None: + # The first report of the comma's OWN decision in the library + # (#289) -- C2's structural flag already reports on the comma + # path, but it reports what the parse could not recognize, not a + # fork it called. P6's attachment fork has separately reported on + # a family-comma path since 2.3 ("Berg, Jan vd"). One per + # DECISION, in either direction, and never twice for one name -- + # the structure decision reports where it is taken and this one + # reports where the family comma stands. + # + # The report tracks the FORK BEING CONSULTED, not the lean -- + # exactly as the trailing slot always has ('Jack MA' reported + # before #289 too, even where the peel declined the pick for want + # of words to spare). So a caseless script ('毛泽东, MA') and an + # all-lower spelling ('Smith, ma') still called this fork and + # report it, read positionally; and a two-piece post-comma part + # whose FIRST piece is the class member ('Smith, MA PhD') reports + # once through that same first-piece read (F4/F5 review finding, + # 2026-09-17 -- reverses an earlier round's `ambiguous_lean(...) + # is not None` gate, which wrongly excluded both). + # 'Smith, A.B.' is in the loop since `Policy.unlisted_dotted_ + # suffixes` shipped: its token carries the by-shape tag, so the + # fork IS consulted at this comma and reports exactly once -- + # which is also true with the switch OFF, the shape tag going on + # either way (its own case rows pin both). + for text in ("Smith, MA", "Smith, Ma", "Smith, ma", "John Smith, MA", + "John Smith, Ed", "毛泽东, MA", "Smith, MA PhD", + "Smith, A.B."): + kinds = [a.kind.value for a in _assigned( + text, lexicon=Lexicon.default()).ambiguities] + assert kinds.count("suffix-or-name") == 1, (text, kinds) + # a name outside the class reports nothing at its comma, as every + # release before this one + for text in ("Smith, John", "John Smith, PhD", "Smith, Dr."): + kinds = [a.kind.value for a in _assigned( + text, lexicon=Lexicon.default()).ambiguities] + assert "suffix-or-name" not in kinds, text + + +def test_the_family_comma_report_detail_is_verbatim() -> None: + # The family-comma path's own emitter (assign's half of the FIRST + # comma-path report; segment's structure-flip half is pinned in + # test_segment.py). Pinned verbatim for the same reason. + out = _assigned("Smith, MA", lexicon=Lexicon.default()) + (amb,) = [a for a in out.ambiguities + if a.kind is AmbiguityKind.SUFFIX_OR_NAME] + assert amb.detail == ( + "'MA' after the comma is also an ordinary name word; read as " + "a credential") + + +def test_jack_ma_s_two_detail_strings_are_verbatim() -> None: + # The trailing slot's OWN report existed before #289 (a bare + # ambiguous acronym was always a coin-flip); what #289 changes is + # which way 'Jack MA' flips, not that it reports. Pinning both + # strings verbatim: the credential lean also turns 'Jack' into the + # only name word left, which is a SECOND fork (GIVEN_OR_FAMILY) + # this one row now calls. + out = _assigned("Jack MA", lexicon=Lexicon.default()) + details = {a.kind.value: a.detail for a in out.ambiguities} + assert details == { + "given-or-family": ( + "'Jack' is the only name word and nothing else decides " + "it; read as a given name by convention, which follows " + "the read order"), + "suffix-or-name": ( + "'MA' written without periods is both a post-nominal and " + "an ordinary name; read as a suffix rather than a name " + "part"), + } + + def test_given_first_positional() -> None: out = _assigned("Dr. Juan de la Vega III") assert _by_role(out, Role.TITLE) == "Dr." @@ -256,11 +361,16 @@ def test_the_trailing_title_is_transparent_to_the_suffix_peel() -> None: """ lex = _LEX.add(suffix_acronyms={"ma"}, suffix_acronyms_ambiguous={"ma"}) + # MOVED by #289, not deleted: spliced, 'John MA' is the bare + # 'Jack MA' shape -- an ALL-CAPS ambiguous acronym in a mixed-case + # name leans credential and is taken with no words to spare, so + # the reserve that used to keep it the family now takes it as the + # suffix instead (decisions.md#S2). out = _assigned("John Mr. MA", lexicon=lex) assert _by_role(out, Role.TITLE) == "Mr." assert _by_role(out, Role.GIVEN) == "John" - assert _by_role(out, Role.FAMILY) == "MA" - assert not _by_role(out, Role.SUFFIX) + assert not _by_role(out, Role.FAMILY) + assert _by_role(out, Role.SUFFIX) == "MA" out = _assigned("John Smith Mr. MA", lexicon=lex) assert _by_role(out, Role.FAMILY) == "Smith" assert _by_role(out, Role.SUFFIX) == "MA" @@ -583,3 +693,30 @@ def test_a_mixed_post_comma_run_keeps_the_walk_order() -> None: out = _assigned("Smith, John Jr.") assert _by_role(out, Role.GIVEN) == "John" assert _by_role(out, Role.SUFFIX) == "Jr." + + +def test_the_comma_report_says_which_way_it_read_the_word() -> None: + """The report's OTHER branch, verbatim (2026-09-18 review round). + + `test_the_family_comma_report_detail_is_verbatim` above pins the + credential wording; only the string distinguishes the two + branches, the kind being the same either way and the roles + telling a caller which reading won only if it already knows which + field to look in. So the NAME branch needs its own row, and the + by-shape half with its switch OFF -- which takes that branch for a + fork the parser considered and declined -- is the second one. + """ + detail = { + text: [a.detail for a in _assigned( + text, policy, lexicon=Lexicon.default()).ambiguities + if a.kind.value == "suffix-or-name"] + for text, policy in (("Smith, Ma", None), + ("Smith, A.B.", + Policy(unlisted_dotted_suffixes=False))) + } + assert detail["Smith, Ma"] == [ + "'Ma' after the comma is also an ordinary name word; read as " + "the given name"] + assert detail["Smith, A.B."] == [ + "'A.B.' after the comma is also an ordinary name word; read as " + "the given name"] diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index fbc2171c..7ed579aa 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -1,11 +1,21 @@ import dataclasses -from nameparser._lexicon import Lexicon +import pytest + +from nameparser import Parser +from nameparser._lexicon import Lexicon, _normalize +from nameparser._pipeline import STAGES +from nameparser._pipeline import _classify as _classify_module from nameparser._pipeline._classify import classify from nameparser._pipeline._extract import extract_delimited from nameparser._pipeline._segment import segment -from nameparser._pipeline._state import ParseState +from nameparser._pipeline._state import ( + SHAPE_ACRONYM_TAG, ParseState, WorkToken, +) from nameparser._pipeline._tokenize import tokenize +from nameparser._pipeline._vocab import ( + ambiguous_class_candidate, ambiguous_class_member, caps_shape_candidate, +) from nameparser._policy import Policy from nameparser._types import AmbiguityKind, Role @@ -44,10 +54,33 @@ def _classified(text: str) -> ParseState: return _classified_with(text, _LEX) +def _state_through(stage_name: str, text: str) -> ParseState: + """Run the pipeline up to and including the named stage (STAGES' + own names) using this module's `_LEX`, for a test that needs a + state classify has not yet touched (#289/#516).""" + state = ParseState(original=text, lexicon=_LEX, policy=Policy()) + for stage in STAGES: + state = stage(state) + if stage.__name__ == stage_name: + break + return state + + def _tags(state: ParseState, text: str) -> frozenset[str]: return next(t.tags for t in state.tokens if t.text == text) +def _tags_by_text(text: str, policy: Policy = Policy(), + lexicon: Lexicon = _LEX) -> dict[str, frozenset[str]]: + """Every token's tags after classify, keyed by text -- the shape + tests below ask about a token whose text is not in `_LEX` at all, + which the by-name `_tags` above still finds fine, but a dict + reads more plainly at more than one lookup per name (#516).""" + state = ParseState(original=text, lexicon=lexicon, policy=policy) + out = classify(segment(tokenize(extract_delimited(state)))) + return {t.text: t.tags for t in out.tokens} + + def test_vocabulary_tags() -> None: out = _classified("Dr. van de la Smith and abdul née PhD Jr") assert "vocab:title" in _tags(out, "Dr.") @@ -282,7 +315,7 @@ def test_a_word_after_the_maiden_marker_reads_as_plain_vocabulary() -> None: def test_a_clauses_own_marker_word_does_not_truncate_the_own_words() -> None: - # #527 review: _tag_marker_runs walks every token, so a maiden + # #527 review: _vocab.tag_marker_runs walks every token, so a maiden # marker WORD that arrives already ROLED (parenthesised clause # content, extract's doing -- WorkToken.role's docstring) is the # CLAUSE's own word, not a bare marker opening a new clause, and @@ -357,3 +390,300 @@ def test_classify_is_per_token_independent_of_the_comma() -> None: assert "conjunction" in _tags(suffix_comma, "Y") assert "initial" not in _tags(suffix_comma, "Y") assert suffix_comma.ambiguities == () + + +def test_classify_records_the_one_case_fact_on_the_state() -> None: + # #289/#516 promotes the fact #527 computed as a local: the suffix + # slot, the post-comma slot and the tail-segment reading all + # consult it, and two sites deciding it apart is what + # ParseState.order's shape exists to prevent. + assert _classified("JOHN SMITH MA").one_case is True + assert _classified("John Smith Ma").one_case is False + assert _classified("john smith ma").one_case is True + # a caseless script has only one case, and answers True harmlessly + assert _classified("毛泽东").one_case is True + # the span is the name's OWN words: a maiden clause beside a + # one-case name does not make it mixed (rules.md#P3) + assert _classified("JUAN GARCIA Y LOPEZ née Jones").one_case is True + + +def test_classify_does_not_overwrite_a_fact_already_recorded() -> None: + # segment writes it first where a comma form asked; classify reads + # what is there rather than deciding it a second time. + state = _state_through("segment", "John Smith, MA") + forced = dataclasses.replace(state, one_case=True) + assert classify(forced).one_case is True + + +def test_classify_s_fork_reads_the_pre_recorded_fact_not_its_own_answer() -> None: + # Not just that the field survives (the test above) -- the FORK + # that reads one_case must consult the recorded value, not + # recompute its own. "Jose e Maria Santos" is mixed case on its + # own words, so an unforced parse takes the conjunction branch and + # reports nothing; forcing one_case=True ahead of classify must + # flip 'e' to an initial and report CONJUNCTION_OR_INITIAL even + # though the tokens themselves never changed case (measured + # 2026-09-17, #289/#516). + unforced = _classified("Jose e Maria Santos") + assert "conjunction" in _tags(unforced, "e") + assert "initial" not in _tags(unforced, "e") + assert unforced.ambiguities == () + + state = _state_through("segment", "Jose e Maria Santos") + forced = dataclasses.replace(state, one_case=True) + out = classify(forced) + assert "initial" in _tags(out, "e") + kinds = [a.kind for a in out.ambiguities] + assert kinds == [AmbiguityKind.CONJUNCTION_OR_INITIAL] + + +def test_a_by_shape_acronym_carries_both_tags() -> None: + # #516: the class membership is `vocab:suffix-ambiguous`, so the + # peel needs no new branch -- and `shape:acronym` rides beside it + # because the `vocab:` namespace records MEMBERSHIP and the word + # is not in the vocabulary. The two consumers that must tell them + # apart read the shape tag. + tags = _tags_by_text("John Smith X.Y.Z.")["X.Y.Z."] + assert "shape:acronym" in tags + assert "vocab:suffix-ambiguous" in tags + assert "vocab:suffix" not in tags + # a listed member carries membership and no shape claim + assert "shape:acronym" not in _tags_by_text("John Smith MA")["MA"] + # whole-token vocabulary wins outright + assert "shape:acronym" not in _tags_by_text("John Smith M.A.")["M.A."] + assert "vocab:suffix" in _tags_by_text("John Smith M.A.")["M.A."] + + +def test_the_switch_leaves_the_shape_tag_and_takes_the_membership() -> None: + # OFF reads the token as name material everywhere, as 2.3 did for + # a token no chunk claimed (the roman-chunk retirement is not + # behind this switch) -- and still reports, the parser having + # chosen the name reading over a credential one. + off = _tags_by_text("John Smith X.Y.Z.", policy=Policy( + unlisted_dotted_suffixes=False))["X.Y.Z."] + assert "shape:acronym" in off + assert "vocab:suffix-ambiguous" not in off + + +def test_no_listed_member_ever_carries_the_shape_tag() -> None: + # The invariant, stated over the TAGS rather than over a branch: + # SHAPE_ACRONYM_TAG says "the WRITING made this credential-shaped, + # the vocabulary did not", so a token whose normalized text IS a + # listed ambiguous acronym must never carry it -- the tag silences + # `_pieces.listed_lean`, which is the listing's whole effect. + # + # The custom lexicon is the case that was broken (review round, + # 2026-09-18): a DOTTED entry matches `suffix_acronyms_ambiguous` + # whole while `suffix_as_written`'s period-free acronym lookup + # ('ab') misses it, so the chunk view reached the shape branch and + # 'Jack A.B.' read family where 'Jack MA' reads suffix. + dotted = Lexicon.default().add(suffix_acronyms={"a.b"}, + suffix_acronyms_ambiguous={"a.b"}) + for lex, texts in ((Lexicon.default(), + ("John Smith MA", "John Smith Ma", "Smith, MA", + "John Smith BA", "John Smith X.Y.Z.")), + (dotted, + ("Jack A.B.", "John Smith A.B.", "Smith, A.B."))): + for policy in (Policy(), Policy(unlisted_caps_suffixes=True), + Policy(unlisted_dotted_suffixes=False)): + for text in texts: + for word, tags in _tags_by_text(text, lexicon=lex, + policy=policy).items(): + if SHAPE_ACRONYM_TAG in tags: + assert _normalize(word) not in \ + lex.suffix_acronyms_ambiguous, (text, word) + + +def test_delimited_content_never_joins_the_shape_class() -> None: + # A bracketed clause is decided by extract's escape, not by the + # trailing slot. 'Bridge (1.4)' cannot exercise the `token.role is + # None` guard on its own -- a digit chunk never reaches the shape + # verdict at all, guard or no guard -- so 'Bridge (A.B)' is the + # control that actually load-bears it: without the guard, the + # nickname 'A.B' would gain SHAPE_ACRONYM_TAG and (with the switch + # on) `vocab:suffix-ambiguous`, and classify's own nickname check + # would then misreport SUFFIX_OR_NICKNAME on a token the escape + # already decided (measured regression, #516 review round). + for text, word in (("Bridge (1.4)", "1.4"), ("Bridge (A.B)", "A.B")): + tags = _tags_by_text(text)[word] + assert "shape:acronym" not in tags, text + assert "vocab:suffix-ambiguous" not in tags, text + + +@pytest.mark.parametrize("word,policy,one_case", [ + ("X.Y.Z.", Policy(), None), + ("A.B.", Policy(), None), + ("M.A.", Policy(), None), + ("A.B.C.", Policy(), None), + ("Msc.Ed.", Policy(), None), + ("Lt.Gov.", Policy(), None), + ("1.4", Policy(), None), + ("Xyz.", Policy(), None), + ("MA", Policy(), None), + ("田.中.", Policy(), None), + # #516's caps half: OFF is silent (matches the tag either way, + # since the token never carries the fact); ON needs the REAL + # `one_case` fact, which these rows hand to `caps_shape_candidate` + # -- the predicate classify calls, and since the review round the + # only route into the caps half at all. + ("XYZ", Policy(), None), + ("XYZ", Policy(unlisted_caps_suffixes=True), False), + ("MC", Policy(unlisted_caps_suffixes=True), False), + ("X", Policy(unlisted_caps_suffixes=True), False), + ("XY2", Policy(unlisted_caps_suffixes=True), False), + ("DUPONT", Policy(unlisted_caps_suffixes=True), False), + # #516 review round, F1/F1b: LISTED members must keep the LEAN + # (SHAPE_ACRONYM_TAG must NOT ride beside their membership tag), + # and UNLISTED means in no wordlist at all -- a particle, an + # ambiguous particle and a conjunction, capitalized, must not + # join the class either. + ("MA", Policy(unlisted_caps_suffixes=True), False), + ("BA", Policy(unlisted_caps_suffixes=True), False), + ("DE", Policy(unlisted_caps_suffixes=True), False), + ("Y", Policy(unlisted_caps_suffixes=True), False), + ("VAN", Policy(unlisted_caps_suffixes=True), False), + # 'Y' declines at the `len(text) >= 2` shape gate and never + # actually reaches the conjunction-exclusion check at all; + # 'AND' is the row that genuinely exercises it (quality-review + # finding). + ("AND", Policy(unlisted_caps_suffixes=True), False), + # #516 review round (second finding): the four wordlists the + # first fix round's exclusion left with no row of their own + # (titles, given_name_titles, bound_given_names, suffix_words -- + # `given_name_titles` is always a subset of `titles` in the + # shipped lexicon, so 'AUNT' declines via BOTH the outer + # `vocab:title` guard and this elif's own check, and the row + # still pins that it declines either way), and the maiden-marker + # gap itself. + ("SIR", Policy(unlisted_caps_suffixes=True), False), + ("AUNT", Policy(unlisted_caps_suffixes=True), False), + ("ABDUL", Policy(unlisted_caps_suffixes=True), False), + ("JR", Policy(unlisted_caps_suffixes=True), False), + ("NEE", Policy(unlisted_caps_suffixes=True), False), +]) +def test_ambiguous_class_candidate_agrees_with_the_tag( + word: str, policy: Policy, one_case: bool | None) -> None: + # `ambiguous_class_candidate` (`_vocab.py`) and classify's own tag + # emission ask the SAME questions twice, of necessity -- `segment` + # runs before `classify` and has no tags to read yet. Kept from + # drifting by this test rather than by a sentence alone: both + # docstrings point here by name. `MA.`/`Ed.` are deliberately + # absent: a listed member's DOTTED spelling is carried by the tag + # path alone (`ambiguous_class_member`'s docstring), the one place + # these two answers are meant to differ. + # + # Membership alone is not enough to catch #516's review-round + # regression: `ambiguous_class_candidate("MA", ...)` was ALWAYS + # True (the listed half), on both sides of the bug, so comparing + # candidacy to `vocab:suffix-ambiguous` presence alone never + # noticed that classify's caps branch was ALSO wrongly adding + # `shape:acronym` beside it, silencing `listed_lean`. The second + # assertion is what catches that: SHAPE_ACRONYM_TAG must appear + # if and only if the class was joined BY SHAPE, never for a + # listed member. + # + # `candidate` is spelled as the DISJUNCTION the two stages make + # between them, because that is what `segment` asks in two calls: + # `ambiguous_class_candidate` for the listed and dotted halves at + # its single-token test, and `caps_shape_candidate` for the caps + # half at its multi-token run test. The caps half used to hang off + # `ambiguous_class_candidate` behind an optional `one_case` no + # production caller passed, so it answered False for every name + # the library parsed and only this test reached it; the parameter + # is gone and the rows point at the live predicate instead. + lex = Lexicon.default() + tags = _tags_by_text(f"John Smith {word}", lexicon=lex, + policy=policy)[word] + candidate = (ambiguous_class_candidate(word, lex, policy) + or (policy.unlisted_caps_suffixes + and caps_shape_candidate(word, lex, policy, one_case))) + assert candidate == ("vocab:suffix-ambiguous" in tags) + shape_member = candidate and not ambiguous_class_member(word, lex) + assert (SHAPE_ACRONYM_TAG in tags) == shape_member + + +def test_the_caps_branch_reads_the_name_level_case_not_the_own_span( + monkeypatch: pytest.MonkeyPatch) -> None: + """The mutation control for #516's `one_case` vs `one_case_own` + split, run rather than described. + + A maiden marker OPENING the name leaves `own_words` empty, so the + NAME-level fact is one-case and nothing may join the caps class. + `one_case_own` is a different question -- "one case AND this token + is one of the name's own words" -- and every token past the clause + cut answers it False by construction, so reading it here makes a + wholly one-case name look mixed for exactly those tokens. + + The revert is applied at runtime: `_tags_for` is wrapped so the + caps branch sees `one_case_own` where it should see `one_case`. + Without the wrapper the reading below holds; with it, 'XYZ' joins + the class and the family name is lost. + """ + on = Policy(unlisted_caps_suffixes=True) + parser = Parser(policy=on) + name = parser.parse("née JONES XYZ") + assert (name.given, name.middle, name.family) == ("née", "JONES", "XYZ") + assert name.ambiguities == () + + real = _classify_module._tags_for + + def reverted(token: WorkToken, n: str, state: ParseState, + marker_tag: str | None, one_case_own: bool, + one_case: bool) -> frozenset[str]: + return real(token, n, state, marker_tag, + one_case_own=one_case_own, one_case=one_case_own) + + monkeypatch.setattr(_classify_module, "_tags_for", reverted) + broken = Parser(policy=on).parse("née JONES XYZ") + assert broken.family != "XYZ", ( + "the revert changed nothing, so this control measures nothing: " + "check that classify's caps branch still reads `one_case`") + + +def test_the_caps_shape_is_silent_until_its_switch_is_on() -> None: + # OFF (the default) emits NOTHING -- not the membership tag and + # not the shape tag either, which is where this half differs from + # the dotted one: there is no fork to report while a caller has + # not asked for the reading (#516). + assert _tags_by_text("John Smith XYZ")["XYZ"] == frozenset() + on = Policy(unlisted_caps_suffixes=True) + tags = _tags_by_text("John Smith XYZ", policy=on)["XYZ"] + assert "shape:acronym" in tags and "vocab:suffix-ambiguous" in tags + + +def test_the_caps_shape_is_a_testable_predicate() -> None: + # Spelled for an implementer: isalpha() and isupper() with at + # least two characters, in a MIXED-case name. 'MC' is real suffix + # vocabulary only in the SHIPPED lexicon (this module's own _LEX + # is deliberately minimal and does not carry it), so that one + # assertion uses Lexicon.default() rather than the module fixture. + on = Policy(unlisted_caps_suffixes=True) + # a digit anywhere disqualifies it + assert "shape:acronym" not in _tags_by_text("John Smith XY2", policy=on)["XY2"] + # a single capital stays what it is today, an initial + assert "shape:acronym" not in _tags_by_text("John Smith X", policy=on)["X"] + assert "initial" in _tags_by_text("John Smith X", policy=on)["X"] + # a one-case name has no contrast to read + assert "shape:acronym" not in _tags_by_text("JOHN SMITH XYZ", policy=on)["XYZ"] + # #516 review round (second finding): a token PAST the maiden + # clause cut must decline on the same one-case ground as every + # other token, not on `one_case_own`'s span alone -- 'NEE' sits + # outside `one_case_own`'s span by construction (it opens the + # clause), so testing that flag alone made a wholly one-case name + # look mixed for this one token and only this one. + assert "shape:acronym" not in _tags_by_text( + "JOHN SMITH NEE", policy=on, lexicon=Lexicon.default())["NEE"] + # UNLISTED means no whole-token suffix vocabulary claims it: 'MC' + # is suffix vocabulary and never reaches this switch + assert "shape:acronym" not in _tags_by_text( + "John Smith MC", policy=on, lexicon=Lexicon.default())["MC"] + # an interior period is the other switch's shape, not this one: + # 'X.Y' already carries `shape:acronym` at the DEFAULT (the + # dotted half, `unlisted_dotted_suffixes`, on by default), so the + # bare "in tags" assertion this row had before passed regardless + # of whether the caps switch touched it at all -- a differential + # is the real test (quality-review finding): turning the caps + # switch on must not CHANGE anything about a dotted token's tags. + assert (_tags_by_text("John Smith X.Y", policy=on)["X.Y"] + == _tags_by_text("John Smith X.Y")["X.Y"]) + diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 4d049942..ca50e2c5 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -775,8 +775,16 @@ def test_the_reserve_mirrors_the_bare_acronym_fork() -> None: # then the suffix, and one piece remains -- no family. The reserve # now runs that same peel over the view and declines (#425); it # used to count 'Ma' as a name word and join. + # + # MOVED by #289, not deleted: 'Ma' is Title-case in a mixed-case + # name, so it leans SURNAME and the peel declines it even with + # words to spare -- the walk stops at the declined pick, 'jr' + # never reached behind it, so the reserve now sees the SAME + # suffixes on both sides of the join (none) and the join stands + # (the accepted cost decisions.md#S2 records for + # 'abdul Smith Jr Ma'). out = _grouped("abdul Smith jr Ma", lexicon=_AMBIGUOUS_LEX) - assert _piece_texts(out) == [["abdul", "Smith", "jr", "Ma"]] + assert _piece_texts(out) == [["abdul Smith", "jr", "Ma"]] def test_the_join_never_turns_a_suffix_into_a_name() -> None: @@ -786,8 +794,13 @@ def test_the_join_never_turns_a_suffix_into_a_name() -> None: # words and changes nothing else, so it declines -- 1.4.0's # reading, and 'John Smith Ma's. With a family behind it the # acronym peels either way, and the join stands. + # + # MOVED by #289, not deleted: 'Ma' now leans SURNAME (Title case, + # mixed-case name) on BOTH sides of the join, so the two views' + # suffix readings still agree and the join stands -- 'abdul Smith' + # given, 'Ma' family (decisions.md#S2's corpus row). out = _grouped("abdul Smith Ma", lexicon=_AMBIGUOUS_LEX) - assert _piece_texts(out) == [["abdul", "Smith", "Ma"]] + assert _piece_texts(out) == [["abdul Smith", "Ma"]] out = _grouped("abdul Smith Berg Ma", lexicon=_AMBIGUOUS_LEX) assert _piece_texts(out) == [["abdul Smith", "Berg", "Ma"]] @@ -877,9 +890,14 @@ def test_the_chain_keeps_an_acronym_assign_will_not_peel() -> None: lex = _AMBIGUOUS_LEX.add(titles={"st"}, particles={"st"}) out = _grouped("St van Berg Ma", lexicon=lex) assert _piece_texts(out) == [["St", "van Berg Ma"]] - # with a given word of its own the three pieces survive the chain + # MOVED by #289, not deleted: with a given word of its own the + # three pieces used to survive the chain (words to spare read 'Ma' + # as a credential); now 'Ma' leans SURNAME (Title case, mixed-case + # name) and the peel declines it regardless of the count, so the + # second re-ask absorbs it into the particle run too + # (decisions.md#S2). out = _grouped("St John van Berg Ma", lexicon=lex) - assert _piece_texts(out) == [["St", "John", "van Berg", "Ma"]] + assert _piece_texts(out) == [["St", "John", "van Berg Ma"]] def test_the_chain_keeps_a_numeral_the_peel_does_not_take() -> None: @@ -897,8 +915,13 @@ def test_the_chain_stops_before_a_bare_acronym_with_words_to_spare() -> None: # S2's other fork, the same way: 'John Smith Ma' peels the acronym # as a credential, so 'John van der Berg Ma' does too -- 1.4.0 read # suffix 'Ma' there, and 2.0 had let the chain take it. + # + # MOVED by #289, not deleted: 'Ma' is Title-case in a mixed-case + # name, so it now leans SURNAME and the chain's re-ask no longer + # stops before it -- 'Ma' joins the particle run instead + # (decisions.md#S2's corpus row). out = _grouped("John van der Berg Ma", lexicon=_AMBIGUOUS_LEX) - assert _piece_texts(out) == [["John", "van der Berg", "Ma"]] + assert _piece_texts(out) == [["John", "van der Berg Ma"]] def test_the_maiden_walk_stops_before_the_numeral_too() -> None: diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 91153f53..c7d4b5ef 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -5,30 +5,48 @@ These pin the two contracts that shape cannot reach: a defensive branch no parse can produce, and the stability its readers rest on. """ +from collections.abc import Sequence, Set + import pytest -from nameparser._lexicon import Lexicon +from nameparser._lexicon import Lexicon, _normalize +from nameparser._pipeline import STAGES from nameparser._pipeline._assign import assign from nameparser._pipeline._classify import classify from nameparser._pipeline._group import group from nameparser._pipeline._pieces import ( _numeral_behind_the_initial_veto, is_leading_title, leading_titles, - peel_trailing, peel_walk, segment_suffix_reading, trailing_titles, + own_words, peel_trailing, peel_walk, segment_suffix_reading, + trailing_titles, ) from nameparser._pipeline._segment import segment -from nameparser._pipeline._state import ParseState +from nameparser._pipeline._state import ParseState, WorkToken from nameparser._pipeline._tokenize import tokenize +from nameparser._pipeline._vocab import is_one_case, is_title_shaped, tag_marker_runs from nameparser._policy import Policy -def _through_group(text: str) -> ParseState: +def _through_group(text: str, policy: Policy = Policy()) -> ParseState: state = ParseState(original=text, lexicon=Lexicon.default(), - policy=Policy()) + policy=policy) for stage in (tokenize, segment, classify, group): state = stage(state) return state +def _state_through(stage_name: str, text: str) -> ParseState: + """Run the pipeline up to and including the named stage (STAGES' + own names), for a test that needs a state a later stage has not + yet touched (#289/#516).""" + state = ParseState(original=text, lexicon=Lexicon.default(), + policy=Policy()) + for stage in STAGES: + state = stage(state) + if stage.__name__ == stage_name: + break + return state + + def test_the_numeral_veto_refuses_a_multi_token_piece() -> None: """The len(piece) != 1 guard, which no parse can exercise. @@ -69,7 +87,8 @@ def test_the_reading_is_positional_and_total() -> None: assign's gate and its router are what remain.""" state = _through_group("Smith, MD PSM I") reading = segment_suffix_reading( - state.pieces[1], state.piece_tags[1], list(state.tokens), True) + state.pieces[1], state.piece_tags[1], list(state.tokens), True, + state.one_case) assert reading is not None assert len(reading) == len(state.pieces[1]) assert all(isinstance(v, bool) for v in reading) @@ -86,11 +105,12 @@ def test_the_reading_does_not_move_when_roles_are_assigned() -> None: """ state = _through_group("Smith, PSM I") before = segment_suffix_reading( - state.pieces[1], state.piece_tags[1], list(state.tokens), True) + state.pieces[1], state.piece_tags[1], list(state.tokens), True, + state.one_case) after_state = assign(state) after = segment_suffix_reading( after_state.pieces[1], after_state.piece_tags[1], - list(after_state.tokens), True) + list(after_state.tokens), True, after_state.one_case) assert before == after == (True, True) @@ -103,8 +123,8 @@ def test_strict_ends_the_run_at_the_initial_shaped_numeral() -> None: """ state = _through_group("Smith, PSM I") args = (state.pieces[1], state.piece_tags[1], list(state.tokens)) - assert segment_suffix_reading(*args, True) == (True, True) - assert segment_suffix_reading(*args, False) is None + assert segment_suffix_reading(*args, True, state.one_case) == (True, True) + assert segment_suffix_reading(*args, False, state.one_case) is None def _leading(text: str) -> int: @@ -168,7 +188,7 @@ def _trailing(text: str) -> int: pieces, ptags = state.pieces[0], state.piece_tags[0] tokens = list(state.tokens) rest = peel_walk(leading_titles(pieces, ptags, tokens), ptags) - peeled = peel_trailing(rest, pieces, ptags, tokens) + peeled = peel_trailing(rest, pieces, ptags, tokens, state.one_case) return trailing_titles(rest[:peeled.names], pieces, ptags, tokens) @@ -235,6 +255,98 @@ def test_the_trailing_run_refuses_a_joined_piece() -> None: assert _trailing("John Smith Prof. and Dr.") == 3 +def _peel_inputs(text: str, policy: Policy = Policy() + ) -> tuple[list[int], Sequence[Sequence[int]], + Sequence[Set[str]], + Sequence[WorkToken]]: + """The trailing peel's own inputs for segment 0, run through + `group` (#289/#516's `one_case` reaches the peel only after group + has assigned tags, so the shorter `_through_group` fixture -- not + a bare tokenize+segment -- is what these tests need). + + The walk starts AFTER the leading title run, exactly as + `_assign_main` and `_trailing` above start it: `peel_walk(0, ...)` + would leave a leading title piece sitting in `rest` as an ordinary + name piece the walk never filters out (peel_walk only drops + group-flagged suffix pieces), and 'Mr MA' needs the title excluded + to reach the one-piece floor its own test pins. + """ + state = _through_group(text, policy) + pieces, ptags = state.pieces[0], state.piece_tags[0] + tokens = state.tokens + start = leading_titles(pieces, ptags, tokens) + return peel_walk(start, ptags), pieces, ptags, tokens + + +def test_peel_trailing_takes_the_three_lean_outcomes() -> None: + # #289 at the trailing slot, the three outcomes against the peel's + # three: a CREDENTIAL lean consumes with no words to spare, a NAME + # lean declines WITH words to spare, and no lean at all leaves + # rules.md#S2's count deciding exactly as it did. + # Every case still REPORTS: the pick is appended either way, which + # is what the fork's report is built from. + for text, one_case, names, picked in ( + ("Jack MA", False, 1, True), # credential lean, k == 2 + ("Jack Ma", False, 2, True), # name lean, k == 2 + ("Jack MA", True, 2, True), # one case: today's count + ("John Smith Ma", False, 3, True), # name lean, k == 3 + ("John Smith MA", False, 2, True), # credential lean + ("JOHN SMITH MA", True, 2, True), # one case: the count + ): + rest, pieces, ptags, tokens = _peel_inputs(text) + peeled = peel_trailing(rest, pieces, ptags, tokens, + one_case=one_case) + assert peeled.names == names, (text, one_case) + assert bool(peeled.picks) is picked, (text, one_case) + + +def test_peel_trailing_keeps_its_two_piece_floor_under_a_lean() -> None: + # The floor is not what the lean moves. One piece behind a title + # never reaches the peel at all -- measured 2026-09-15, 'Mr MA' + # reads title 'Mr', family 'MA' and reports nothing -- and a lean + # that reached below `k >= 2` would read it as a title with a + # credential and no name at all. + rest, pieces, ptags, tokens = _peel_inputs("Mr MA") + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case=False) + assert peeled.names == len(rest) + assert peeled.picks == () + + +def test_peel_trailing_stops_at_a_declined_ambiguous_pick() -> None: + # The accepted cost (decisions.md#S2): the surname lean breaks the + # walk AT 'Ma', so the unambiguous 'Jr' in front of it is never + # reached and becomes a name word. The walk stops at the declined + # pick rather than continuing past it. + rest, pieces, ptags, tokens = _peel_inputs("abdul Smith Jr Ma") + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case=False) + assert peeled.names == len(rest) + + +def test_a_shape_only_token_reports_without_being_taken() -> None: + # `Policy(unlisted_dotted_suffixes=False)`: name material + # everywhere, as 2.3 read it -- and + # the fork is still reported, because the parser chose the name + # reading over a credential one and that is the call a caller + # wants told (#516). + rest, pieces, ptags, tokens = _peel_inputs( + "John Smith X.Y.Z.", Policy(unlisted_dotted_suffixes=False)) + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case=False) + assert peeled.names == len(rest) + assert peeled.picks != () + + +def test_a_by_shape_token_takes_the_count_and_never_a_lean() -> None: + # A token admitted by SHAPE carries no writing convention to read, + # so the count decides it in either case spelling (#516). + for text, names in (("John Smith X.Y.Z.", 2), + ("john smith x.y.z.", 2), + ("Jack X.Y.Z.", 2)): + rest, pieces, ptags, tokens = _peel_inputs(text) + peeled = peel_trailing(rest, pieces, ptags, tokens, one_case=False) + assert peeled.names == names, text + assert peeled.picks != (), text + + @pytest.mark.parametrize("text, expected", [ ("Xyz.", True), # H2 (rules.md): an unlisted Latin abbreviation ("김민준.", False), # #323: hangul has no period abbreviations @@ -250,3 +362,130 @@ def test_leading_title_shape_refuses_an_initialless_script( state = _through_group(text + " Smith") assert is_leading_title(state.pieces[0][0], state.piece_tags[0][0], state.tokens) is expected + + +@pytest.mark.parametrize("text", [ + "Xyz.", "Dr.", "Xyz", "X.", "田中.", "김민준.", "たなか.", "Kim김.", "J.", +]) +def test_is_title_shaped_and_is_leading_title_agree(text: str) -> None: + # The two spellings of H2's shape test -- `is_title_shaped`'s own + # docstring and `is_leading_title`'s inline copy each point here -- + # kept in step by a check over the UNION of both predicates' own + # example tables (this one and test_vocab.py's + # test_is_title_shaped_is_h2_s_shape_alone), rather than by a + # sentence alone (#289/#516 quality-review follow-up). + # 'Dr.' answers True through `is_leading_title`'s EARLIER + # `is_title_piece` (LISTED vocabulary) branch, not through the + # inline shape copy this test means to pin -- it stays in the + # union for parity with `is_title_shaped`'s own table (which + # answers True the same way, for the same reason: listed or not + # is a question neither predicate asks), not because it exercises + # the shape branch on either side. + state = _through_group(text + " Smith") + assert is_title_shaped(text) == is_leading_title( + state.pieces[0][0], state.piece_tags[0][0], state.tokens) + + +def test_own_words_is_the_name_s_own_span_and_its_clause_cut() -> None: + # rules.md#P3 says the case question is asked of the name's OWN + # words. This helper is that span, taken off the token stream so + # both the site that runs before classify and classify itself ask + # one function (#289/#516). + state = _state_through("segment", "Jane née Jones Smith") + texts, cut = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers) + assert texts == ["Jane"] + assert cut == 1 + # a delimited clause's tokens arrive with a role already set, so + # they are not the name's own words either + state = _state_through("segment", "Andrew (Andy) Perkins") + texts, cut = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers) + assert texts == ["Andrew", "Perkins"] + assert cut == 3 + # `comma_offsets` is load-bearing only for a marker PHRASE whose + # words would otherwise straddle a comma: 'z domu' is one entry, + # and 'z' and 'domu' sit on opposite sides of the comma in 'Anna + # z, domu Nowak'. With the real offsets the two words are in + # different buckets, the phrase cannot complete, and clause_at + # falls through to the whole name; passing () collapses every + # token into one bucket, the phrase wrongly completes, and the + # cut lands at 'z' instead. Measured 2026-09-17. + state = _state_through("segment", "Anna z, domu Nowak") + texts, cut = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers) + assert texts == ["Anna", "z", "domu", "Nowak"] + assert cut == 4 + texts, cut = own_words(state.tokens, (), state.lexicon.maiden_markers) + assert texts == ["Anna"] + assert cut == 1 + + +def test_own_words_takes_a_marker_map_when_the_caller_has_one() -> None: + # classify has already decided which tokens are marker RUN heads, + # so it hands that map over rather than paying a second walk. + state = _state_through("segment", "Jane née Jones Smith") + heads = {1: "vocab:maiden-marker", 2: "vocab:maiden-marker-cont"} + assert own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, + heads) == (["Jane"], 1) + # an incomplete phrase entry: no head tag, so the map's answer is + # "no clause" -- and the self-built path agrees, since both now + # run the SAME completion test (#289/#516) + state = _state_through("segment", "Anna z Nowak") + assert own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, + {}) == (["Anna", "z", "Nowak"], 3) + assert own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers)[1] == 3 + + +def test_own_words_two_spellings_agree_on_the_one_case_verdict() -> None: + # #289/#516's fix: own_words' self-built path (no marker_tags map) + # now calls _vocab.tag_marker_runs -- the SAME run-completion test + # classify uses -- instead of approximating it with a text-only + # head walk (the retired first_marker_head). The two spellings + # below -- an explicit map built ahead of time, and none at all -- + # are now the SAME computation, so they agree on the CUT, not just + # the verdict, by construction rather than by corpus luck. + # + # 'ANNA z Nowak, MD' is why the old approximation was not merely + # imprecise but wrong: 'z' opens the phrase entry 'z domu', which + # completes neither way, but the text-only walk took the open + # alone as the cut (1: just 'ANNA', one-case True), while the real + # run-completion test finds no run and falls through to the full + # own-words span (mixed case, False) -- a flipped verdict, not a + # different span with the same answer. Measured 2026-09-17. + for text in ("Anna z Nowak", "Anna z (domu) Nowak", "ANNA z Nowak, MD"): + state = _state_through("segment", text) + folded = [_normalize(t.text) for t in state.tokens] + built_map = tag_marker_runs(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, folded) + tagged = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, built_map) + walked = own_words(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers) + assert tagged == walked, text + assert is_one_case(tagged[0]) == is_one_case(walked[0]), text + + +def test_tag_marker_runs_answers_in_ascending_index_order() -> None: + """`own_words` depends on it, and nothing else said so. + + Its own docstring states the requirement -- "`marker_tags`' keys + must arrive in index order for the walk below to find the SMALLEST + head in one pass" -- and then satisfies it by BREAKING at the first + head. So a map whose keys arrived out of order would cut the span + at whichever head happened to come first, silently, and every + caller's one-case verdict with it. The guarantee lives in + `tag_marker_runs`' left-to-right walk; this is the assertion that + holds it there (2026-09-18 review round). + """ + for text in ("Anna z Nowak nee Jones", "Jane née Jones geb Schmidt", + "Anna z domu Nowak nee Jones", "John Smith"): + state = _state_through("segment", text) + folded = [_normalize(t.text) for t in state.tokens] + keys = list(tag_marker_runs(state.tokens, state.comma_offsets, + state.lexicon.maiden_markers, + folded)) + assert keys == sorted(keys), text diff --git a/tests/v2/pipeline/test_segment.py b/tests/v2/pipeline/test_segment.py index cd51fa29..b95710e0 100644 --- a/tests/v2/pipeline/test_segment.py +++ b/tests/v2/pipeline/test_segment.py @@ -10,7 +10,8 @@ # synthetic vocabulary: behavior given a lexicon, never default() contents _LEX = Lexicon( - suffix_acronyms=frozenset({"phd"}), + suffix_acronyms=frozenset({"phd", "md", "do", "dds", "ma", "ed"}), + suffix_acronyms_ambiguous=frozenset({"ma", "ed", "do"}), suffix_words=frozenset({"jr", "v"}), ) @@ -120,3 +121,89 @@ def test_strict_comma_suffixes_veto_lenient_only_members() -> None: policy=dataclasses.replace(Policy(), lenient_comma_suffixes=False)) out = segment(tokenize(extract_delimited(state))) assert out.structure is Structure.FAMILY_COMMA + + +def test_structure_flips_for_the_ambiguous_class_on_a_name_word_count() -> None: + # rules.md#C1 for the ambiguous class: two or more NAME words + # before the comma read the part after it as the credential run. + # 'John Smith, MA' flips; 'Smith Jr., MA' does not, having two + # TOKENS and one name word -- which is what keeps Smith in the + # family (#289, decisions.md#S2). + assert _segmented("John Smith, MA").structure is Structure.SUFFIX_COMMA + assert _segmented("Smith Jr., MA").structure is Structure.FAMILY_COMMA + assert _segmented("Smith, MA").structure is Structure.FAMILY_COMMA + # the extension reaches the LISTED set whatever the case says, so + # a one-case name flips too -- 1.4.0 read all three as suffixes + assert _segmented("JOHN SMITH, MA").structure is Structure.SUFFIX_COMMA + assert _segmented("john smith, ma").structure is Structure.SUFFIX_COMMA + assert _segmented("John Smith, Ed").structure is Structure.SUFFIX_COMMA + assert _segmented("Davis Royce, Ed").structure is Structure.SUFFIX_COMMA + assert _segmented("Royce, Ed").structure is Structure.FAMILY_COMMA + + +def test_structure_flips_for_a_by_shape_member_too() -> None: + # #516: an unlisted dotted token joins the class the same way, via + # `_vocab.ambiguous_class_candidate` -- 'A.B.' is two unclaimed + # single-letter chunks, not vocabulary at all, so this is the + # by-shape twin of the test above. 'Smith Jr., A.B.' does not flip + # for the SAME reason 'Smith Jr., MA' does not (#516 review round: + # is_wholly_suffix must never admit the shape class here, or C1's + # legacy TOKEN-count disjunct flips it wrongly on 'Jr.'). + assert _segmented("John Smith, A.B.").structure is Structure.SUFFIX_COMMA + assert _segmented("Smith, A.B.").structure is Structure.FAMILY_COMMA + assert _segmented("Smith Jr., A.B.").structure is Structure.FAMILY_COMMA + + +def test_segment_records_the_case_fact_only_where_it_asked() -> None: + # (a-lazy): the fact costs nothing on a name whose comma form + # could not turn on it, and nothing at all on a comma-less name. + assert _segmented("John Smith MA").one_case is None + assert _segmented("Smith, John Q. Public").one_case is None + assert _segmented("John Smith, MA").one_case is False + assert _segmented("JOHN SMITH, MA").one_case is True + # the regression test for the eager-gate fix: a single-token + # post-comma part that is NOT a member of the ambiguous class + # must not force the fact either -- membership is tested + # case-free first, and only a genuine candidate pays for it. + assert _segmented("Smith, John").one_case is None + assert _segmented("John Smith, Jr.").one_case is None + + +def test_a_tail_segment_of_leaning_credentials_is_not_flagged() -> None: + # The third reading site, and the one place this design QUIETS a + # report: 'DO' leans credential in a mixed-case name, so the third + # segment is a credential run and rules.md#C2's flag stops firing. + state = _segmented("Steven Hardman, MD, DO, DDS") + assert not [a for a in state.ambiguities + if a.kind is AmbiguityKind.COMMA_STRUCTURE] + # the one-case spelling keeps today's reading, flag and all + state = _segmented("STEVEN HARDMAN, MD, DO, DDS") + assert [a for a in state.ambiguities + if a.kind is AmbiguityKind.COMMA_STRUCTURE] + + +def test_the_structure_flip_reports_a_verbatim_detail() -> None: + # The first report of the comma's OWN decision in the library + # (#289): C2's structural flag already reports on the comma path, + # but it reports what the parse could not recognize, not a fork + # it called. P6's attachment fork has separately reported on a + # family-comma path since 2.3 ("Berg, Jan vd"). Pinned verbatim so + # a wording edit is a deliberate one, + # not a silent drift the case table's looser `ambiguities=` tuple + # check would never catch. + state = _segmented("John Smith, MA") + (amb,) = [a for a in state.ambiguities + if a.kind is AmbiguityKind.SUFFIX_OR_NAME] + assert amb.detail == ( + "'MA' after the comma is also an ordinary name word; the part " + "before the comma holds 2 name words, so it is read as a " + "credential run") + + +def test_the_structure_flip_report_counts_the_real_pre_comma_words() -> None: + # The count is INTERPOLATED, not a hardcoded "two": a three-word + # pre-comma part reports its own count. + state = _segmented("John Q. Public, MA") + (amb,) = [a for a in state.ambiguities + if a.kind is AmbiguityKind.SUFFIX_OR_NAME] + assert "holds 3 name words" in amb.detail diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index 09750e2b..664d04d3 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -18,6 +18,7 @@ def test_state_defaults_are_empty() -> None: assert s.comma_offsets == () and s.interpunct_offsets == () assert s.dropped == () and s.piece_tags == () assert s.segmenter is None + assert s.one_case is None def test_state_is_frozen_and_replace_works() -> None: @@ -53,7 +54,12 @@ def test_stage_field_ownership() -> None: # a character offset that tokenize resolves to a token index "tokenize": {"tokens", "comma_offsets", "interpunct_offsets", "ambiguities"}, - "segment": {"segments", "structure", "ambiguities"}, + # segment records `one_case` too, lazily, only where a comma + # form could turn it on -- a single-token part after the first + # comma, the shape the ambiguous class comes in (#289/#516). + # A comma-less name, or one whose post-comma part is not a + # single token, never asks and pays nothing. + "segment": {"segments", "structure", "ambiguities", "one_case"}, # script_segment splits one unspaced CJK token into n+1 pieces # (n = 1 from the vocabulary, any n from a segmenter), so it # rewrites tokens and shifts every later index the earlier @@ -67,7 +73,7 @@ def test_stage_field_ownership() -> None: # classify also emits SUFFIX_OR_NICKNAME: the delimiter escape # that decides it lives in extract_delimited, which has no token # index to point at, so the report is raised here instead - "classify": {"tokens", "ambiguities"}, + "classify": {"tokens", "ambiguities", "one_case"}, # group also emits PARTICLE_OR_GIVEN: the prefix chain takes # the particle branch of a fork whose given branch _assign # takes, so each stage reports the side it decides diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 83903dc8..053f0f10 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -2,11 +2,17 @@ import pytest -from nameparser._lexicon import Lexicon, _normalize, _title_key +from nameparser import Parser +from nameparser._lexicon import ( + Lexicon, _VOCAB_FIELDS, _normalize, _title_key, +) from nameparser._pipeline._vocab import ( + ambiguous_class_candidate, ambiguous_class_member, ambiguous_lean, + caps_shape_candidate, effective_script, is_initial, is_initial_shaped, is_one_case, - is_suffix_lenient, is_suffix_strict, is_wholly_suffix, - maiden_marker_run, resolve_script_set, single_script, + is_suffix_lenient, is_suffix_strict, is_title_shaped, is_wholly_suffix, + maiden_marker_run, name_word_count, period_joined_vocab, + resolve_script_set, single_script, ) from nameparser._policy import (Policy, Script, _NO_INITIALS, _SCRIPT_RANGES) @@ -251,6 +257,234 @@ def test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal() -> None: ["V."], lex, Policy(lenient_comma_suffixes=False)) +def test_is_wholly_suffix_never_reads_the_by_shape_class() -> None: + # #516 review round: an EARLIER version of this predicate admitted + # a by-shape member unconditionally, bypassing both the lean and + # the NAME-word count -- combined with C1's own legacy TOKEN-count + # disjunct in _segment.py, that flipped 'Smith Jr., A.B.' to a + # one-word given with a self-contradicting report. Proved by + # mutation to be otherwise unreached, and dropped: the by-shape + # class reaches the comma form only through + # `ambiguous_class_candidate`, never through this predicate, on + # or off. + lex, pol = Lexicon.default(), Policy() + assert not is_wholly_suffix(["A.B."], lex, pol) + assert not is_wholly_suffix( + ["A.B."], lex, Policy(unlisted_dotted_suffixes=False)) + # whole-token vocabulary is untouched either way -- it never went + # through the shape branch this predicate lost + assert is_wholly_suffix(["A.B.C."], lex, pol) + assert is_wholly_suffix(["A.B.C."], lex, + Policy(unlisted_dotted_suffixes=False)) + + +def test_is_wholly_suffix_reads_the_credential_lean() -> None: + # The third reading site (#289): segment's structure decision and + # its tail segments ask this, and 'Steven Hardman, MD, DO, DDS' + # loses its comma-structure flag because 'DO' leans credential + # here. A caller with nothing to say passes nothing and gets the + # answer every release before this one gave. + lex, pol = Lexicon.default(), Policy() + assert not is_wholly_suffix(["DO"], lex, pol) + assert is_wholly_suffix(["DO"], lex, pol, one_case=False) + assert not is_wholly_suffix(["Do"], lex, pol, one_case=False) + assert not is_wholly_suffix(["DO"], lex, pol, one_case=True) + assert not is_wholly_suffix(["DO", "Smith"], lex, pol, one_case=False) + + +def test_ambiguous_class_member_is_the_comma_form_s_candidate() -> None: + # The comma structure asks a different question from the lean: is + # this token a member of the ambiguous class AT ALL, in any case? + # -- because the count of NAME words before the comma is what + # decides there, and it decides for the listed set too + # ('JOHN SMITH, MA', 1.4.0 parity restored). Case-free: this + # predicate takes no `one_case` at all, unlike the lean. + lex = Lexicon.default() + assert ambiguous_class_member("MA", lex) + assert ambiguous_class_member("Ma", lex) + assert ambiguous_class_member("ed", lex) + assert not ambiguous_class_member("PhD", lex) + assert not ambiguous_class_member("Smith", lex) + # whole-token vocabulary wins over any shape reading + assert not ambiguous_class_member("M.A.", lex) + # a period ANYWHERE excludes membership here, deliberately + # stricter than S2's own dotted-form test: the trailing-period + # spelling still leans (ambiguous_lean('MA.', ...) reads as 'MA' + # does) but reaches the comma slot through the TAG path, not this + # predicate, which only the comma-form's own candidate check and + # the credential-lean disjunct in is_wholly_suffix consult. + assert not ambiguous_class_member("MA.", lex) + assert not ambiguous_class_member("Ed.", lex) + + +def test_period_joined_vocab_retires_the_single_character_chunk() -> None: + # #516, NARROWLY: the chunk rule survives except where every chunk + # the vocabulary matches is a single ASCII character, which is the + # roman numeral reaching a word that is not about generations at + # all. CHARACTER because '2' is a digit and in the set, ASCII + # because '씨' is the one that must KEEP its claim. The roster + # itself is asserted below, not just described, so a future + # vocabulary change cannot silently drift this test's premise. + lex = Lexicon.default() + assert {c for c in lex.suffix_acronyms | lex.suffix_words + if len(c) == 1 and c.isascii()} == {"2", "i", "v"} + assert period_joined_vocab("R.A.I.", lex) == "shape" + assert period_joined_vocab("X.Y.I.", lex) == "shape" + assert period_joined_vocab("J.u.n.i.o.r.", lex) == "shape" + assert period_joined_vocab("Msc.Ed.", lex) == "suffix" # 'ed', two chars + assert period_joined_vocab("JD.CPA", lex) == "suffix" + assert period_joined_vocab("J.씨", lex) == "suffix" # not ASCII + assert period_joined_vocab("Lt.Gov.", lex) == "title" # title wins + # the shape itself: two or more chunks nothing claims + assert period_joined_vocab("X.Y.Z.", lex) == "shape" + assert period_joined_vocab("B.Tech.", lex) == "shape" + assert period_joined_vocab("Q.W.E.R.T.", lex) == "shape" + assert period_joined_vocab("E.S.Q.", lex) == "shape" + # one trailing period is not the shape, and never was + assert period_joined_vocab("Xyz.", lex) is None + # a bare digit chunk is never an acronym by shape either + assert period_joined_vocab("1.4", lex) is None + # nor is a CJK word glued into period-separated single characters: + # a script with no period abbreviations at all has nothing for + # interior periods to abbreviate (#323's reasoning, shared with + # is_title_shaped) + assert period_joined_vocab("田.中.", lex) is None + assert period_joined_vocab("이.박.", lex) is None + assert period_joined_vocab("たな.か.", lex) is None + + +def test_ambiguous_class_candidate_admits_a_by_shape_member() -> None: + # #516: the comma form's own candidate test reaches a by-shape + # member too, once Policy admits it -- case-free either way, the + # periods being the whole signal. + lex, pol = Lexicon.default(), Policy() + assert ambiguous_class_candidate("A.B.", lex, pol) + assert ambiguous_class_candidate("MA", lex, pol) # listed, unaffected + assert not ambiguous_class_candidate( + "A.B.", lex, Policy(unlisted_dotted_suffixes=False)) + # whole-token vocabulary wins over the shape reading here too + assert not ambiguous_class_candidate("A.B.C.", lex, pol) + assert not ambiguous_class_candidate("M.A.", lex, pol) + assert not ambiguous_class_candidate("Smith", lex, pol) + + +def test_a_listed_dotted_entry_is_not_read_by_shape() -> None: + # Review round, 2026-09-18. A caller may list a DOTTED entry in + # `suffix_acronyms_ambiguous`, and then the whole token matches + # that set while `suffix_as_written`'s period-free acronym lookup + # ('ab') misses it -- so the chunk view reached the shape verdict + # and called a LISTED member by-shape, which silences the case + # lean everywhere downstream (`_pieces.listed_lean` declines + # wherever SHAPE_ACRONYM_TAG rides). Both halves of the class + # test the same membership now. + lex = Lexicon.default().add(suffix_acronyms={"a.b"}, + suffix_acronyms_ambiguous={"a.b"}) + assert not ambiguous_class_candidate("A.B.", lex, Policy()) + # the shipped vocabulary carries no dotted ambiguous entry, so + # nothing default moves + assert ambiguous_class_candidate("A.B.", Lexicon.default(), Policy()) + + +def test_a_callers_own_conjunction_marker_keeps_its_word_a_name() -> None: + # The exclusion end to end through a CALLER's vocabulary rather + # than the shipped one: `conjunctions_ambiguous` has no subset + # check of its own (an orphan decides nothing, _lexicon's own + # note), so a caller can list a word there alone -- and the caps + # shape test must still decline it. 'John Smith ZZQ' reads suffix + # 'ZZQ' with the switch on and the default vocabulary; one + # wordlist entry is the whole difference (2026-09-18 review + # round). + on = Policy(unlisted_caps_suffixes=True) + plain = Parser(policy=on).parse("John Smith ZZQ") + assert (plain.family, plain.suffix) == ("Smith", "ZZQ") + listed = Parser( + lexicon=Lexicon.default().add(conjunctions_ambiguous={"zzq"}), + policy=on).parse("John Smith ZZQ") + assert (listed.middle, listed.family, listed.suffix) == ( + "Smith", "ZZQ", "") + assert listed.ambiguities == () + + +def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: + # The roster is `_lexicon._VOCAB_FIELDS`, not a list written into + # the predicate, and this is what says so. The hand-written one it + # replaced named ELEVEN of the thirteen, leaving out `surnames` + # and `honorific_tails` -- so a caller who listed a word as a + # SURNAME still had it read as a credential, which is this + # switch's own worst failure arriving through the one wordlist + # that says "this is a family name". + # + # Measured per field rather than asserted about the source: each + # loop adds the same unlisted word to ONE field and checks the + # predicate declines it. A field whose exclusion is dropped fails + # here by name. + on = Policy(unlisted_caps_suffixes=True) + base = Lexicon.default() + assert caps_shape_candidate("ZZQX", base, on, False) + for field in _VOCAB_FIELDS: + extra: dict[str, set[str]] = {field: {"zzqx"}} + if field == "honorific_tails": + # honorific_tails ⊆ suffix_words is a Lexicon invariant, + # so this entry cannot be added alone -- which is also why + # the field was already excluded transitively, and why it + # joins the roster for completeness rather than for a + # behavior change + extra["suffix_words"] = {"zzqx"} + elif field == "given_name_titles": + extra["titles"] = {"zzqx"} # ⊆ titles in practice + elif field == "particles_ambiguous": + extra["particles"] = {"zzqx"} # enforced subset + elif field == "suffix_acronyms_ambiguous": + extra["suffix_acronyms"] = {"zzqx"} # enforced subset + lex = base.add(**extra) # type: ignore[arg-type] + assert not caps_shape_candidate("ZZQX", lex, on, False), field + + +def test_is_title_shaped_is_h2_s_shape_alone() -> None: + # Shared with _pieces.is_leading_title's own inline copy + # (#289/#516, quality-review finding): both must answer alike for + # name_word_count's comma-form count not to disagree with the + # leading peel about what a title is. + assert is_title_shaped("Xyz.") # unlisted, H2-shaped + # LISTED or not is a vocabulary question this predicate never + # asks -- 'Dr.' wears the same shape 'Xyz.' does, and answers the + # same way; the caller's own listed lookup is what tells them + # apart (is_title_piece/lexicon.titles, at each call site) + assert is_title_shaped("Dr.") + assert not is_title_shaped("Xyz") # no trailing period + assert not is_title_shaped("X.") # one letter: an initial, + # not an abbreviation + assert not is_title_shaped("田中.") # initialless script (#323) + + +def test_name_word_count_counts_names_not_tokens() -> None: + # rules.md#C1's count for the ambiguous class: 'Smith Jr.' is two + # tokens and ONE name word, which is what keeps its family where a + # token count would hand it to `given`. + lex, pol = Lexicon.default(), Policy() + assert name_word_count(["John", "Smith"], lex, pol) == 2 + assert name_word_count(["Smith", "Jr."], lex, pol) == 1 + assert name_word_count(["Dr.", "Smith"], lex, pol) == 1 + assert name_word_count(["Davis", "Royce"], lex, pol) == 2 + assert name_word_count(["Royce"], lex, pol) == 1 + # #289/#516, quality-review finding: the title half asks H2's + # shape test too, not just the listed lookup -- an UNLISTED + # period-marked opener now counts the way a LISTED one does + # ('Xyz.' beside 'Dr.', both 1), where before this it counted as + # a plain name word and could flip a comma structure a listed + # title of the same shape would not. + assert name_word_count(["Xyz.", "Smith"], lex, pol) == 1 + # 2026-09-18 review round: the two arms behind 'Mr Smith, Ma' and + # 'Smith Jr, Ma', neither of which had a unit row. A BARE title + # word and a BARE suffix word each count as no name -- the period + # is no part of either test -- so both parts hold ONE name word + # and neither comma flips. The parses are pinned in cases.py; the + # arms are pinned here, because a count of 1 for the wrong reason + # reads identically at the parse. + assert name_word_count(["Mr", "Smith"], lex, pol) == 1 + assert name_word_count(["Smith", "Jr"], lex, pol) == 1 + + # Stored form: space-joined, per-word normalized -- what _normset # writes for this field. Built as bare frozensets rather than through a # Lexicon so these exercise the predicate and nothing else. @@ -529,3 +763,24 @@ def test_is_one_case() -> None: # token that is caseless does not break a Latin name's verdict assert is_one_case(["john", "e", "山田"]) assert not is_one_case(["John", "e", "山田"]) + + +def test_ambiguous_lean_reads_the_written_case() -> None: + # #289: in a name written in more than one case, an all-caps + # member of the ambiguous set leans CREDENTIAL and a member in any + # other cased form that is not wholly lower leans SURNAME. A + # lowercase member carries no lean, and neither does anything at + # all in a one-case name -- both fall through to today's count. + assert ambiguous_lean("MA", one_case=False) == "credential" + assert ambiguous_lean("Ma", one_case=False) == "name" + assert ambiguous_lean("ma", one_case=False) is None + assert ambiguous_lean("MA", one_case=True) is None + assert ambiguous_lean("Ma", one_case=True) is None + # a trailing period is not the signal and does not disturb one: + # 'MA.' is still written in capitals ('.' has no case) + assert ambiguous_lean("MA.", one_case=False) == "credential" + assert ambiguous_lean("Ma.", one_case=False) == "name" + # a caseless token can be written against nothing, so it leans + # neither way even where the name around it is mixed + assert ambiguous_lean("씨", one_case=False) is None + assert ambiguous_lean("毛", one_case=False) is None diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index 20d1c5dc..f4ba8e8f 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -122,6 +122,15 @@ def has_boundary_or_waiver(self) -> bool: "maiden-parens": Policy(maiden_delimiters=frozenset({("(", ")")})), "keep-emoji": Policy(strip_emoji=False), "strict-comma-suffixes": Policy(lenient_comma_suffixes=False), + #: The two switches over the by-shape credential class (#516). + #: Named after the Policy FIELD rather than after the behavior, + #: which is deliberate: rules.md's statements stay + #: implementation-free, so the annotation slot is the one place + #: the doc can put the caller-facing name of the switch its prose + #: describes. The suffix says which way the field is set, since + #: one is on by default and the other off. + "unlisted_dotted_suffixes-off": Policy(unlisted_dotted_suffixes=False), + "unlisted_caps_suffixes-on": Policy(unlisted_caps_suffixes=True), } #: D-section subjects: zero-arg constructions whose diagnostics the #: warns=/raises= assertion forms exercise. diff --git a/tests/v2/test_facade_cases.py b/tests/v2/test_facade_cases.py index 975157b0..30487b15 100644 --- a/tests/v2/test_facade_cases.py +++ b/tests/v2/test_facade_cases.py @@ -44,6 +44,8 @@ "script_orders", "segment_scripts", "lenient_comma_suffixes", + "unlisted_dotted_suffixes", + "unlisted_caps_suffixes", "strip_emoji", "strip_bidi", }) @@ -106,6 +108,65 @@ # they assert are 1.4.0's for exactly that reason. "han_unspaced_no_script_orders_reports_the_convention", "kana_honorific_no_script_orders_reports_the_convention", + # #516's all-caps half: unlisted_caps_suffixes has no v1 spelling + # (v1 has no by-shape credential class at all), so every row that + # sets it is core-only. + "caps_surname_is_swallowed_with_the_switch_on", + "caps_surname_reports_but_does_not_move_at_two_words", + "unlisted_caps_reads_by_position_with_the_switch_on", + "the_caps_comma_count_needs_two_name_words", + "the_caps_comma_count_declines_at_one_word", + "one_case_input_never_reaches_the_caps_switch", + "suffix_vocabulary_never_reaches_the_caps_switch", + "the_caps_comma_count_reaches_a_multi_word_run", + "the_caps_comma_multi_word_run_declines_at_one_word", + # #516 review round: the F1/F1b/F2/F5 regression-guard rows, all + # under the same non-default policy. + "caps_switch_does_not_silence_the_listed_lean", + "caps_switch_does_not_silence_the_comma_lean", + "caps_switch_does_not_claim_a_capitalized_particle", + "caps_switch_does_not_claim_a_capitalized_particle_phrase", + "caps_switch_does_not_move_a_roman_numeral", + "caps_switch_does_not_move_a_roman_numeral_with_words_to_spare", + "caps_switch_does_not_move_a_title_floor_control", + "caps_switch_does_not_reach_delimited_content", + "caps_switch_run_test_declines_a_pure_listed_run", + "caps_switch_does_not_claim_a_one_case_maiden_marker", + "caps_switch_does_not_claim_a_mixed_case_maiden_marker", + # The 2026-09-18 review round's negative control for the + # tail-segment class test: `unlisted_dotted_suffixes` has no v1 + # spelling either (v1 reads no token by shape), so the row that + # turns it OFF is core-only, exactly as the caps rows above are. + # Its default-policy twin is an ordinary row and runs here. + "the_by_shape_tail_segment_is_flagged_with_the_switch_off", + # The same round's unpinned-branch rows, every one under a policy + # with no v1 spelling: the caps switch, and -- for the two strict + # comma rows -- `lenient_comma_suffixes`, which the block above + # already names two rows for. + "caps_switch_reads_the_name_level_case_past_a_clause", + "caps_one_case_comma_declines_a_single_token", + "caps_one_case_comma_declines_a_run", + "caps_run_needs_every_token_not_any", + "caps_run_declines_a_bound_given_head", + "caps_run_declines_a_conjunction", + "caps_switch_leaves_a_capitalized_title_a_title", + "strict_comma_reads_the_dotted_numeral_as_a_name_word", + "strict_comma_reads_the_bare_numeral_into_the_run", + "the_comma_flip_is_read_under_the_declared_order", + "two_caps_credentials_peel_as_a_run", + "the_caps_shape_is_script_agnostic_cyrillic", + "the_caps_shape_is_script_agnostic_accented", + "the_dotted_switch_off_still_reports_the_post_comma_fork", + "the_dotted_switch_off_leaves_a_lone_token_to_the_convention", + "the_dotted_switch_off_leaves_a_titled_token_a_name", + # The 2026-09-18 verification round's two unpinned-branch rows, + # both under a policy with no v1 spelling: the caps switch again, + # and `unlisted_dotted_suffixes` turned OFF, which is the only + # policy that makes the chain emitter's by-shape tag reachable + # (at the default, classify writes both tags and the listed one + # answers first). + "the_caps_shape_never_reaches_a_tail_segment", + "the_chain_reports_the_by_shape_half_too", }) diff --git a/tests/v2/test_layering.py b/tests/v2/test_layering.py index 70210512..c5db0a47 100644 --- a/tests/v2/test_layering.py +++ b/tests/v2/test_layering.py @@ -65,15 +65,25 @@ "_pipeline/_state.py": ("nameparser._types", "nameparser._lexicon", "nameparser._policy"), "_pipeline/__init__.py": ("nameparser._pipeline.",), - "_pipeline/_vocab.py": _PIPELINE_STAGE_ALLOWED, + # Tighter than the stage allowance on purpose: _vocab takes + # normalized-or-raw text explicitly, no state, with named + # departures (its own docstring) -- tag_marker_runs is one, and it + # reads WorkToken.role/span and comma_bucket, never a whole + # ParseState. Widening this past _state is the tell that a text- + # level predicate has grown a dependency on a stage. + "_pipeline/_vocab.py": ("nameparser._lexicon", "nameparser._policy", + "nameparser._pipeline._state"), "_pipeline/_extract.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_tokenize.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_script_segment.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_segment.py": _PIPELINE_STAGE_ALLOWED, "_pipeline/_classify.py": _PIPELINE_STAGE_ALLOWED, - # Piece-level predicates shared by group and assign - # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). Tighter than the - # stage allowance on purpose: it is a leaf both stages sit on, so + # Piece-level predicates shared by four stages -- segment, + # classify, group and assign + # (mechanisms.md#ONE-PREDICATE-PER-QUESTION). Segment joined with + # #289/#516's own-words span, and "both stages" below was written + # when there were two. Tighter than the + # stage allowance on purpose: it is a leaf those stages sit on, so # it may read the token type and the vocabulary layer and NOTHING # else -- not even _lexicon, which is the likeliest next reach (a # title predicate wanting _title_key) and so the one this entry @@ -219,7 +229,7 @@ def test_every_pipeline_module_is_keyed_in_allowed() -> None: f"from the layering contract entirely: {missing}. Add each with " f"the narrowest prefix tuple that admits what it actually " f"imports -- _PIPELINE_STAGE_ALLOWED for a stage, something " - f"tighter for a leaf both stages sit on (see _pipeline/_pieces.py)" + f"tighter for a leaf several stages sit on (see _pipeline/_pieces.py)" ) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 5eb2fe3e..35e73896 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -863,12 +863,27 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: # 'Berg, Jan vd' the comma shape P6 gives to fix(#380); this rule is # the third corner, trailing and comma-less, and must not reach # either of the others. + # 2026-09-18 (#289): 'Jack MA' is a probe here NOW, where it was + # not before. This rule's own sentence -- a two-token name ending + # in a credential acronym that keeps it in `suffix` -- became TRUE + # of 'Jack MA' the day the case contrast started deciding it, and + # the only thing keeping the two apart is this regex's anchoring + # to the mc/mp pair. `fix(#289) a written case contrast decides a + # bare ambiguous acronym` is the rule that owns the name. "fix(suffix-routing) a two-token name ending in a credential acronym keeps it in `suffix`": - ("Mc Donald", "Berg, Jan vd", "John Smith MP", "Jack Ma"), + ("Mc Donald", "Berg, Jan vd", "John Smith MP", "Jack Ma", + "Jack MA"), # decisions.md#ma-do turns on the BARE spelling keeping its surname # while the dotted one reads as a credential, so both spellings of # the bare one are probes here -- and 'Jack Ma' is a probe for the # acronym rule above as well, since 'ma' is acronym vocabulary too. + # 2026-09-18 (#289): 'Jack MA' no longer keeps its surname -- the + # written case contrast reads it as a credential with nothing to + # spare -- so the sentence above is now true of 'Jack Ma' alone. + # It stays a probe here for a different reason than it arrived: + # the name is claimed by `fix(#289) a written case contrast + # decides a bare ambiguous acronym`, and this rule, anchored to + # the DOTTED spelling, must still never reach it. "fix(suffix-routing) the dotted M.A. spelling reads as a credential (ma-do)": ("Jack Ma", "Jack MA", "John Smith M.A."), # #484: the connective rule is case-sensitive on the single letters @@ -1138,6 +1153,55 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: "ХОСЕ И МАРИЯ САНТОС", "хосе и мария сантос", "محمد و علي السيد"), + # #289/#516's three rules are literal-anchored alternations, so + # _CORPUS_CLAIMS cannot see a widening that reaches only names the + # corpora lack -- these probes are the wall, and the keys are the + # FULL issue strings for the same reason #383/#479's are. + # + # #289's rule is anchored to the names a MIXED-case spelling + # moves, so every probe here is a name whose writing carries no + # contrast at all -- one case, lower case, or a caseless script -- + # plus the two dotted shapes the vocabulary settles on its own. + # A rule reaching one would be absorbing a regression in the half + # of rules.md#S2 this change did not touch. '毛泽东, MA' is the + # sharpest of them: it DOES gain the fork's report, having one + # pre-comma word and no case to read, and it must still never be + # claimed by a rule about written case. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + ("JOHN SMITH MA", "ANH DO", "anh van do", "jack ma", "Jack Ma", + "Jack Ma.", "毛泽东, MA", "Smith, A.B.C.", "Jack M.A."), + # #516's dotted rule must never reach the shapes the vocabulary + # SETTLES (a whole-token match, or a surviving chunk claim), the + # leading dotted runs rules.md#S2 excludes by position, the single + # trailing period AGENTS.md's trailing-abbreviation reading owns, + # or a digit chunk, which the alphabetic gate refuses. + "fix(#516) an unlisted dotted acronym is read by position": + ("Jack M.A.", "John Smith M.A.", "Smith, A.B.C.", + "Doe, John Msc.Ed.", "John Doe Msc.Ed.", "X.Y.Z. Smith", + "J.R.R. Tolkien", "A.B. Vajpayee", "E.T. Smith", + "John Smith Xyz.", "John Smith 1.4", "Bridge (1.4)", + # 2026-09-18 review round: the three gates on the shape + # verdict that had no probe of their own, each a DIFFERENT + # conjunct. The initialless-script gate ('田.中.' -- a script + # that writes no abbreviations has nothing for interior + # periods to abbreviate); the `token.role is None` gate + # ('Bridge (A.B)' -- delimited content the escape kept as a + # nickname, where 'Bridge (1.4)' above declines one gate + # earlier and so cannot exercise this one); and the + # alphabetic-chunk gate at a COMMA ('John Smith, 1.4', the + # comma twin of the bare 'John Smith 1.4' already here). + "John Smith 田.中.", "Bridge (A.B)", "John Smith, 1.4"), + # Policy.unlisted_caps_suffixes is OFF by default, so its whole + # population is a probe here: the corpora run at the default, and + # a rule of this arc reaching one of these names would mean the + # DEFAULT changed. 'Mr XXX' is the boundary from the other side -- + # one piece behind a title never reaches the peel's two-piece + # floor, switch or no switch. + "fix(#289/#516) the ambiguous credential class reports at slots " + "that were silent": + ("Jean DUPONT", "Jean Pierre DUPONT", "Minjun KIM", + "John Smith XYZ", "Smith, LEED AP", "John Smith, LEED AP", + "Mr XXX"), } @@ -1882,6 +1946,52 @@ class _LatinCopy(NamedTuple): # regression. One set, identical in all four ledgers. frozenset({"Ahmad Jayadi, CHA", "Aishwarya Rai", "John Smith RAI", "John Smith, RAI", "Lala Lajpat Rai"}), + # #289/#516's movers, one corpus name per alternative -- lists of + # names, not copies of any wordlist, so there is no vocabulary for + # them to drift from. Five sets rather than one, because two of + # the three rules hold a name back at one baseline where an older + # rule already owns its diff (each ledger's own comment says which + # and why), and the report-only rule gains a name from 2.2.0 on. + # + # What selects every one of them is a SHAPE the vocabulary + # participates in only at one end: a trailing or post-comma member + # of SUFFIX_ACRONYMS_AMBIGUOUS read by WRITTEN CASE (#289), or a + # multi-chunk dotted token NO vocabulary claims at all (#516). A + # member copying SUFFIX_ACRONYMS_AMBIGUOUS would reach 'Jack Ma', + # 'JOHN SMITH MA', 'ANH DO' and 'anh van do', every one of which + # stays exactly as it read -- the case contrast is the whole + # subject, and a wordlist cannot spell it. A member spelled as the + # dotted shape would reach 'Jack M.A.', 'John Smith Ph.D.' and + # 'Doe, John Msc.Ed.', which the vocabulary answers for and this + # change does not touch. _MUST_NOT_MATCH carries both rosters. + frozenset({"Davis Royce, Ed", "Freiherr von Berg MA", + "JOHN SMITH, MA", "Jack MA", r"Jack MA\.", "Jack Wei Ma", + "John Smith Ma", "John Smith, Ed", "John Smith, MA", + "John Smith, Ma", "John de Ma", "John van der Berg Ma", + r"Smith Jr\., MA", r"Smith Jr\., Ma", "Smith, MA", + "abdul Smith Berg Ma", "abdul Smith Jr Ma", + "abdul Smith Ma", "john smith, ma"}), + frozenset({"Davis Royce, Ed", "Freiherr von Berg MA", + "JOHN SMITH, MA", "Jack MA", r"Jack MA\.", "Jack Wei Ma", + r"John Prof\. MA", "John Smith Ma", "John Smith, Ed", + "John Smith, MA", "John Smith, Ma", "John de Ma", + "John van der Berg Ma", + r"Smith Jr\., MA", r"Smith Jr\., Ma", "Smith, MA", + "abdul Smith Berg Ma", "abdul Smith Jr Ma", + "abdul Smith Ma", "john smith, ma"}), + frozenset({r"Jack X\.Y\.I\.", r"John Smith B\.Tech\.", + r"John Smith C\.H\.A\.", r"John Smith E\.S\.Q\.", + r"John Smith Q\.W\.E\.R\.T\.", r"John Smith X\.Y\.Z\.", + r"John Smith, A\.B\.", r"Smith, E\.S\.Q\.", + r"john smith x\.y\.z\."}), + frozenset({r"J\.A\. K\.D\.", r"Jack X\.Y\.Z\.", + r"John Smith J\.u\.n\.i\.o\.r\.", r"John Smith R\.A\.I\.", + "Royce, Ed", r"Smith Jr\., A\.B\.", r"Smith, A\.B\.", + "Smith, Ma"}), + frozenset({r"J\.A\. K\.D\.", r"Jack X\.Y\.Z\.", + r"John Smith J\.u\.n\.i\.o\.r\.", r"John Smith R\.A\.I\.", + "Royce, Ed", r"Smith Jr\., A\.B\.", r"Smith, A\.B\.", + "Smith, Ma", "Steven Hardman, MD, DO, DDS"}), # #449's movers, one corpus name per alternative -- a list of # names, not a copy of any wordlist, so there is no vocabulary for # it to drift from. What selects these names is a SHAPE the @@ -1977,15 +2087,16 @@ class _LatinCopy(NamedTuple): # words; the same reason as fix(#400)'s pair above. frozenset({"^", "\\s"}), frozenset({"^", "[\\s,]"}), - # fix(#385/#402)'s 24 spellings: the 27 corpus names whose + # fix(#385/#402)'s 25 spellings: the 28 corpus names whose # all-particle part moved, listed because "a part of nothing but # particles" is not a property a regex over the raw string can # state. A list of names, the fix(#445) precedent -- that rule and # fix(#410) and fix(#335) are the 1.4.0 ledger's literal lists. + # 'anh van mc' joined 2026-09-18 with its case row. frozenset({"anh do", "smith van der", "yin le", "yin a le", "vai la", "jong van der", "jong, van der", "juan van der", "mesnil garcia de", "mesnil garcia van", "mesnil de", - "sander van", "van ma van", "anh van do", + "sander van", "van ma van", "anh van do", "anh van mc", "beethoven ludwig van", "berg jan de jr\\.", "john van mc", "jong anke de", "juan de", "ménil christophe de", "ménil de", "nguyen thi van", "nguyen, van le", @@ -2037,7 +2148,8 @@ class _LatinCopy(NamedTuple): # without which this whole alternation would be invisible to # _alternations above. One set, identical in all four ledgers. frozenset({r"Andrew Perkins \x28Mgr\.\x29", r"Dr\. John Smith Prof\.", - r"Dr\. Smith Sir\.", r"John Prof\. MA", r"John Smith Dr\.", + r"Dr\. Smith Sir\.", r"JOHN PROF\. MA", r"John Prof\. MA", + r"John Smith Dr\.", r"John Smith Jr\. Prof\.", r"John Smith Mr\.", r"John Smith Prof\.", r"John Smith Prof\. Dr\.", r"John Smith Prof\. Jr\.", r"John Smith Rev\.", @@ -2571,8 +2683,13 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'maiden', 'middle'), "309e39fc2475", None), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), + # 2026-09-18: 126 -> 131. Five corpus names arrived with + # #289/#516's own case rows -- 'J.씨', 'John Smith 田.中.', + # '毛泽东, MA', '田中 太郎, MA', '마틴 킹, MA' -- all of them + # names this rule already described. Growth into new + # corpus, not a widened regex. "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(126, ('family', 'given', 'middle'), "a3053e6567fb", None), + _Claim(135, ('family', 'given', 'middle'), "fd79e8e1e2ef", None), "fix(#274) maiden markers consumed": _Claim(33, ('family', 'maiden', 'middle'), "6f8bf7136b09", None), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": @@ -2606,14 +2723,22 @@ def _claim(rule: dict) -> _Claim: # on this rule's roles at any baseline -- {title, middle} -- # so this is reach without absorption, as the paragraph # above records for the four names before it. + # 2026-09-18: 289 -> 302. Thirteen comma-bearing corpus names + # arrived with #289/#516's case rows, every one inside + # this rule's Latin comma range. Growth into new corpus, + # not a widened regex. + # 2026-09-18, second round: 302 -> 304. Two more comma names, + # 'John Smith, Ma' and 'Smith Jr., Ma' -- rules.md#C1's own + # witnesses that the NAME-word count is asked before the + # written case is. "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(289, ('given', 'suffix', 'title'), "837dc1177415", None), + _Claim(310, ('given', 'suffix', 'title'), "2803b1932de1", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": _Claim(2, ('family', 'given', 'suffix'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('given', 'middle', 'title'), "263d5957cfc1", None), + _Claim(5, ('given', 'middle', 'title'), "e5aa72ebb9bf", None), # `middle` left the ROLES in the same edit, at the gate's # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, # no name the rule still explains moves a middle name. @@ -2628,8 +2753,12 @@ def _claim(rule: dict) -> _Claim: _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee", None), + # 2026-09-18: 18 -> 20. Two new corpus names, 'Smith, MA' and + # 'Smith, Ma' -- the pair #289's lean reads in opposite + # directions, both of them lone post-comma credentials + # by this rule's own description. "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444", None), + _Claim(20, ('family', 'given', 'suffix', 'title'), "c94afd0c7617", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": @@ -2638,8 +2767,12 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), # 288 -> 289 with fix(comma-family) above and for the same # one name, both rules matching on the bare comma. + # 2026-09-18: 289 -> 302, the same thirteen new comma names as + # the lone-post-comma rule above and for the same reason. + # 2026-09-18, second round: 302 -> 304, the same two new comma + # names as the rule above. "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(289, ('family', 'given'), "837dc1177415", None), + _Claim(310, ('family', 'given'), "2803b1932de1", None), "fix(#397) NOT WANTED: a trailing Catalan/Polish linking 'i' is read as a generation marker and the family is lost": _Claim(1, ('family', 'suffix'), "498602f3cfd0", None), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": @@ -2654,10 +2787,14 @@ def _claim(rule: dict) -> _Claim: _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608", None), "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7", None), + # 2026-09-18: 26 -> 29. The three CJK comma forms #289's own + # case rows added ('毛泽东, MA', '田中 太郎, MA', + # '마틴 킹, MA'). Growth into new corpus. "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(26, ('given', 'suffix'), "7145d3aa16ca", None), + _Claim(32, ('given', 'suffix'), "a11674bdc8f2", None), + # 2026-09-18: 26 -> 29, the same three new CJK comma forms. "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(26, ('family', 'given', 'suffix', 'title'), "7145d3aa16ca", None), + _Claim(32, ('family', 'given', 'suffix', 'title'), "a11674bdc8f2", None), # 37 -> 35 with the 2026-09-05 narrowing, which is a rule # NARROWING and not corpus movement: the three negative # lookbehinds stop the regex matching a listed honorific @@ -2677,8 +2814,9 @@ def _claim(rule: dict) -> _Claim: # the #322/#323 rule at the foot of the ledger explains them # instead, and the three comma rows produce no diff at this # baseline at all. `radar unclassified` stays 0. + # 2026-09-18: 40 -> 41. One new corpus name, 'J.씨'. "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(40, ('family', 'given', 'suffix'), "4d7bacfc28a4", None), + _Claim(44, ('family', 'given', 'suffix'), "1587ce883dab", None), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745", None), # 4 -> 6 on 2026-09-13: 'ХОСЕ И МАРИЯ САНТОС' and 'хосе и @@ -2734,8 +2872,12 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "42b69cf1b320", None), "fix(#424) accepted: the maiden walk keeps the numeral an initial before the marker vetoes": _Claim(1, ('family', 'maiden', 'middle', 'suffix'), "08c0158c8d3f", None), + # 2026-09-18 (#289): the reach is unchanged at one name and the + # ROLES grew by `suffix` -- 'Freiherr von Berg MA' now + # moves three roles where it moved two, which is the + # `fields` re-scope this ledger's own comment argues. "fix(#424) accepted: the chain keeps an acronym assign will not peel behind a title-and-particle word": - _Claim(1, ('family', 'given'), "faa4bedda537", None), + _Claim(1, ('family', 'given', 'suffix'), "faa4bedda537", None), "fix(#424) a title-led chain before the numeral is the one name piece": _Claim(1, ('family', 'suffix'), "5b3a743f9e35", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": @@ -2793,7 +2935,7 @@ def _claim(rule: dict) -> _Claim: # ones are protected by the [[never]] entry above, which is # what _EXCLUSION_EFFECT's grown `absorbed_by` records. "fix(#385/#402) an all-particle name part initials its words (R2)": - _Claim(27, ('_initials',), "6b242c287db8", ('DEFAULT',)), + _Claim(29, ('_initials',), "541df583f5fc", ('DEFAULT',)), "fix(#360) los joined the particles, so it no longer initials": _Claim(1, ('_initials',), "cd721215f463", ('DEFAULT',)), # #269's derived-view rule, added 2026-09-13. One corpus name, @@ -2834,8 +2976,11 @@ def _claim(rule: dict) -> _Claim: _Claim(101, ('_initials',), "e91031622dca", ('DEFAULT',)), "fix(initials-per-word) a bound-given run initials each word (facade, since 2.0.0)": _Claim(41, ('_initials',), "e99f56c955d5", ('DEFAULT',)), + # 2026-09-18: 109 -> 110. One new corpus name, + # 'john van der berg ma' -- rules.md#P2's one-case contrast, + # and a particle chain like every other member. "fix(initials-per-word) a particle chain inside a name part initials each word (facade, since 2.0.0)": - _Claim(109, ('_initials',), "ae9c8f674e0c", ('DEFAULT',)), + _Claim(110, ('_initials',), "9193a1954e3b", ('DEFAULT',)), "fix(initials-per-word) the Ph. D. merge initials each word (facade, since 2.0.0)": _Claim(18, ('_initials',), "f67d8ebddd56", ('DEFAULT',)), # The 2.3 title-run bundle's five rules, last in every @@ -2863,9 +3008,12 @@ def _claim(rule: dict) -> _Claim: # move. "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(3, ('family', 'given', 'suffix', 'title'), "07b02286cd81", None), + # 2026-09-18 (#289/#516): 16 -> 17. One new corpus name, + # 'JOHN PROF. MA', added to the alternation deliberately -- + # the ledger comment argues it. No role joined the five. "fix(#316) a trailing period-marked title word reads as a title": - _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), - "562e0e82a22b", None), + _Claim(17, ('family', 'given', 'middle', 'suffix', 'title'), + "0dc3551e57da", None), # The CJK member of the same argument, its own literal rule # (an alternation holding a script-classified member belongs # to the honorific pin above -- until #322/#323 that was every @@ -2907,6 +3055,39 @@ def _claim(rule: dict) -> _Claim: "fix(#383/#479) a single-letter connective joins only on case evidence": _Claim(2, ('family', 'given', 'middle'), "ec00806a06f0", ('DEFAULT',)), + # #289's alternation, appended with #516's and the report + # rule's below it. Literal-anchored to the movers a written + # case contrast decides, `orders` DEFAULT: the class is a + # SHAPE and not a wordlist, so a member spelled as the class + # would claim the one-case spellings that do not move. + # _MUST_NOT_MATCH carries those, and at a reach this small it + # is the wall this row cannot be. + # 2026-09-18, second round: two more names, 'John Smith, Ma' + # and 'Smith Jr., Ma', the Title-case halves of two minimal + # pairs rules.md#C1 now states. No role joined the list. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + # 2026-09-18, review round: 'John de Ma' joins, the + # one-particle spelling of 'John van der Berg Ma'. Its diff + # is the restored chain report at the 2.x baselines and the + # role move at 1.4.0 and 2.2/2.3; no role joined the list. + _Claim(19, ('family', 'given', 'middle', 'suffix'), "7465eef956d4", ('DEFAULT',)), + # #516's alternation. Literal-anchored to the by-shape movers, + # `orders` DEFAULT. Same reasoning as the rule above: the + # class is a shape the vocabulary does not spell, so the + # probes and not this number are what bound it. + "fix(#516) an unlisted dotted acronym is read by position": + _Claim(9, ('family', 'given', 'middle', 'suffix'), "9bbaf4e84dc0", ('DEFAULT',)), + # 2026-09-18, verification round. ONE name, and a rule this PR + # did not earn: 'Doe, John MA' has read middle since 2.0.0 and + # reads suffix at 1.4.0, so the diff exists at this baseline + # alone. It entered the corpus as a rules.md#S2 boundary + # example line, which is what forced the classification. + # Literal-anchored, `orders` DEFAULT: the class is a SLOT, and + # a member spelled as the slot would claim every family-comma + # name whose given part ends in an ambiguous acronym -- + # including the ones v1 read the same way. + "fix(given-part-trailing-slot) a credential acronym ending the given part of a family-comma listing reads as a middle name": + _Claim(1, ('middle', 'suffix'), "c164207a6725", ('DEFAULT',)), }, "expected_since_2.0.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -3010,16 +3191,22 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe", None), "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda", None), + # 2026-09-18: 126 -> 131. Five corpus names arrived with + # #289/#516's own case rows -- 'J.씨', 'John Smith 田.中.', + # '毛泽东, MA', '田中 太郎, MA', '마틴 킹, MA' -- all of them + # names this rule already described. Growth into new + # corpus, not a widened regex. "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(126, ('_ambiguities', 'family', 'given', 'middle'), "a3053e6567fb", None), + _Claim(135, ('_ambiguities', 'family', 'given', 'middle'), "fd79e8e1e2ef", None), # 37 -> 35 with the same 2026-09-05 narrowing as the 1.4 twin, # whose entry carries the reason. Here the one name that # changed hands, '김민준 박사님', goes to the spaced rule # fix(#307/#308/#320) -- the label its title states. # 35 -> 40 since #323 by the same five names, the 1.4 twin's # entry carrying the arithmetic. + # 2026-09-18: 40 -> 41. One new corpus name, 'J.씨'. "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(40, ('family', 'given', 'suffix'), "4d7bacfc28a4", None), + _Claim(44, ('family', 'given', 'suffix'), "1587ce883dab", None), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2", None), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": @@ -3060,12 +3247,21 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "9523e518e6ec", None), "fix(#421) the bound-given join never absorbs a split credential": _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef", None), + # 2026-09-18 (#289): the reach is unchanged at two names and the + # ROLES moved -- `given` out, `middle` in. 'abdul Smith Ma' + # stops diffing at this baseline altogether and + # 'abdul Smith Jr Ma' now reads middle 'Jr'; the ledger's + # own dated paragraph carries the argument. "fix(#425) the bound-given reserve runs assign's peel over the joined view": - _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e", None), + _Claim(2, ('family', 'middle', 'suffix'), "ef1ab03b617e", None), "fix(#424) the particle chain stops before the trailing numeral": _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf", None), + # 2026-09-18 (#289): 1 -> 2. 'john van der berg ma' entered + # the corpus as rules.md#P2's one-case contrast, and it is the + # spelling this rule still describes -- see the ledger's own + # dated paragraph, which replaced a `dormant` key with it. "fix(#424) the particle chain stops before a bare acronym with words to spare": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b", None), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "18fbd95cd89e", None), "fix(#424) a title-led chain before the numeral is the one name piece": _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": @@ -3073,7 +3269,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), + _Claim(5, ('_ambiguities', 'given', 'middle', 'title'), "e5aa72ebb9bf", None), # `middle` left the ROLES in the same edit, and the reach # grew with the rules corpus; the 1.4.0 roster above carries # both, and says the same at the other two baselines. @@ -3081,8 +3277,12 @@ def _claim(rule: dict) -> _Claim: _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), + # 2026-09-18: 18 -> 20. Two new corpus names, 'Smith, MA' and + # 'Smith, Ma' -- the pair #289's lean reads in opposite + # directions, both of them lone post-comma credentials + # by this rule's own description. "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('suffix', 'title'), "1f79efa10444", None), + _Claim(20, ('suffix', 'title'), "c94afd0c7617", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": @@ -3093,8 +3293,11 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "faa2c70fc49e", None), "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65", ('DEFAULT', 'FAMILY_FIRST')), + # 2026-09-18: 1 -> 2. One new corpus name, 'Jack MA.', the + # all-caps spelling #289's case contrast now reads as a + # credential; the rule's own 'Jack Ma.' is untouched. "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4", None), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "29b08f75e6ff", None), "fix(#367) an inferred title no longer displaces a leading particle either": _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": @@ -3130,7 +3333,7 @@ def _claim(rule: dict) -> _Claim: # digests match the 1.4.0 ledger's, which is the point of # copying the list verbatim rather than restating it. "fix(#385/#402) an all-particle name part initials its words (R2)": - _Claim(27, ('_initials',), "6b242c287db8", ('DEFAULT',)), + _Claim(29, ('_initials',), "541df583f5fc", ('DEFAULT',)), "fix(#360) los joined the particles, so it no longer initials": _Claim(1, ('_initials',), "cd721215f463", ('DEFAULT',)), # fix(#462) reaches more than it explains -- the reach is the @@ -3164,9 +3367,12 @@ def _claim(rule: dict) -> _Claim: # move. "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), + # 2026-09-18 (#289/#516): 16 -> 17. One new corpus name, + # 'JOHN PROF. MA', added to the alternation deliberately -- + # the ledger comment argues it. No role joined the five. "fix(#316) a trailing period-marked title word reads as a title": - _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), - "562e0e82a22b", None), + _Claim(17, ('family', 'given', 'middle', 'suffix', 'title'), + "0dc3551e57da", None), # The CJK member of the same argument, its own literal rule # (an UNDECLARED alternation holding a script-classified # member belongs to the honorific pin above -- see the 1.4.0 @@ -3204,6 +3410,39 @@ def _claim(rule: dict) -> _Claim: "ec00806a06f0", ('DEFAULT',)), "fix(#383/#479) a marked connective letter in a one-case name is reported": _Claim(6, ('_ambiguities',), "03de2830707e", ('DEFAULT',)), + # #289's alternation, appended with #516's and the report + # rule's below it. Literal-anchored to the movers a written + # case contrast decides, `orders` DEFAULT: the class is a + # SHAPE and not a wordlist, so a member spelled as the class + # would claim the one-case spellings that do not move. + # _MUST_NOT_MATCH carries those, and at a reach this small it + # is the wall this row cannot be. + # 2026-09-18, second round: two more names, 'John Smith, Ma' + # and 'Smith Jr., Ma', the Title-case halves of two minimal + # pairs rules.md#C1 now states. No role joined the list. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + # 2026-09-18, review round: 'John de Ma' joins, the + # one-particle spelling of 'John van der Berg Ma'. Its diff + # is the restored chain report at the 2.x baselines and the + # role move at 1.4.0 and 2.2/2.3; no role joined the list. + _Claim(20, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "fe5f68146005", ('DEFAULT',)), + # #516's alternation. Literal-anchored to the by-shape movers, + # `orders` DEFAULT. Same reasoning as the rule above: the + # class is a shape the vocabulary does not spell, so the + # probes and not this number are what bound it. + "fix(#516) an unlisted dotted acronym is read by position": + _Claim(9, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "9bbaf4e84dc0", ('DEFAULT',)), + # The report-only rule. `_ambiguities` alone, so a widening that + # took a ROLE would change the roles here before it reached + # the gate -- which is the one thing this row can say about a + # rule whose whole subject is a report. + "fix(#289/#516) the ambiguous credential class reports at slots that were silent": + _Claim(8, ('_ambiguities',), "4f4844f54ba1", ('DEFAULT',)), + # The 2026-09-18 review round's honorific-peel rule: three + # composed CJK forms, radar tier, classified because the arc + # intended the move (the ledger comment carries the argument). + "fix(#289/#516) the glued CJK honorific peel reads the case lean": + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "4fb6dbdbccd4", ('DEFAULT',)), }, # The 2.3 cycle's first rule, and a facade-only render fix: every # role is identical, so `_initials` alone. Reach and digest as in @@ -3313,9 +3552,12 @@ def _claim(rule: dict) -> _Claim: # move. "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), + # 2026-09-18 (#289/#516): 16 -> 17. One new corpus name, + # 'JOHN PROF. MA', added to the alternation deliberately -- + # the ledger comment argues it. No role joined the five. "fix(#316) a trailing period-marked title word reads as a title": - _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), - "562e0e82a22b", None), + _Claim(17, ('family', 'given', 'middle', 'suffix', 'title'), + "0dc3551e57da", None), # The CJK member of the same argument, its own literal rule # (an UNDECLARED alternation holding a script-classified # member belongs to the honorific pin above -- see the 1.4.0 @@ -3356,6 +3598,39 @@ def _claim(rule: dict) -> _Claim: "ec00806a06f0", ('DEFAULT',)), "fix(#383/#479) a marked connective letter in a one-case name is reported": _Claim(6, ('_ambiguities',), "03de2830707e", ('DEFAULT',)), + # #289's alternation, appended with #516's and the report + # rule's below it. Literal-anchored to the movers a written + # case contrast decides, `orders` DEFAULT: the class is a + # SHAPE and not a wordlist, so a member spelled as the class + # would claim the one-case spellings that do not move. + # _MUST_NOT_MATCH carries those, and at a reach this small it + # is the wall this row cannot be. + # 2026-09-18, second round: two more names, 'John Smith, Ma' + # and 'Smith Jr., Ma', the Title-case halves of two minimal + # pairs rules.md#C1 now states. No role joined the list. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + # 2026-09-18, review round: 'John de Ma' joins, the + # one-particle spelling of 'John van der Berg Ma'. Its diff + # is the restored chain report at the 2.x baselines and the + # role move at 1.4.0 and 2.2/2.3; no role joined the list. + _Claim(20, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "fe5f68146005", ('DEFAULT',)), + # #516's alternation. Literal-anchored to the by-shape movers, + # `orders` DEFAULT. Same reasoning as the rule above: the + # class is a shape the vocabulary does not spell, so the + # probes and not this number are what bound it. + "fix(#516) an unlisted dotted acronym is read by position": + _Claim(9, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "9bbaf4e84dc0", ('DEFAULT',)), + # The report-only rule. `_ambiguities` alone, so a widening that + # took a ROLE would change the roles here before it reached + # the gate -- which is the one thing this row can say about a + # rule whose whole subject is a report. + "fix(#289/#516) the ambiguous credential class reports at slots that were silent": + _Claim(9, ('_ambiguities',), "f89a7428e317", ('DEFAULT',)), + # The 2026-09-18 review round's honorific-peel rule: three + # composed CJK forms, radar tier, classified because the arc + # intended the move (the ledger comment carries the argument). + "fix(#289/#516) the glued CJK honorific peel reads the case lean": + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "4fb6dbdbccd4", ('DEFAULT',)), }, "expected_since_2.1.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -3497,12 +3772,21 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "9523e518e6ec", None), "fix(#421) the bound-given join never absorbs a split credential": _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef", None), + # 2026-09-18 (#289): the reach is unchanged at two names and the + # ROLES moved -- `given` out, `middle` in. 'abdul Smith Ma' + # stops diffing at this baseline altogether and + # 'abdul Smith Jr Ma' now reads middle 'Jr'; the ledger's + # own dated paragraph carries the argument. "fix(#425) the bound-given reserve runs assign's peel over the joined view": - _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e", None), + _Claim(2, ('family', 'middle', 'suffix'), "ef1ab03b617e", None), "fix(#424) the particle chain stops before the trailing numeral": _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf", None), + # 2026-09-18 (#289): 1 -> 2. 'john van der berg ma' entered + # the corpus as rules.md#P2's one-case contrast, and it is the + # spelling this rule still describes -- see the ledger's own + # dated paragraph, which replaced a `dormant` key with it. "fix(#424) the particle chain stops before a bare acronym with words to spare": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b", None), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "18fbd95cd89e", None), "fix(#424) a title-led chain before the numeral is the one name piece": _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": @@ -3510,7 +3794,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), + _Claim(5, ('_ambiguities', 'given', 'middle', 'title'), "e5aa72ebb9bf", None), # `middle` left the ROLES in the same edit, and the reach # grew with the rules corpus; the 1.4.0 roster above carries # both, and says the same at the other two baselines. @@ -3518,8 +3802,12 @@ def _claim(rule: dict) -> _Claim: _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), + # 2026-09-18: 18 -> 20. Two new corpus names, 'Smith, MA' and + # 'Smith, Ma' -- the pair #289's lean reads in opposite + # directions, both of them lone post-comma credentials + # by this rule's own description. "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('suffix', 'title'), "1f79efa10444", None), + _Claim(20, ('suffix', 'title'), "c94afd0c7617", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": @@ -3530,8 +3818,11 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "faa2c70fc49e", None), "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65", ('DEFAULT', 'FAMILY_FIRST')), + # 2026-09-18: 1 -> 2. One new corpus name, 'Jack MA.', the + # all-caps spelling #289's case contrast now reads as a + # credential; the rule's own 'Jack Ma.' is untouched. "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4", None), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "29b08f75e6ff", None), "fix(#367) an inferred title no longer displaces a leading particle either": _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": @@ -3567,7 +3858,7 @@ def _claim(rule: dict) -> _Claim: # baseline, and the reaches and digests agree because the two # files carry the same literal list. "fix(#385/#402) an all-particle name part initials its words (R2)": - _Claim(27, ('_initials',), "6b242c287db8", ('DEFAULT',)), + _Claim(29, ('_initials',), "541df583f5fc", ('DEFAULT',)), "fix(#360) los joined the particles, so it no longer initials": _Claim(1, ('_initials',), "cd721215f463", ('DEFAULT',)), # fix(#462), reach and digest as in the 2.0.0 mapping: the same @@ -3593,9 +3884,12 @@ def _claim(rule: dict) -> _Claim: # move. "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), + # 2026-09-18 (#289/#516): 16 -> 17. One new corpus name, + # 'JOHN PROF. MA', added to the alternation deliberately -- + # the ledger comment argues it. No role joined the five. "fix(#316) a trailing period-marked title word reads as a title": - _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), - "562e0e82a22b", None), + _Claim(17, ('family', 'given', 'middle', 'suffix', 'title'), + "0dc3551e57da", None), # The CJK member of the same argument, its own literal rule # (an UNDECLARED alternation holding a script-classified # member belongs to the honorific pin above -- see the 1.4.0 @@ -3630,6 +3924,39 @@ def _claim(rule: dict) -> _Claim: "ec00806a06f0", ('DEFAULT',)), "fix(#383/#479) a marked connective letter in a one-case name is reported": _Claim(6, ('_ambiguities',), "03de2830707e", ('DEFAULT',)), + # #289's alternation, appended with #516's and the report + # rule's below it. Literal-anchored to the movers a written + # case contrast decides, `orders` DEFAULT: the class is a + # SHAPE and not a wordlist, so a member spelled as the class + # would claim the one-case spellings that do not move. + # _MUST_NOT_MATCH carries those, and at a reach this small it + # is the wall this row cannot be. + # 2026-09-18, second round: two more names, 'John Smith, Ma' + # and 'Smith Jr., Ma', the Title-case halves of two minimal + # pairs rules.md#C1 now states. No role joined the list. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + # 2026-09-18, review round: 'John de Ma' joins, the + # one-particle spelling of 'John van der Berg Ma'. Its diff + # is the restored chain report at the 2.x baselines and the + # role move at 1.4.0 and 2.2/2.3; no role joined the list. + _Claim(20, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "fe5f68146005", ('DEFAULT',)), + # #516's alternation. Literal-anchored to the by-shape movers, + # `orders` DEFAULT. Same reasoning as the rule above: the + # class is a shape the vocabulary does not spell, so the + # probes and not this number are what bound it. + "fix(#516) an unlisted dotted acronym is read by position": + _Claim(9, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "9bbaf4e84dc0", ('DEFAULT',)), + # The report-only rule. `_ambiguities` alone, so a widening that + # took a ROLE would change the roles here before it reached + # the gate -- which is the one thing this row can say about a + # rule whose whole subject is a report. + "fix(#289/#516) the ambiguous credential class reports at slots that were silent": + _Claim(8, ('_ambiguities',), "4f4844f54ba1", ('DEFAULT',)), + # The 2026-09-18 review round's honorific-peel rule: three + # composed CJK forms, radar tier, classified because the arc + # intended the move (the ledger comment carries the argument). + "fix(#289/#516) the glued CJK honorific peel reads the case lean": + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "4fb6dbdbccd4", ('DEFAULT',)), }, "expected_since_2.3.0.toml": { # #383/#479's three rules, the first this ledger carries. The @@ -3654,6 +3981,39 @@ def _claim(rule: dict) -> _Claim: _Claim(7, ('_ambiguities',), "2eb6eff33836", ('DEFAULT',)), "fix(#383/#479) a bare capital connective in an all-upper name stops initialing": _Claim(1, ('_initials',), "7ff29af96914", ('DEFAULT',)), + # #289's alternation, appended with #516's and the report + # rule's below it. Literal-anchored to the movers a written + # case contrast decides, `orders` DEFAULT: the class is a + # SHAPE and not a wordlist, so a member spelled as the class + # would claim the one-case spellings that do not move. + # _MUST_NOT_MATCH carries those, and at a reach this small it + # is the wall this row cannot be. + # 2026-09-18, second round: two more names, 'John Smith, Ma' + # and 'Smith Jr., Ma', the Title-case halves of two minimal + # pairs rules.md#C1 now states. No role joined the list. + "fix(#289) a written case contrast decides a bare ambiguous acronym": + # 2026-09-18, review round: 'John de Ma' joins, the + # one-particle spelling of 'John van der Berg Ma'. Its diff + # is the restored chain report at the 2.x baselines and the + # role move at 1.4.0 and 2.2/2.3; no role joined the list. + _Claim(20, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "fe5f68146005", ('DEFAULT',)), + # #516's alternation. Literal-anchored to the by-shape movers, + # `orders` DEFAULT. Same reasoning as the rule above: the + # class is a shape the vocabulary does not spell, so the + # probes and not this number are what bound it. + "fix(#516) an unlisted dotted acronym is read by position": + _Claim(9, ('_ambiguities', 'family', 'given', 'middle', 'suffix'), "9bbaf4e84dc0", ('DEFAULT',)), + # The report-only rule. `_ambiguities` alone, so a widening that + # took a ROLE would change the roles here before it reached + # the gate -- which is the one thing this row can say about a + # rule whose whole subject is a report. + "fix(#289/#516) the ambiguous credential class reports at slots that were silent": + _Claim(9, ('_ambiguities',), "f89a7428e317", ('DEFAULT',)), + # The 2026-09-18 review round's honorific-peel rule: three + # composed CJK forms, radar tier, classified because the arc + # intended the move (the ledger comment carries the argument). + "fix(#289/#516) the glued CJK honorific peel reads the case lean": + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "4fb6dbdbccd4", ('DEFAULT',)), }, } @@ -3822,6 +4182,28 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: "Carod i": "fix(#397) NOT WANTED: a trailing Catalan/Polish linking " "'i' is read as a generation marker and the family is lost", + # #289/#516's one. '田中 太郎, MA' is this arc's own case row + # (radar tier): two whitespace tokens before the comma make it + # two NAME words by rules.md#C1's count, so the suffix reading + # is taken and the Japanese surname split then runs over the + # pre-comma part. The native-script order rule WINS it -- what + # moves `family` and `given` here is the family-first ordering + # of '田中 太郎', which the comma-compound rule describes only + # half of -- and it wins by file order alone, the two rules' + # `fields` overlapping without nesting. + "田中 太郎, MA": + "fix(#271/#272/#298) native-script CJK: family-first " + "order, hangul segmentation, the kana license and the dots", + # #289's one contract-tier contest here. The lone-post-comma + # routing rule WINS 'Smith, MA', and it is the right winner at + # THIS baseline: what the diff shows against v1 is a lone piece + # after the comma routing to `suffix` instead of to `first`, + # which is that rule's own sentence. The caps lean is why the + # sentence became true for this name; it is not what the + # baseline-relative diff is about. + "Smith, MA": + "fix(comma-family) lone post-comma piece routes to " + "suffix/title, not first", "田中さん II": "fix(cjk-glued-honorific-peel) glued honorific peels into " "suffix", @@ -5034,7 +5416,13 @@ class _Excluded(NamedTuple): # every baseline (rules.md#S1's escape unwraps it), so the # #322/#323 rule carries the name's diff and this exclusion # silences none of it. - _Excluded(58, "57618fbebc6b", ()), + # 58 -> 59 on 2026-09-18 for 'Bridge (A.B)', added as the + # #516 review round's control for classify's `token.role is + # None` guard -- its bracketed clause matches this shape and + # costs the entry nothing: 1.4.0 reads the parenthesized pair + # as a nickname exactly as the tree does, so there is no diff + # to silence. + _Excluded(59, "572f5b8f850a", ()), } @@ -5270,8 +5658,15 @@ def test_a_rule_reaching_no_corpus_name_says_why_it_is_kept() -> None: "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 1), ("fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set", "feat(#273) typographic nickname delimiters recognized by default", 1), + # 20 → 23 on 2026-09-18: the review round's three + # `tolerated=True` rows for the honorific peel behind a + # leaning credential ('Kim김민준씨, MA', 'Jo김민준씨, DO', + # 'Dr. 김민준씨, MA') put three more corpus names inside BOTH + # regexes. The pair is unchanged and no rule was widened -- + # this counter is the shared corpus REACH, so it moves with + # every corpus addition that lands in the overlap. ("fix(cjk-comma-compound) comma routing compounds with the CJK order flip", - "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 20), + "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 23), ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`", 1), ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 6835775c..3cb2dd06 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -202,10 +202,15 @@ def test_ambiguous_acronym_reports_the_reading_it_took() -> None: [AmbiguityKind.SUFFIX_OR_NAME] assert [t.text for t in took_suffix.ambiguities[0].tokens] == ["MA"] + # MOVED by #289, not deleted: 'MA' is written in capitals inside a + # mixed-case name, so it now leans CREDENTIAL and is taken with no + # words to spare -- 'Jack' is the only name word left, which is + # what also turns on the GIVEN_OR_FAMILY fork (decisions.md#S2). took_family = parse("Jack MA") - assert took_family.family == "MA" - assert [a.kind for a in took_family.ambiguities] == \ - [AmbiguityKind.SUFFIX_OR_NAME] + assert took_family.suffix == "MA" + assert not took_family.family + assert set(a.kind for a in took_family.ambiguities) == \ + {AmbiguityKind.SUFFIX_OR_NAME, AmbiguityKind.GIVEN_OR_FAMILY} @pytest.mark.parametrize("text", [ @@ -249,9 +254,16 @@ def test_ambiguous_acronym_detail_names_the_role_it_got() -> None: # the unpeeled piece is the last NAME piece, which is the family # name only under GIVEN_FIRST -- FAMILY_FIRST puts it in given, so # the detail has to follow the role actually assigned + # + # MOVED by #289, not deleted: 'Jack MA' is mixed case now, so 'MA' + # leans CREDENTIAL and is peeled as a suffix under every order -- + # there is no unpeeled acronym left for the detail to name a role + # for. 'JACK MA' (one case, no lean) is the input that still + # exercises the mechanism this test pins: the count alone decides, + # and the detail names whichever role the order actually gave it. fam_first = Parser(policy=Policy(name_order=FAMILY_FIRST)) - n = fam_first.parse("Jack MA") - assert (n.family, n.given) == ("Jack", "MA") + n = fam_first.parse("JACK MA") + assert (n.family, n.given) == ("JACK", "MA") assert "given name" in n.ambiguities[0].detail assert "family name" not in n.ambiguities[0].detail @@ -561,14 +573,22 @@ def test_the_reserve_spares_the_family_the_acronym_fork_would_take() -> None: n, m = parse(bound), parse(plain) assert (n.family, n.suffix) == (m.family, m.suffix) assert n.family != "" + # MOVED by #289, not deleted: 'Ed' is Title-case in a mixed-case + # name, so it now leans SURNAME and the peel declines it with + # words to spare -- the walk stops at the declined pick, 'Jr' + # never reached behind it, and both join as name words + # (decisions.md#S2's accepted cost, the 'abdul Smith Jr Ma' shape). n = parse("abu Bakar Jr Ed") - # spaced run, spaced render (#436); the family claim is what moves here - assert (n.family, n.suffix) == ("Bakar", "Jr Ed") + assert (n.family, n.suffix) == ("Ed", "") # and the join never turns a suffix into a name: unjoined, the # acronym is a credential with words to spare, so 'abdul Smith # Ma' reads as 'John Smith Ma' does (1.4.0 parity restored) + # + # MOVED by #289, not deleted: 'Ma' is Title-case in a mixed-case + # name, so it leans SURNAME on both sides of the join and stays a + # name word instead of a credential (decisions.md#S2). n, m = parse("abdul Smith Ma"), parse("John Smith Ma") - assert (n.family, n.suffix) == (m.family, m.suffix) == ("Smith", "Ma") + assert (n.family, n.suffix) == (m.family, m.suffix) == ("Ma", "") def test_a_joined_pair_is_never_peeled_as_a_title() -> None: @@ -607,7 +627,11 @@ def test_the_chain_and_the_walk_stop_where_the_peel_begins() -> None: ("John van der Berg V", "van der Berg", "V"), ("John van der Berg X", "van der Berg", "X"), ("abdul van der Berg V", "van der Berg", "V"), - ("John van der Berg Ma", "van der Berg", "Ma")): + # MOVED by #289, not deleted: 'Ma' is Title-case in a + # mixed-case name, so it now leans SURNAME and the chain's + # re-ask absorbs it into the particle run instead of + # leaving it for assign to peel (decisions.md#S2). + ("John van der Berg Ma", "van der Berg Ma", "")): n = parse(text) assert (n.family, n.suffix) == (family, suffix), text n = parse("John née Jones Smith V") @@ -632,8 +656,15 @@ def test_the_chain_and_the_walk_stop_where_the_peel_begins() -> None: # behind a title-and-particle word the chain takes the name's first # word (#367), and the acronym it leaves has no words to spare for # assign: the chain keeps it rather than leave it as the family + # + # MOVED by #289, ACCEPTED COST (decisions.md#S2): 'MA' is written + # in capitals inside a mixed-case name, so it now leans CREDENTIAL + # and the caps lean makes it reachable where the count alone was + # not -- the lean ends the chain's re-ask with no words to spare, + # where the count would not have. This was listed as staying + # fix(#424)'s and measured not to: 1.4.0 read "von Berg", "MA". n = parse("Freiherr von Berg MA") - assert (n.title, n.family, n.suffix) == ("Freiherr", "von Berg MA", "") + assert (n.title, n.family, n.suffix) == ("Freiherr", "von Berg", "MA") # the numeral keeps its three pieces behind the same word, and the # chain, now the one name piece, reads as 'Dr. Smith V' reads n = parse("Freiherr von Richthofen V") diff --git a/tests/v2/test_policy.py b/tests/v2/test_policy.py index a0c8ec65..b1ca4c6c 100644 --- a/tests/v2/test_policy.py +++ b/tests/v2/test_policy.py @@ -6,7 +6,7 @@ from nameparser._policy import ( DEFAULT_SCRIPT_ORDERS, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, PolicyPatch, Script, UNSET, - _SCRIPT_RANGES, _script_matcher, apply_patch, + _BOOL_FIELDS, _SCRIPT_RANGES, _script_matcher, apply_patch, ) from nameparser._types import Role @@ -288,12 +288,52 @@ def test_unset_fields_are_distinguishable_from_defaults() -> None: def test_policy_rejects_non_bool_flags() -> None: # "no" and "false" are truthy: storing them would silently invert # the caller's intent downstream. - for flag in ("middle_as_family", "lenient_comma_suffixes", - "strip_emoji", "strip_bidi"): + # + # Swept over the DATACLASS, not over a list written here. The list + # this replaced named 2.3's four flags and was never extended when + # 2.4 added `unlisted_dotted_suffixes` and `unlisted_caps_suffixes`, + # so both shipped with no coverage for the check the library was + # already making -- the drift AGENTS.md's guard-the-whole-family + # rule is about, in the test rather than in the guard. + assert set(_BOOL_FIELDS) == { + f.name for f in dataclasses.fields(Policy) + if isinstance(getattr(Policy(), f.name), bool)} + for flag in _BOOL_FIELDS: with pytest.raises(TypeError, match="must be a bool"): Policy(**{flag: "no"}) # type: ignore[arg-type] +#: 2.3's Policy fields, in 2.3's order, read off the released tree with +#: `git show 1f78bef:nameparser/_policy.py`. A dated snapshot of an +#: immutable commit, so it cannot go stale (AGENTS.md's counting +#: claims). +_FIELDS_AT_2_3 = ( + "name_order", "script_orders", "segment_scripts", "patronymic_rules", + "middle_as_family", "nickname_delimiters", "maiden_delimiters", + "extra_suffix_delimiters", "lenient_comma_suffixes", "strip_emoji", + "strip_bidi", +) + + +def test_the_2_3_positional_fields_did_not_move() -> None: + # Policy and PolicyPatch are not `kw_only`, so a field's POSITION + # is API: `Policy(GIVEN_FIRST, ..., True, False)` binds by + # position, and inserting a field mid-class silently re-binds every + # argument after it. 2.4's two switches were first written beside + # `lenient_comma_suffixes`, which moved `strip_emoji` and + # `strip_bidi` two places to the right; they are appended now, and + # this is what holds them there. + for cls in (Policy, PolicyPatch): + names = tuple(f.name for f in dataclasses.fields(cls)) + assert names[:len(_FIELDS_AT_2_3)] == _FIELDS_AT_2_3, cls.__name__ + assert names[len(_FIELDS_AT_2_3):] == ( + "unlisted_dotted_suffixes", "unlisted_caps_suffixes"), cls.__name__ + # the positional binding itself, not just the names + assert Policy(GIVEN_FIRST, (), frozenset(), frozenset(), False, + frozenset(), frozenset(), frozenset(), True, + False).strip_emoji is False + + def test_patronymic_rules_generator_errors_propagate_untouched() -> None: # A ValueError raised inside the caller's own generator must not be # rewritten as "unknown patronymic rule" with the traceback erased. @@ -739,3 +779,101 @@ def test_policy_patch_one_shot_bad_tail_defers_without_silent_drop() -> None: with pytest.raises(TypeError, match=r"script_orders entries must be .* got 5"): apply_patch(Policy(), patch) + + +def test_unlisted_dotted_suffixes_is_a_validated_bool_defaulting_on() -> None: + # #516's dotted half is a switch, and it is ON: a token of two or + # more period-separated chunks that no vocabulary claims reads by + # position, which is what the periods are for. + assert Policy().unlisted_dotted_suffixes is True + assert Policy(unlisted_dotted_suffixes=False).unlisted_dotted_suffixes \ + is False + with pytest.raises(TypeError, match="unlisted_dotted_suffixes"): + Policy(unlisted_dotted_suffixes="no") # type: ignore[arg-type] + # the patch mirrors it as a SCALAR, override not merge + assert Policy().patched( + PolicyPatch(unlisted_dotted_suffixes=False) + ).unlisted_dotted_suffixes is False + + +def test_unlisted_dotted_suffixes_off_reads_name_material() -> None: + # The switch's whole behavior, both directions, on one name -- + # "name material" for a token no chunk claims; the roman-chunk + # retirement (rules.md#S3) and real chunk-level vocabulary + # ('Msc.Ed.', 'JD.CPA') are not behind this switch either way. + from nameparser import Parser + + on = Parser().parse("John Smith X.Y.Z.") + off = Parser(policy=Policy(unlisted_dotted_suffixes=False)).parse( + "John Smith X.Y.Z.") + assert (on.family, on.suffix) == ("Smith", "X.Y.Z.") + assert (off.middle, off.family, off.suffix) == ("Smith", "X.Y.Z.", "") + # OFF still reports: the parser chose the name reading over a + # credential one, and that is the fork (#516) + assert [a.kind.value for a in off.ambiguities] == ["suffix-or-name"] + # and the vocabulary is untouched either way + for p in (Parser(), Parser(policy=Policy(unlisted_dotted_suffixes=False))): + assert p.parse("John Smith M.A.").suffix == "M.A." + assert p.parse("Doe, John Msc.Ed.").suffix == "Msc.Ed." + + +def test_unlisted_caps_suffixes_is_a_validated_bool_defaulting_off() -> None: + # #516's all-caps half is OPT-IN, and the asymmetry with the + # dotted half is the whole decision: an all-caps surname is a real + # writing convention that shape cannot separate from a credential. + assert Policy().unlisted_caps_suffixes is False + assert Policy(unlisted_caps_suffixes=True).unlisted_caps_suffixes is True + with pytest.raises(TypeError, match="unlisted_caps_suffixes"): + Policy(unlisted_caps_suffixes="yes") # type: ignore[arg-type] + assert Policy().patched( + PolicyPatch(unlisted_caps_suffixes=True) + ).unlisted_caps_suffixes is True + + +def test_unlisted_caps_suffixes_on_reads_an_all_caps_word() -> None: + # Frames, recorded honestly (#516 review round, F4, corrected + # 2026-09-18): the switch is OPT-IN and OFF-BAND -- + # `tools/perf/call_count.py`'s reference name and the + # DEFAULT-policy comma harness both measure +0, the only figures + # test_benchmark.py's band gates. With the switch ON, a genuine + # comma candidate pays for the fact it forces: re-measured + # 2026-09-18 on this tree, same-interpreter harness (the resolved + # `sys.executable` used on both sides, `Parser().parse` and + # `Parser(policy=Policy(unlisted_caps_suffixes=True)).parse`, mean + # of 50 after one warm-up parse) -- `"Smith, John"` is +7 + # (206 -> 213), `"Smith, XYZ"` (a real candidate) is +40 + # (205 -> 245). An earlier round's comment here read +6/+39 + # against baselines 207/206, which do not reproduce (this tree's + # own default reading of the two names is 206/205, matching + # f7089763 exactly); the earlier round's post-consolidation + # measurement itself was one frame off on each name, not the + # consolidation's own effect, which still stands (the prior, + # three-copies-of-the-predicate tree, cbd87a7d, reads 215 and 256 + # with the switch on -- +9/+51 against the same 206/205 baseline, + # so the consolidation saved 2 and 11). Nothing gates either number; + # they are reported here, dated, so a reader who turns the switch + # on knows what it costs and a later re-measurement does not read + # as a silent drift. + from nameparser import Parser + + on = Parser(policy=Policy(unlisted_caps_suffixes=True)) + off = Parser() + # what it buys + assert on.parse("John Smith XYZ").suffix == "XYZ" + assert on.parse("John Smith, XYZ").suffix == "XYZ" + # what it costs, and why the default is off + assert on.parse("Jean Pierre DUPONT").suffix == "DUPONT" + assert off.parse("Jean Pierre DUPONT").family == "DUPONT" + # the default emits nothing at all + assert off.parse("John Smith XYZ").ambiguities == () + # the boundaries: one case, one letter, a digit, and vocabulary. + # 'John Smith X' is NOT the single-letter control -- 'X' is a bare + # roman numeral (rules.md#S2's numeral fork) and reads as suffix + # 'X' with the switch either way, unrelated to this one; 'Z' is + # not vocabulary at all and stays unaffected by the switch, which + # is the actual boundary (a single capital never satisfies the + # `len(text) >= 2` half of the shape test, on or off). + assert on.parse("JOHN SMITH XYZ").family == "XYZ" + assert on.parse("John Smith Z").family == off.parse("John Smith Z").family + assert on.parse("John Smith XY2").family == "XY2" + assert on.parse("John Smith MC").suffix == "MC" diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index cc42bfa2..2721fa29 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -111,6 +111,12 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: # source now fails here instead of being silently unpinned. _SOURCES: dict[tuple[str, str], str | None] = { ("_pieces", "_PERIOD_ABBREV"): "period_abbreviation", + # SAME object as the entry above, not a second copy: _pieces + # imports it from _vocab (#289/#516, quality-review finding -- + # is_title_shaped moved there so name_word_count could share + # is_leading_title's H2 shape test), so it is visible under both + # module names and the completeness scan below sees it twice. + ("_vocab", "_PERIOD_ABBREV"): "period_abbreviation", ("_group", "_D"): None, ("_vocab", "_DOTTED"): None, ("_group", "_PH"): None, diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 5f2bce63..be19aff7 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1820,6 +1820,25 @@ class _ShapeMismatch(NamedTuple): # measured. The winners are pinned in _CROSS_RULE_WINNERS. "john e smith": ("_initials",), "john e jones": ("_initials",), + # #289/#516's one, adjudicated 2026-09-18, below both cohorts + # above for the same provenance reason. '田中 太郎, MA' entered + # the corpus with this arc's own case row (radar tier, a + # composed CJK-plus-Latin comma form the writing system does + # not produce), and two rules admit its {family, given} diff: + # the compound CJK rule and the native-script order rule. Their + # `fields` OVERLAP without nesting, so neither is the narrower + # and `precedes_narrower` has nothing to say; file order is the + # whole decision. The shape is this run's, not guessed. + "田中 太郎, MA": ("family", "given"), + # #289's one contract-tier contest at this baseline. 'Smith, + # MA' moves {given, suffix} -- v1 read given 'MA', family + # 'Smith', and the caps lean now reads family 'Smith', suffix + # 'MA' -- and two rules admit it: the lone-post-comma routing + # rule, whose Latin comma regex reaches every such name, and + # this arc's own. `fields` overlap on {given, suffix} without + # nesting, so neither is the narrower and file order is the + # whole decision. The shape is this run's, not guessed. + "Smith, MA": ("given", "suffix"), }, # #501's six, moved here from _WATCHED_DIFFS with their shapes # unchanged. The four CJK rows sit at 2.0.0 alone: the honorific @@ -1953,6 +1972,21 @@ class _ShapeMismatch(NamedTuple): #: of the three baselines it diffs at since #501 pinned its 2.x pair. #: That is why the population is 48 names where the tests/-only scan #: says 50. +#: AMENDED 2026-09-18 (#289/#516): the parenthetical above is dated +#: 2026-09-05 and stays as written, but its live half has stopped +#: being true. `radar unclassified` is now 0 / 5 / 6 / 7 / 7 at +#: 1.4.0 / 2.0.0 / 2.1.0 / 2.2.0 / 2.3.0, so the two sets no longer +#: coincide: a diffing radar name can be watched by no rule at all. +#: The seven at 2.3.0 are this arc's own unclassified SUFFIX_OR_NAME +#: reports -- 'Smith, E.T., Jr.', 'Smith, J.R.', 'Smith, A.P.', +#: 'MD, DO, DDS', '毛泽东, MA', '田中 太郎, MA', '마틴 킹, MA' -- each +#: a fork the ambiguous credential class now calls out loud at a slot +#: that was silent, on a name the contract does not answer for. The +#: POPULATION CLAUSE itself still holds unchanged, because it asks +#: for a name EXPLAINED by a ledger rule and these are explained by +#: none: an unclassified radar diff is already printed by every run, +#: which is the weak-watcher case the clause was written to exclude. +#: What is retired is only the claim that the two sets coincide. #: Recounted 2026-09-07 with #342, which moved three names across the #: literal clause at once: 'Aishwarya Rai' gained a case row and left #: the population, while 'Lala Lajpat Rai' and 'John Smith, RAI' are @@ -2123,7 +2157,12 @@ class _ShapeMismatch(NamedTuple): "Jong, van der": ("_initials",), "Jose E. Maria Santos": ("_initials",), "Lala Lajpat Rai": ("family", "middle", "suffix"), - "MD, DO, DDS": ("given", "title"), + # 2026-09-18 (#289): the shape GREW by `_ambiguities`. A + # bare listed member in the post-comma slot now reports + # `suffix-or-name` wherever the fork is consulted, and 'DO' + # here is one; the roles are untouched. Re-recorded in the + # commit that moved it, per this dict's own rule. + "MD, DO, DDS": ("_ambiguities", "given", "title"), "Mesnil Garcia van": ("_initials",), "Ph. D., Jr.": ("family", "suffix", "title"), "Sander van": ("_initials",), @@ -2164,7 +2203,12 @@ class _ShapeMismatch(NamedTuple): "Jong, van der": ("_initials",), "Jose E. Maria Santos": ("_initials",), "Lala Lajpat Rai": ("family", "middle", "suffix"), - "MD, DO, DDS": ("given", "title"), + # 2026-09-18 (#289): the shape GREW by `_ambiguities`. A + # bare listed member in the post-comma slot now reports + # `suffix-or-name` wherever the fork is consulted, and 'DO' + # here is one; the roles are untouched. Re-recorded in the + # commit that moved it, per this dict's own rule. + "MD, DO, DDS": ("_ambiguities", "given", "title"), "Mesnil Garcia van": ("_initials",), "Ph. D., Jr.": ("family", "suffix", "title"), "Sander van": ("_initials",), diff --git a/tools/differential/corpus_cjk_tolerated.jsonl b/tools/differential/corpus_cjk_tolerated.jsonl index 8feebee3..cb31023e 100644 --- a/tools/differential/corpus_cjk_tolerated.jsonl +++ b/tools/differential/corpus_cjk_tolerated.jsonl @@ -2,10 +2,18 @@ "Dr 김민준, Jr." "Dr 김민준씨, Jr." "Dr 김민준씨, V." +"Dr. 김민준씨, MA" +"J.씨" +"John Smith 田.中." +"Jo김민준씨, DO" +"Kim김민준씨, MA" "マイケル." "威廉·莎士比亚, PhD" "毛 泽东 Dr." +"毛泽东 MA" +"毛泽东, MA" "王先生, V." +"田中 太郎, MA" "田中, 太郎さん" "田中." "田中. 太郎" @@ -39,6 +47,7 @@ "김민준씨." "김민준씨., J.씨" "남궁민수, 지훈" +"마틴 킹, MA" "선생님, J.씨" "양 지훈." "양. 지훈" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b46c21ce..5a4757f4 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -17,6 +17,7 @@ "Andrew Perkins (MBA)" "Anh Do" "Anh Van Do" +"Anh van Do" "Anna () Smith" "Anna z (domu) Nowak" "Anna z Nowak" @@ -30,7 +31,11 @@ "Berg, abdul V" "Berg, abdul van" "Berg, abdul vd" +"Davis Royce, Ed" "Del Toro" +"Doe, John MA" +"Doe, John Msc.Ed." +"Doe, John X.Y.Z." "Dr Jr" "Dr King Jr" "Dr." @@ -61,9 +66,15 @@ "J. Smith" "J. née Jones Smith V" "J.R. Smith" +"JACK MA" +"JOHN PROF. MA" +"JOHN SMITH MA" "JUAN GARCIA Y LOPEZ" +"Jack MA" +"Jack Ma" "Jack Ma." "Jack Wei Ma" +"Jack X.Y.I." "Jane (née Jones) Smith" "Jane Smith (Nee)" "Jane Smith (Nee) (Jones)" @@ -85,29 +96,43 @@ "Jane van der Berg née y Jones" "Jane „JD Smith" "Jean 'JD' Smith" +"Jean DUPONT" +"Jean Pierre DUPONT" "John . Smith" "John Doctor Smith" "John Ma" "John Prof. MA" "John Smith" +"John Smith 1.4" "John Smith Esq." "John Smith J.u.n.i.o.r." "John Smith Jr." "John Smith Jr. Prof." "John Smith M.A." "John Smith MD PhD" +"John Smith Ma" "John Smith Mc V" +"John Smith Msc.Ed." "John Smith PhD" "John Smith Prof." "John Smith Prof. Dr." "John Smith Prof. Jr." "John Smith Q.W.E.R.T." "John Smith Sir" +"John Smith X.Y.Z." +"John Smith XYZ" "John Smith Xyz." +"John Smith, A.B." +"John Smith, Ed" "John Smith, Jones" "John Smith, LEED AP" +"John Smith, MA" "John Smith, MD, Bart" +"John Smith, MD, Ma" +"John Smith, MD, R.A.I." +"John Smith, MD, XYZ" "John Smith, MD,, Jr." +"John Smith, Ma" "John Smith, Mr." "John Smith, Mr. Jr." "John Smith, PhD" @@ -174,7 +199,9 @@ "QC MP" "Rev. John Smith" "Rinpoche" +"Royce, Ed" "SHIRLEY MACLAINE" +"STEVEN HARDMAN, MD, DO, DDS" "Salam, abd Allah" "Sean O'Connor" "Sheik Abu Bakar" @@ -192,11 +219,14 @@ "Smith (Jones)" "Smith (née Jones)" "Smith Jr." +"Smith Jr., MA" +"Smith Jr., Ma" "Smith Jr., Mr." "Smith Prof." "Smith Sir." "Smith née Jones" "Smith née Jones PhD" +"Smith, A.B." "Smith, Abd" "Smith, Dr." "Smith, Dr. Jr." @@ -207,7 +237,9 @@ "Smith, John V" "Smith, John V." "Smith, Jr." +"Smith, MA" "Smith, MD PhD" +"Smith, Ma" "Smith, Major. John" "Smith, Ms." "Smith, Ms. Jane" @@ -218,6 +250,7 @@ "Smith, Sr." "Smith, de Mesnil Jean" "Smith. John" +"Steven Hardman, MD, DO, DDS" "The Right Hon. the President of the Queen's Bench Division" "Van Johnson" "Vega, Juan de la" @@ -253,6 +286,8 @@ "ibn Awf abdul Rahman" "john e smith" "john smith phd" +"john smith x.y.z." +"john van der berg ma" "jose e maria santos" "juan de la vega" "juan garcia y lopez" diff --git a/tools/differential/corpus_shapes.jsonl b/tools/differential/corpus_shapes.jsonl index aaff9d87..8ed94df5 100644 --- a/tools/differential/corpus_shapes.jsonl +++ b/tools/differential/corpus_shapes.jsonl @@ -1,22 +1,60 @@ +{"name": "ANH DO", "shape": 1} +{"name": "Bridge (1.4)", "shape": 1} +{"name": "Bridge (A.B)", "shape": 1} {"name": "Dr. Juan de la Vega III", "shape": 1} +{"name": "J.A. K.D.", "shape": 1} +{"name": "J.R.R. Tolkien", "shape": 1} +{"name": "JACK MA", "shape": 1} {"name": "JOHN E SMITH", "shape": 1} +{"name": "JOHN SMITH MA", "shape": 1} {"name": "JOSE E MARIA SANTOS", "shape": 1} {"name": "JOSEP CAROD I ROVIRA", "shape": 1} {"name": "JUAN GARCIA Y LOPEZ", "shape": 1} {"name": "JUAN Y GARCIA", "shape": 1} +{"name": "Jack MA", "shape": 1} +{"name": "Jack MA.", "shape": 1} +{"name": "Jack Ma", "shape": 1} +{"name": "Jack X.Y.I.", "shape": 1} +{"name": "Jack X.Y.Z.", "shape": 1} +{"name": "Jean DUPONT", "shape": 1} +{"name": "Jean Pierre DUPONT", "shape": 1} {"name": "John \"Jack\" Kennedy", "shape": 1} +{"name": "John Doe JD.CPA", "shape": 1} {"name": "John Jack Andrew Kennedy", "shape": 1} {"name": "John Smith", "shape": 1} +{"name": "John Smith 1.4", "shape": 1} +{"name": "John Smith B.Tech.", "shape": 1} +{"name": "John Smith C.H.A.", "shape": 1} +{"name": "John Smith E.S.Q.", "shape": 1} +{"name": "John Smith J.u.n.i.o.r.", "shape": 1} {"name": "John Smith Jr.", "shape": 1} +{"name": "John Smith Ma", "shape": 1} +{"name": "John Smith Q.W.E.R.T.", "shape": 1} +{"name": "John Smith R.A.I.", "shape": 1} +{"name": "John Smith X.Y.Z.", "shape": 1} +{"name": "John Smith XYZ", "shape": 1} +{"name": "John Smith Xyz.", "shape": 1} {"name": "John V. Smith", "shape": 1} +{"name": "John de Ma", "shape": 1} {"name": "John e Smith", "shape": 1} +{"name": "John van der Berg Ma", "shape": 1} {"name": "Jose E Maria Santos", "shape": 1} {"name": "Jose e Maria Santos", "shape": 1} {"name": "Juan Garcia Y Lopez", "shape": 1} {"name": "Juan de la Vega", "shape": 1} +{"name": "Lt.Gov. John Doe", "shape": 1} {"name": "Md Abdul Karim", "shape": 1} +{"name": "Minjun KIM", "shape": 1} +{"name": "Mr MA", "shape": 1} {"name": "Sir Bob Andrew Dole", "shape": 1} +{"name": "X.Y.Z. Smith", "shape": 1} +{"name": "abdul Smith Jr Ma", "shape": 1} +{"name": "anh van do", "shape": 1} +{"name": "anh van mc", "shape": 1} +{"name": "jack ma", "shape": 1} {"name": "john e smith", "shape": 1} +{"name": "john smith MA", "shape": 1} +{"name": "john smith x.y.z.", "shape": 1} {"name": "jose e maria santos", "shape": 1} {"name": "josep carod i rovira", "shape": 1} {"name": "juan garcia y lopez", "shape": 1} @@ -26,18 +64,37 @@ {"name": "محمد و علي", "shape": 1} {"name": "Beethoven, Ludwig van", "shape": 2} {"name": "Doe, John A.", "shape": 2} +{"name": "Doe, John Msc.Ed.", "shape": 2} +{"name": "John Smith, 1.4", "shape": 2} {"name": "Kennedy, John (Jack)", "shape": 2} +{"name": "Royce, Ed", "shape": 2} {"name": "Salem, Abdul Rahman Ahmed", "shape": 2} +{"name": "Smith Jr., A.B.", "shape": 2} {"name": "Smith Jr., John", "shape": 2} +{"name": "Smith Jr., MA", "shape": 2} +{"name": "Smith, A.B.", "shape": 2} +{"name": "Smith, A.B.C.", "shape": 2} {"name": "Smith, Dr. John", "shape": 2} +{"name": "Smith, E.S.Q.", "shape": 2} {"name": "Smith, John", "shape": 2} {"name": "Smith, John Jr.", "shape": 2} {"name": "Smith, John V, Jr.", "shape": 2} {"name": "Smith, John, Extra, Jr.", "shape": 2} +{"name": "Smith, MA", "shape": 2} +{"name": "Smith, Ma", "shape": 2} {"name": "de la Vega, Juan", "shape": 2} +{"name": "Davis Royce, Ed", "shape": 3} {"name": "Dr. John P. Doe-Ray, CLU, CFP, LUTC", "shape": 3} +{"name": "JOHN SMITH, MA", "shape": 3} {"name": "John Smith Jr., PhD", "shape": 3} +{"name": "John Smith, A.B.", "shape": 3} +{"name": "John Smith, Ed", "shape": 3} +{"name": "John Smith, MA", "shape": 3} +{"name": "John Smith, MD, Ma", "shape": 3} +{"name": "John Smith, MD, R.A.I.", "shape": 3} {"name": "John Smith, PhD", "shape": 3} +{"name": "Steven Hardman, MD, DO, DDS", "shape": 3} +{"name": "john smith, ma", "shape": 3} {"name": "John Smith, Dr.", "shape": 4} {"name": "de Mesnil Jean, Dr.", "shape": 4} {"name": "de la Cruz Juan Carlos", "shape": 4} diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index a0310db4..f22c6a8a 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1192,6 +1192,14 @@ issue = "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stay # surname (v1 parity restored); side effect: parenthesized/quoted "MA" # or "DO" no longer escape to suffix (v1 did, since v1 treated them as # unambiguous there) -- they now fall through to nickname parsing. +# +# 2026-09-18 (#289): the first clause holds for the spelling it names +# and not for its all-caps twin. 'Jack Ma' still keeps its surname by +# the words-to-spare guard; 'Jack MA' is now read as a credential by +# the written case contrast, with nothing to spare -- the guard never +# reaches it. The parenthesized side effect above is untouched: a +# bracketed clause is decided before the class is asked at all, so +# 'Andrew Perkins (MA)' keeps its nickname whatever the case. # Not expected to fire against this corpus (no such strings survived # into the v1 banks); kept for documentation completeness. # The members are the entries themselves, undotted. They carried @@ -1836,8 +1844,21 @@ issue = "fix(#424) accepted: the chain keeps an acronym assign will not peel beh # fields-only fix(suffix-routing) catch-all -- the count was the tell. # #451 deleted that catch-all, so this ledger has no fields-only rule # left to absorb anything. +# +# 2026-09-18 (#289): a THIRD reading, and it is the one that ships. +# The comment above says the chain "asks the peel again and takes +# it"; with the case signal the peel takes the ACRONYM instead, so +# the name reads title 'Freiherr', family 'von Berg', suffix 'MA' -- +# neither 1.4.0's (first 'von Berg', last 'MA') nor 2.0-2.3's +# (family 'von Berg MA'). Accepted rather than repaired: the lean +# ends P2's chain with no words to spare where the count would not, +# and `fields` gains `suffix` because the diff moves it. Measured at +# this baseline: given 'von Berg' -> '', family 'MA' -> 'von Berg', +# suffix '' -> 'MA'. The rule kept explaining nothing between the +# case signal landing and this edit -- the diff had grown past the +# declaration, which is the under-declared half of #452. name_regex = "(?i)^freiherr\\s+von\\s+berg\\s+ma$" -fields = ["given", "family"] +fields = ["given", "family", "suffix"] [[change]] issue = "fix(#424) a title-led chain before the numeral is the one name piece" @@ -2746,6 +2767,15 @@ issue = "fix(suffix-routing) the dotted M.A. spelling reads as a credential (ma- # read differently ON PURPOSE, which is why both spellings of the bare # one are _MUST_NOT_MATCH probes here. # +# 2026-09-18 (#289): the quoted sentence holds for the spelling it +# names and not for its all-caps twin. 'Jack Ma' is still kept intact +# by the words-to-spare guard; 'Jack MA' is now read as a credential +# by the written case contrast, with nothing to spare -- the guard +# never reaches it, and the periods gate is not what decides it. Only +# 'Jack Ma' is still a probe here; 'Jack MA' moved to the fix(#289) +# rule's own roster at the bottom of this file, where the roster +# comment says why. decisions.md#ma-do carries the amendment. +# # Literal-anchored, and it could not be anything else. Measured: the # member `m\.?a\.?` matches the fragment 'M.A.' in the corpus, and # _normalize leaves that as 'm.a', which is not a SUFFIX_ACRONYMS entry @@ -2844,14 +2874,17 @@ issue = "fix(#385/#402) an all-particle name part initials its words (R2)" # 1.4.0 contributed nothing for such a part ('Anh Do' -> 'A.'; the # tree gives 'A. D.'). The fields did not move, only the view, which # is why no rule could classify this until #484 gave the gate an -# initials column. Reach exactly the 27 names that move (24 spellings, -# three of them case variants), listed literally rather than shaped: +# initials column. Reach exactly the 28 names that move (25 spellings, +# three of them case variants), listed literally rather than shaped +# -- 'anh van mc' joined on 2026-09-18 with the case row that +# admitted it to the corpus, the unambiguous-vocabulary twin of the +# 'anh van do' already here: # the shape is "a part of nothing but particles", which no regex over # the raw string can state, and the fix(#445) rule above set the # precedent for a name list in THIS ledger (fix(#410) and fix(#335) # are two more). 'de los Santos' is NOT here: its # initials moved for a different reason (the rule below). -name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" +name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|anh van mc|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" fields = ["_initials"] orders = ["DEFAULT"] @@ -3127,6 +3160,21 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # reading nobody wants, and this reading is the one the decision # chose. # +# 2026-09-18 (#516): this rule stopped explaining anything, and the +# reason is not a revert. The by-shape credential class reads a word +# of two or more period-separated chunks that no vocabulary claims by +# POSITION (rules.md#S3), so 'John Smith E.S.Q.' is a credential again +# with no wordlist entry behind it at all -- the reading this baseline +# has, recovered by a different route than the acronym entry gave it. +# The vocabulary decision the `change` tag records still stands and is +# still the history of that entry; what is gone is its diff AT THIS +# BASELINE, where `_ambiguities` is not a compared surface at all. +# Above 2.0 the name still diffs, on the `suffix-or-name` report the +# positional reading brings with it, and fix(#516) classifies that -- +# each 2.x copy's own `dormant` string says so. Declared +# `dormant` rather than deleted, so the harness says so out loud if a +# future change takes the positional reading away again. +# # Literal, no alternation, and exactly as wide as the diff: 'esq' # keeps its SUFFIX_WORDS membership, so the acronym entry's only # unique coverage was this one spelling and this one corpus name. @@ -3134,6 +3182,7 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # 'John Smith Esq' (the word list still reads it), 'Esq. Smith' # (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone # post-comma credential). +dormant = "'John Smith E.S.Q.' no longer diffs at this baseline: #516's by-shape credential class reads the multi-dot spelling as a credential by position, with no wordlist entry, which is the reading this baseline already had. Kept because the vocabulary change it records is real history and the rule wakes if that reading is lost again" name_regex = "^John Smith E\\.S\\.Q\\.$" fields = ["family", "middle", "suffix"] @@ -3192,6 +3241,16 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # +# 2026-09-18 (#289): the last clause is retracted. S2's reserve is no +# longer what decides 'John Prof. MA': written capitals in a +# mixed-case name lean credential with nothing to spare, so the name +# reads title 'Prof.', family 'John', suffix 'MA' -- the transparency +# this rule is about is unchanged and now shows a DIFFERENT reading +# through it, which is the point of transparency. This rule keeps the +# name at every baseline where it already had it; the fix(#289) rule +# at the bottom of each file carries the names whose reading moved and +# that nothing else claims. +# # Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and @@ -3254,7 +3313,18 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +# +# 2026-09-18 (#289/#516): a SEVENTEENTH member, 'JOHN PROF. MA'. It +# arrived in the rules corpus as rules.md#H5's one-case witness that +# the suffix reading is taken over what STANDS once the title chain +# has run, not over what stood -- the reading the mixed-case spelling +# stopped demonstrating when the caps lean started deciding it. The +# name is wholly one case, so no lean fires and nothing about it is +# this arc's: what moves is exactly this rule's trailing-title chain +# ({title, family, suffix}, inside the five roles already declared), +# and it stops diffing at 2.3.0 for the same reason its sixteen +# siblings do. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|JOHN PROF\\. MA|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] [[change]] @@ -3449,3 +3519,153 @@ issue = "fix(#383/#479) a single-letter connective joins only on case evidence" name_regex = "^(?:jose e maria santos|JUAN GARCIA Y LOPEZ)$" fields = ["given", "middle", "family"] orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289) a written case contrast decides a bare ambiguous acronym" +# A member of suffix_acronyms_ambiguous written in CAPITALS inside a +# mixed-case name is written the way a credential is written, and is +# taken as one with nothing to spare ('Jack MA' -> given 'Jack', +# suffix 'MA'); one written in any other cased form is written the way +# a surname is written and stays one even with words to spare ('John +# Smith Ma' -> middle 'Smith', family 'Ma'). The same lean reaches the +# post-comma given slot ('Smith, MA' -> family 'Smith', suffix 'MA'; +# 'Smith, Ma' keeps its given), and the comma form's words-to-spare +# test becomes a count of NAME words, which is case-blind -- so +# 'John Smith, MA', 'John Smith, Ed', 'JOHN SMITH, MA' and +# 'john smith, ma' all read the credential where the token count read +# a given name. See rules.md#S2, rules.md#C1 and rules.md#P2. +# +# A name written wholly in one case carries no contrast and does not +# move on the LEAN at all -- 'JOHN SMITH MA', 'ANH DO', 'anh van do' +# and 'Jack Ma' are _MUST_NOT_MATCH probes in +# tests/v2/test_ledger_guards.py. The one-case comma forms in the list +# above are here on the NAME-word count instead, which asks nothing +# about case; that is why they sit beside the lean's own names rather +# than in a rule of their own. +# +# Accepted costs, both recorded rather than repaired: the walk stops +# at a declined pick instead of continuing past it, so a genuine +# suffix standing in front of a name-leaning acronym is never reached +# ('abdul Smith Jr Ma' reads middle 'Jr'), and the lean ends P2's +# particle chain with no words to spare where the count would not +# ('Freiherr von Berg MA', whose own rule further up this file carries +# the reading at 1.4.0). +# +# The Title-case half is witnessed at the comma by two minimal pairs +# rather than argued: 'John Smith, MA'/'John Smith, Ma' both read the +# credential, because two NAME words before the comma decide before +# the case is looked at, and 'Smith Jr., MA'/'Smith Jr., Ma' split, +# because one name word leaves the case to decide and Title-case +# declines. 'Smith Jr., Ma' diffs here for a reading this change did +# NOT move -- the pre-comma restructure v2 has done since 2.0, at +# 1.4.0, and the fork's new report at the 2.x baselines -- while +# 'John Smith, Ma' moves on the COUNT, as its all-caps twin does, and +# has no 1.4.0 diff at all, v1 having read the credential there too. +# Both sit on this rule because the rule is what states which way +# each member of a pair goes. +# +# Literal-anchored to the measured movers. The four 2.x ledgers share +# ONE set; this copy is that set less one name, for the reason the +# paragraph below gives. "One set for all five" was written before +# that carve-out existed and is corrected in all five copies +# (2026-09-18). The set is literal because the class is +# wide (every mixed-case name whose trailing piece is a listed +# ambiguous acronym, plus every comma form with two name words before +# the comma) and a regex for the class would claim the one-case +# spellings that do not move. A name here that does not diff at a +# given baseline is simply one the reading already agreed with there. +# `fields` is per-baseline, the union the run at THAT baseline +# measures (#452). +# +# One name is held OUT of the set here and only here: 'John Prof. MA' +# is claimed at this baseline by `fix(#316) a trailing period-marked +# title word reads as a title`, whose own comment carries the dated +# retraction of what the name used to demonstrate. That rule declares +# a `title` role this one does not, describes the whole of the diff +# rather than the acronym's half of it, and stands first -- so the +# name is narrowed out rather than left to be decided by file order, +# which the harness refuses to let go undeclared. +name_regex = "^(?:Davis Royce, Ed|Freiherr von Berg MA|JOHN SMITH, MA|Jack MA|Jack MA\\.|Jack Wei Ma|John Smith Ma|John Smith, Ed|John Smith, MA|John Smith, Ma|John de Ma|John van der Berg Ma|Smith Jr\\., MA|Smith Jr\\., Ma|Smith, MA|abdul Smith Berg Ma|abdul Smith Jr Ma|abdul Smith Ma|john smith, ma)$" +fields = ["family", "given", "middle", "suffix"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#516) an unlisted dotted acronym is read by position" +# A token of two or more period-separated chunks that no vocabulary +# claims joins the ambiguous credential class by SHAPE and is read by +# POSITION, exactly as a bare listed acronym is: a credential where +# the name has words to spare, a name word where it does not, either +# reading reported (rules.md#S3). Case says nothing here -- the +# periods are the evidence -- which is why 'john smith x.y.z.' moves +# with 'John Smith X.Y.Z.'. The same admission reaches the comma form, +# where the NAME-word count decides ('John Smith, A.B.' -> suffix +# 'A.B.'; 'Smith, A.B.' keeps its given and only reports). +# +# Two names here restore a reading a PRIOR bundle's vocabulary change +# took away, by a different route than it lost it: 'John Smith E.S.Q.' +# and 'John Smith C.H.A.' fell to a bare positional read when 'esq' +# and 'cha' left SUFFIX_ACRONYMS +# (decisions.md#suffix-acronym-collisions), and the shape class reads +# them as the credentials they always were with no wordlist entry. +# At the baselines where +# that restoration lands on the baseline's own reading, only the new +# report is left and `fields` says so. +# +# The narrow half of the same change: a chunk match no longer makes a +# dotted token a suffix where EVERY matched chunk is a single ASCII +# character, which reaches only the roman numerals -- 'Jack X.Y.I.' +# moves to family for that reason and not for the shape one. +# +# The shapes the vocabulary settles are untouched and are +# _MUST_NOT_MATCH probes in tests/v2/test_ledger_guards.py: a +# whole-token match ('Jack M.A.', 'John Smith Ph.D.', 'Smith, +# A.B.C.'), a surviving chunk claim ('Doe, John Msc.Ed.'), a leading +# dotted run ('X.Y.Z. Smith', 'J.R.R. Tolkien', 'A.B. Vajpayee'), the +# single trailing period that is no shape at all ('John Smith Xyz.'), +# and a digit chunk, which the alphabetic gate refuses ('Bridge (1.4)'). +# +# Literal-anchored to the measured movers, one set for all five +# ledgers, for the reason the fix(#289) rule above gives: the class is +# a SHAPE, and a regex for the shape would claim every dotted token +# the vocabulary already answers for. +name_regex = "^(?:Jack X\\.Y\\.I\\.|John Smith B\\.Tech\\.|John Smith C\\.H\\.A\\.|John Smith E\\.S\\.Q\\.|John Smith Q\\.W\\.E\\.R\\.T\\.|John Smith X\\.Y\\.Z\\.|John Smith, A\\.B\\.|Smith, E\\.S\\.Q\\.|john smith x\\.y\\.z\\.)$" +fields = ["family", "given", "middle", "suffix"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(given-part-trailing-slot) a credential acronym ending the given part of a family-comma listing reads as a middle name" +# THIS PR DID NOT MOVE THIS READING, and the rule exists only because +# the name entered the corpus here. 'Doe, John MA' reads suffix 'MA' +# on the 1.4.0 wheel and middle 'MA' at 2.0.0, 2.1.0, 2.2.0, 2.3.0 and +# on this tree alike -- a 2.0-era reading, so the diff is against v1 +# only and no 2.x ledger carries it. The name arrived as a rules.md#S2 +# boundary example line (the verification round of 2026-09-18), which +# is why a reading five years old is being classified today. +# +# The mechanism is a SLOT rather than a vocabulary or a case lean: +# assign's family-comma emitter reads the first piece after the comma, +# so a class member standing at the END of the given part is never +# asked about, and the positional reading takes it as a middle name in +# silence. rules.md#S2 states which slots report and names this one as +# the slot that does not; rules.md#S3 carries the same boundary for +# the dotted spelling. +# +# The dotted sibling is NOT here and must not be folded in: 'Doe, John +# X.Y.Z.' reads middle 'X.Y.Z.' at 1.4.0 too, so it produces no diff +# at this baseline at all. One name, one reading, one spelling -- the +# acronym half is the only half with a v1 disagreement. +# +# Whether to widen the emitter to this slot is open, recorded in +# decisions.md#S2's 2026-09-18 comma-reach entry, which now carries +# the 1.4.0 measurement as a PARITY question beside the noise one. If +# that widening ever lands, the roles here stop moving and this rule +# retires; a `suffix-or-name` report would be added on top and is not +# a field this rule claims. +# +# Literal-anchored for the same reason the fix(#289) rule above gives: +# the class is a slot, and a regex for the slot would claim every +# family-comma name whose given part ends in an ambiguous acronym, +# including the ones v1 agreed with. +name_regex = "^Doe, John MA$" +fields = ["middle", "suffix"] +orders = ["DEFAULT"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 84c64ae7..3a70abdd 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -768,8 +768,24 @@ issue = "fix(#425) the bound-given reserve runs assign's peel over the joined vi # turned a credential into the family). Both 1.4.0 parity, so the # 1.4.0 ledger has no twin. Rules.md examples; no differential corpus # name has either shape. +# +# 2026-09-18 (#289): the sentence above -- "Both 1.4.0 parity, so the +# 1.4.0 ledger has no twin" -- stops being true, and this rule's own +# diff shrinks at the same time. The surname lean (rules.md#S2) never +# lets the peel take a Title-case 'Ma', so the acronym is a name word +# to spare and the join fires: 'abdul Smith Ma' reads given 'abdul +# Smith', family 'Ma' -- which is what THIS baseline reads, so the +# name stops diffing here altogether -- and 'abdul Smith Jr Ma' reads +# given 'abdul Smith', middle 'Jr', family 'Ma', where v1 read first +# 'abdul', last 'Smith' and the acronym as a suffix in both. The +# 1.4.0 ledger gains its twin, inside the fix(#289) rule at the +# bottom of that file. `Jr` as a middle name is an ACCEPTED cost: the +# surname lean breaks the peel at 'Ma', so the suffix behind it is +# never reached. `fields` is narrowed to the union the run measures +# here -- `given` no longer moves on the one name left diffing, and +# a role nothing moves is a standing claim (#452). name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" -fields = ["given", "family", "suffix"] +fields = ["family", "middle", "suffix"] [[change]] issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example" @@ -1040,6 +1056,19 @@ issue = "fix(#424) the particle chain stops before a bare acronym with words to # Smith Ma' reads -- and as 1.4.0 read it, so this rule has no 1.4.0 # twin: a 2.0 regression. A rules.md example; no differential corpus # name has the shape. +# +# 2026-09-18 (#289): the MIXED-CASE spelling stops diffing here, and +# the rule survives on the one-case one. The surname lean +# (rules.md#S2) never lets the peel take a Title-case 'Ma', so +# 'John van der Berg Ma' reads family 'van der Berg Ma' again -- +# which is exactly what this baseline reads. What the chain does when +# the acronym IS taken is unchanged, and the one-case spelling +# 'john van der berg ma' shows it: no case contrast, the count +# decides, the chain stops before the acronym and the name reads +# family 'van der berg', suffix 'ma'. That spelling entered the +# corpus with rules.md#P2's own contrast pair in 2.4, which is why +# this rule went briefly dormant and then woke in the same commit. +# The case-insensitive regex was already written to reach both. name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+ma$" fields = ["family", "suffix", "_ambiguities"] @@ -1755,9 +1784,9 @@ orders = ["FAMILY_FIRST_GIVEN_LAST"] issue = "fix(#385/#402) an all-particle name part initials its words (R2)" # 'Anh Do' -> 'A. D.', 'Juan van der' -> 'J. v. d.': rules.md#R2/#R3, # shipped in 2.2.0, decisions.md#R2. Fields identical at this -# baseline; only the view moved. The same 27-name literal list as the +# baseline; only the view moved. The same 28-name literal list as the # 1.4.0 rule, for the same reason. -name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" +name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|anh van mc|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" fields = ["_initials"] orders = ["DEFAULT"] @@ -1911,6 +1940,24 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # reading nobody wants, and this reading is the one the decision # chose. # +# 2026-09-18 (#516): this rule stopped explaining anything, and the +# reason is not a revert. The by-shape credential class reads a word +# of two or more period-separated chunks that no vocabulary claims by +# POSITION (rules.md#S3), so 'John Smith E.S.Q.' is a credential again +# with no wordlist entry behind it at all -- the reading this baseline +# has, recovered by a different route than the acronym entry gave it. +# The vocabulary decision the `change` tag records still stands and is +# still the history of that entry; what is gone is its ROLE diff. The +# name still diffs here, on the `suffix-or-name` report the positional +# reading brings with it -- `_ambiguities` is a compared surface from +# 2.0 on -- and fix(#516) classifies that, which is why this rule is +# dormant rather than explaining anything: `dormant` means "explained +# no diff", not "the name stopped moving". Only the 1.4.0 copy can say +# the diff is gone outright, that baseline comparing no ambiguities. +# Declared +# `dormant` rather than deleted, so the harness says so out loud if a +# future change takes the positional reading away again. +# # Literal, no alternation, and exactly as wide as the diff: 'esq' # keeps its SUFFIX_WORDS membership, so the acronym entry's only # unique coverage was this one spelling and this one corpus name. @@ -1918,6 +1965,7 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # 'John Smith Esq' (the word list still reads it), 'Esq. Smith' # (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone # post-comma credential). +dormant = "'John Smith E.S.Q.' no longer moves any role this rule declares: #516's by-shape credential class reads the multi-dot spelling as a credential by position, with no wordlist entry, which is the reading this baseline already had. Its remaining diff here is the `suffix-or-name` report, which fix(#516) classifies. Kept because the vocabulary change it records is real history and the rule wakes if that reading is lost again" name_regex = "^John Smith E\\.S\\.Q\\.$" fields = ["family", "middle", "suffix"] @@ -1976,6 +2024,16 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # +# 2026-09-18 (#289): the last clause is retracted. S2's reserve is no +# longer what decides 'John Prof. MA': written capitals in a +# mixed-case name lean credential with nothing to spare, so the name +# reads title 'Prof.', family 'John', suffix 'MA' -- the transparency +# this rule is about is unchanged and now shows a DIFFERENT reading +# through it, which is the point of transparency. This rule keeps the +# name at every baseline where it already had it; the fix(#289) rule +# at the bottom of each file carries the names whose reading moved and +# that nothing else claims. +# # Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and @@ -2038,7 +2096,18 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +# +# 2026-09-18 (#289/#516): a SEVENTEENTH member, 'JOHN PROF. MA'. It +# arrived in the rules corpus as rules.md#H5's one-case witness that +# the suffix reading is taken over what STANDS once the title chain +# has run, not over what stood -- the reading the mixed-case spelling +# stopped demonstrating when the caps lean started deciding it. The +# name is wholly one case, so no lean fires and nothing about it is +# this arc's: what moves is exactly this rule's trailing-title chain +# ({title, family, suffix}, inside the five roles already declared), +# and it stops diffing at 2.3.0 for the same reason its sixteen +# siblings do. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|JOHN PROF\\. MA|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] [[change]] @@ -2263,3 +2332,169 @@ issue = "fix(#383/#479) a marked connective letter in a one-case name is reporte name_regex = "^(?:JOSE E MARIA SANTOS|JOHN E SMITH|john e smith|john e jones|jones, john e|e j smith)$" fields = ["_ambiguities"] orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289) a written case contrast decides a bare ambiguous acronym" +# A member of suffix_acronyms_ambiguous written in CAPITALS inside a +# mixed-case name is written the way a credential is written, and is +# taken as one with nothing to spare ('Jack MA' -> given 'Jack', +# suffix 'MA'); one written in any other cased form is written the way +# a surname is written and stays one even with words to spare ('John +# Smith Ma' -> middle 'Smith', family 'Ma'). The same lean reaches the +# post-comma given slot ('Smith, MA' -> family 'Smith', suffix 'MA'; +# 'Smith, Ma' keeps its given), and the comma form's words-to-spare +# test becomes a count of NAME words, which is case-blind -- so +# 'John Smith, MA', 'John Smith, Ed', 'JOHN SMITH, MA' and +# 'john smith, ma' all read the credential where the token count read +# a given name. See rules.md#S2, rules.md#C1 and rules.md#P2. +# +# A name written wholly in one case carries no contrast and does not +# move on the LEAN at all -- 'JOHN SMITH MA', 'ANH DO', 'anh van do' +# and 'Jack Ma' are _MUST_NOT_MATCH probes in +# tests/v2/test_ledger_guards.py. The one-case comma forms in the list +# above are here on the NAME-word count instead, which asks nothing +# about case; that is why they sit beside the lean's own names rather +# than in a rule of their own. +# +# Accepted costs, both recorded rather than repaired: the walk stops +# at a declined pick instead of continuing past it, so a genuine +# suffix standing in front of a name-leaning acronym is never reached +# ('abdul Smith Jr Ma' reads middle 'Jr'), and the lean ends P2's +# particle chain with no words to spare where the count would not +# ('Freiherr von Berg MA', whose own rule further up this file carries +# the reading at 1.4.0). +# +# The Title-case half is witnessed at the comma by two minimal pairs +# rather than argued: 'John Smith, MA'/'John Smith, Ma' both read the +# credential, because two NAME words before the comma decide before +# the case is looked at, and 'Smith Jr., MA'/'Smith Jr., Ma' split, +# because one name word leaves the case to decide and Title-case +# declines. 'Smith Jr., Ma' diffs here for a reading this change did +# NOT move -- the pre-comma restructure v2 has done since 2.0, at +# 1.4.0, and the fork's new report at the 2.x baselines -- while +# 'John Smith, Ma' moves on the COUNT, as its all-caps twin does, and +# has no 1.4.0 diff at all, v1 having read the credential there too. +# Both sit on this rule because the rule is what states which way +# each member of a pair goes. +# +# Literal-anchored to the measured movers, and ONE SET FOR THE FOUR +# 2.x LEDGERS so the rule says the same thing at each of them. The +# 1.4.0 copy differs by exactly one member and holds 'John Prof. MA' +# out: 1.4.0 read a trailing period-marked title as a title itself, +# so that name's diff there belongs to fix(#316)'s rule and not to +# this one. "One set for all five" was written before that carve-out +# existed and was false in these four copies from the day it did +# (2026-09-18). The set is literal because the class is +# wide (every mixed-case name whose trailing piece is a listed +# ambiguous acronym, plus every comma form with two name words before +# the comma) and a regex for the class would claim the one-case +# spellings that do not move. A name here that does not diff at a +# given baseline is simply one the reading already agreed with there. +# `fields` is per-baseline, the union the run at THAT baseline +# measures (#452). +name_regex = "^(?:Davis Royce, Ed|Freiherr von Berg MA|JOHN SMITH, MA|Jack MA|Jack MA\\.|Jack Wei Ma|John Prof\\. MA|John Smith Ma|John Smith, Ed|John Smith, MA|John Smith, Ma|John de Ma|John van der Berg Ma|Smith Jr\\., MA|Smith Jr\\., Ma|Smith, MA|abdul Smith Berg Ma|abdul Smith Jr Ma|abdul Smith Ma|john smith, ma)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#516) an unlisted dotted acronym is read by position" +# A token of two or more period-separated chunks that no vocabulary +# claims joins the ambiguous credential class by SHAPE and is read by +# POSITION, exactly as a bare listed acronym is: a credential where +# the name has words to spare, a name word where it does not, either +# reading reported (rules.md#S3). Case says nothing here -- the +# periods are the evidence -- which is why 'john smith x.y.z.' moves +# with 'John Smith X.Y.Z.'. The same admission reaches the comma form, +# where the NAME-word count decides ('John Smith, A.B.' -> suffix +# 'A.B.'; 'Smith, A.B.' keeps its given and only reports). +# +# Two names here restore a reading a PRIOR bundle's vocabulary change +# took away, by a different route than it lost it: 'John Smith E.S.Q.' +# and 'John Smith C.H.A.' fell to a bare positional read when 'esq' +# and 'cha' left SUFFIX_ACRONYMS +# (decisions.md#suffix-acronym-collisions), and the shape class reads +# them as the credentials they always were with no wordlist entry. +# At the baselines where +# that restoration lands on the baseline's own reading, only the new +# report is left and `fields` says so. +# +# The narrow half of the same change: a chunk match no longer makes a +# dotted token a suffix where EVERY matched chunk is a single ASCII +# character, which reaches only the roman numerals -- 'Jack X.Y.I.' +# moves to family for that reason and not for the shape one. +# +# The shapes the vocabulary settles are untouched and are +# _MUST_NOT_MATCH probes in tests/v2/test_ledger_guards.py: a +# whole-token match ('Jack M.A.', 'John Smith Ph.D.', 'Smith, +# A.B.C.'), a surviving chunk claim ('Doe, John Msc.Ed.'), a leading +# dotted run ('X.Y.Z. Smith', 'J.R.R. Tolkien', 'A.B. Vajpayee'), the +# single trailing period that is no shape at all ('John Smith Xyz.'), +# and a digit chunk, which the alphabetic gate refuses ('Bridge (1.4)'). +# +# Literal-anchored to the measured movers, one set for all five +# ledgers, for the reason the fix(#289) rule above gives: the class is +# a SHAPE, and a regex for the shape would claim every dotted token +# the vocabulary already answers for. +name_regex = "^(?:Jack X\\.Y\\.I\\.|John Smith B\\.Tech\\.|John Smith C\\.H\\.A\\.|John Smith E\\.S\\.Q\\.|John Smith Q\\.W\\.E\\.R\\.T\\.|John Smith X\\.Y\\.Z\\.|John Smith, A\\.B\\.|Smith, E\\.S\\.Q\\.|john smith x\\.y\\.z\\.)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the ambiguous credential class reports at slots that were silent" +# Every decision at the trailing suffix slot, the post-comma given +# slot and a tail segment now reports the EXISTING SUFFIX_OR_NAME -- +# no new kind, and the report tracks the FORK BEING CONSULTED rather +# than which way it went, as the trailing slot always has. These names +# gain it with no role moving at any baseline: the fork was there all +# along and was called silently. It is the first time the comma's +# OWN decision is reported (rules.md#C1); P6's attachment fork has +# reported on a family-comma path since 2.3 ('Berg, Jan vd'). C2's +# structural flag was already the first comma-path report of a shape. +# +# 'Steven Hardman, MD, DO, DDS' goes the other way and LOSES its +# comma-structure flag: its third segment now reads as the credential +# run it is, because 'DO' leans credential in a mixed-case name. That +# is the one place this arc quiets a report rather than adding one. +# +# `_ambiguities` alone, which keeps `_initials` out of these rows as +# always (#484), and the rule cannot absorb a role diff on any of +# these names for the same reason. Literal-anchored, one set for all +# the 2.x ledgers; the 1.4.0 ledger has no twin, `_ambiguities` being +# a v2 surface that cannot enter a diff below baseline 2.0. +# +# 'Steven Hardman, MD, DO, DDS' is held OUT of the set at THIS +# baseline: `fix(#296) a dropped prenominal takes the name position it +# occupies` reaches it through its own comma anchor, declares the +# three roles this rule does not, and stands first -- so the name is +# narrowed out rather than left to file order. The comma-structure +# flag it loses is a 2.2-era report in any case, and the 2.2.0 and +# 2.3.0 ledgers are where the loss is recorded. +name_regex = "^(?:J\\.A\\. K\\.D\\.|Jack X\\.Y\\.Z\\.|John Smith J\\.u\\.n\\.i\\.o\\.r\\.|John Smith R\\.A\\.I\\.|Royce, Ed|Smith Jr\\., A\\.B\\.|Smith, A\\.B\\.|Smith, Ma)$" +fields = ["_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the glued CJK honorific peel reads the case lean" +# The suffix-run predicate the peel uses to decline a post-comma run +# (rules.md#W3, asking C1's own vocabulary question) now reads the +# same written-case evidence rule S2 reads, because `one_case` is +# passed to it. Before, one name written with two credentials divided +# two ways: a Kim-prefixed 김민준씨 input peeled its honorific when the +# credential was 'PhD' and did not when it was 'MA', the only +# difference being that 'MA' is an ambiguous acronym whose case lean +# the predicate was not given. +# +# Every name here is a COMPOSED form -- a Latin wrapper and a family +# comma around a CJK name -- which no East Asian writing system +# produces, so all three are `tolerated` case rows on the radar tier +# (decisions.md#cjk-comma-demotion). They are classified all the same: +# a radar diff this arc INTENDED is one a release note may be written +# from, and an unclassified one is not. +# +# `title` is in `fields` because 'Dr. 김민준씨, MA' had the whole +# string for its family at these baselines, the title included; at +# 2.0.0 and 2.1.0 'Jo김민준씨, DO' moves `title` instead of `given`, +# and the union is what a per-baseline `fields` records (#452). +name_regex = "^(?:Dr\\. 김민준씨, MA|Jo김민준씨, DO|Kim김민준씨, MA)$" +fields = ["title", "given", "family", "suffix", "_ambiguities"] +orders = ["DEFAULT"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index ab73ca26..3ac9d0cf 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -424,8 +424,24 @@ issue = "fix(#425) the bound-given reserve runs assign's peel over the joined vi # turned a credential into the family). Both 1.4.0 parity, so the # 1.4.0 ledger has no twin. Rules.md examples; no differential corpus # name has either shape. +# +# 2026-09-18 (#289): the sentence above -- "Both 1.4.0 parity, so the +# 1.4.0 ledger has no twin" -- stops being true, and this rule's own +# diff shrinks at the same time. The surname lean (rules.md#S2) never +# lets the peel take a Title-case 'Ma', so the acronym is a name word +# to spare and the join fires: 'abdul Smith Ma' reads given 'abdul +# Smith', family 'Ma' -- which is what THIS baseline reads, so the +# name stops diffing here altogether -- and 'abdul Smith Jr Ma' reads +# given 'abdul Smith', middle 'Jr', family 'Ma', where v1 read first +# 'abdul', last 'Smith' and the acronym as a suffix in both. The +# 1.4.0 ledger gains its twin, inside the fix(#289) rule at the +# bottom of that file. `Jr` as a middle name is an ACCEPTED cost: the +# surname lean breaks the peel at 'Ma', so the suffix behind it is +# never reached. `fields` is narrowed to the union the run measures +# here -- `given` no longer moves on the one name left diffing, and +# a role nothing moves is a standing claim (#452). name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" -fields = ["given", "family", "suffix"] +fields = ["family", "middle", "suffix"] [[change]] issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example" @@ -706,6 +722,19 @@ issue = "fix(#424) the particle chain stops before a bare acronym with words to # Smith Ma' reads -- and as 1.4.0 read it, so this rule has no 1.4.0 # twin: a 2.0 regression. A rules.md example; no differential corpus # name has the shape. +# +# 2026-09-18 (#289): the MIXED-CASE spelling stops diffing here, and +# the rule survives on the one-case one. The surname lean +# (rules.md#S2) never lets the peel take a Title-case 'Ma', so +# 'John van der Berg Ma' reads family 'van der Berg Ma' again -- +# which is exactly what this baseline reads. What the chain does when +# the acronym IS taken is unchanged, and the one-case spelling +# 'john van der berg ma' shows it: no case contrast, the count +# decides, the chain stops before the acronym and the name reads +# family 'van der berg', suffix 'ma'. That spelling entered the +# corpus with rules.md#P2's own contrast pair in 2.4, which is why +# this rule went briefly dormant and then woke in the same commit. +# The case-insensitive regex was already written to reach both. name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+ma$" fields = ["family", "suffix", "_ambiguities"] @@ -1676,9 +1705,9 @@ orders = ["FAMILY_FIRST_GIVEN_LAST"] issue = "fix(#385/#402) an all-particle name part initials its words (R2)" # 'Anh Do' -> 'A. D.', 'Juan van der' -> 'J. v. d.': rules.md#R2/#R3, # shipped in 2.2.0, decisions.md#R2. Fields identical at this -# baseline; only the view moved. The same 27-name literal list as the +# baseline; only the view moved. The same 28-name literal list as the # 1.4.0 rule, for the same reason. -name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" +name_regex = "(?i)^(?:anh do|smith van der|yin le|yin a le|vai la|jong van der|jong, van der|juan van der|mesnil garcia de|mesnil garcia van|mesnil de|sander van|van ma van|anh van do|anh van mc|beethoven ludwig van|berg jan de jr\\.|john van mc|jong anke de|juan de|ménil christophe de|ménil de|nguyen thi van|nguyen, van le|van berg jan de)$" fields = ["_initials"] orders = ["DEFAULT"] @@ -1814,6 +1843,24 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # reading nobody wants, and this reading is the one the decision # chose. # +# 2026-09-18 (#516): this rule stopped explaining anything, and the +# reason is not a revert. The by-shape credential class reads a word +# of two or more period-separated chunks that no vocabulary claims by +# POSITION (rules.md#S3), so 'John Smith E.S.Q.' is a credential again +# with no wordlist entry behind it at all -- the reading this baseline +# has, recovered by a different route than the acronym entry gave it. +# The vocabulary decision the `change` tag records still stands and is +# still the history of that entry; what is gone is its ROLE diff. The +# name still diffs here, on the `suffix-or-name` report the positional +# reading brings with it -- `_ambiguities` is a compared surface from +# 2.0 on -- and fix(#516) classifies that, which is why this rule is +# dormant rather than explaining anything: `dormant` means "explained +# no diff", not "the name stopped moving". Only the 1.4.0 copy can say +# the diff is gone outright, that baseline comparing no ambiguities. +# Declared +# `dormant` rather than deleted, so the harness says so out loud if a +# future change takes the positional reading away again. +# # Literal, no alternation, and exactly as wide as the diff: 'esq' # keeps its SUFFIX_WORDS membership, so the acronym entry's only # unique coverage was this one spelling and this one corpus name. @@ -1821,6 +1868,7 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # 'John Smith Esq' (the word list still reads it), 'Esq. Smith' # (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone # post-comma credential). +dormant = "'John Smith E.S.Q.' no longer moves any role this rule declares: #516's by-shape credential class reads the multi-dot spelling as a credential by position, with no wordlist entry, which is the reading this baseline already had. Its remaining diff here is the `suffix-or-name` report, which fix(#516) classifies. Kept because the vocabulary change it records is real history and the rule wakes if that reading is lost again" name_regex = "^John Smith E\\.S\\.Q\\.$" fields = ["family", "middle", "suffix"] @@ -1879,6 +1927,16 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # +# 2026-09-18 (#289): the last clause is retracted. S2's reserve is no +# longer what decides 'John Prof. MA': written capitals in a +# mixed-case name lean credential with nothing to spare, so the name +# reads title 'Prof.', family 'John', suffix 'MA' -- the transparency +# this rule is about is unchanged and now shows a DIFFERENT reading +# through it, which is the point of transparency. This rule keeps the +# name at every baseline where it already had it; the fix(#289) rule +# at the bottom of each file carries the names whose reading moved and +# that nothing else claims. +# # Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and @@ -1941,7 +1999,18 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +# +# 2026-09-18 (#289/#516): a SEVENTEENTH member, 'JOHN PROF. MA'. It +# arrived in the rules corpus as rules.md#H5's one-case witness that +# the suffix reading is taken over what STANDS once the title chain +# has run, not over what stood -- the reading the mixed-case spelling +# stopped demonstrating when the caps lean started deciding it. The +# name is wholly one case, so no lean fires and nothing about it is +# this arc's: what moves is exactly this rule's trailing-title chain +# ({title, family, suffix}, inside the five roles already declared), +# and it stops diffing at 2.3.0 for the same reason its sixteen +# siblings do. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|JOHN PROF\\. MA|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] [[change]] @@ -2150,3 +2219,169 @@ issue = "fix(#383/#479) a marked connective letter in a one-case name is reporte name_regex = "^(?:JOSE E MARIA SANTOS|JOHN E SMITH|john e smith|john e jones|jones, john e|e j smith)$" fields = ["_ambiguities"] orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289) a written case contrast decides a bare ambiguous acronym" +# A member of suffix_acronyms_ambiguous written in CAPITALS inside a +# mixed-case name is written the way a credential is written, and is +# taken as one with nothing to spare ('Jack MA' -> given 'Jack', +# suffix 'MA'); one written in any other cased form is written the way +# a surname is written and stays one even with words to spare ('John +# Smith Ma' -> middle 'Smith', family 'Ma'). The same lean reaches the +# post-comma given slot ('Smith, MA' -> family 'Smith', suffix 'MA'; +# 'Smith, Ma' keeps its given), and the comma form's words-to-spare +# test becomes a count of NAME words, which is case-blind -- so +# 'John Smith, MA', 'John Smith, Ed', 'JOHN SMITH, MA' and +# 'john smith, ma' all read the credential where the token count read +# a given name. See rules.md#S2, rules.md#C1 and rules.md#P2. +# +# A name written wholly in one case carries no contrast and does not +# move on the LEAN at all -- 'JOHN SMITH MA', 'ANH DO', 'anh van do' +# and 'Jack Ma' are _MUST_NOT_MATCH probes in +# tests/v2/test_ledger_guards.py. The one-case comma forms in the list +# above are here on the NAME-word count instead, which asks nothing +# about case; that is why they sit beside the lean's own names rather +# than in a rule of their own. +# +# Accepted costs, both recorded rather than repaired: the walk stops +# at a declined pick instead of continuing past it, so a genuine +# suffix standing in front of a name-leaning acronym is never reached +# ('abdul Smith Jr Ma' reads middle 'Jr'), and the lean ends P2's +# particle chain with no words to spare where the count would not +# ('Freiherr von Berg MA', whose own rule further up this file carries +# the reading at 1.4.0). +# +# The Title-case half is witnessed at the comma by two minimal pairs +# rather than argued: 'John Smith, MA'/'John Smith, Ma' both read the +# credential, because two NAME words before the comma decide before +# the case is looked at, and 'Smith Jr., MA'/'Smith Jr., Ma' split, +# because one name word leaves the case to decide and Title-case +# declines. 'Smith Jr., Ma' diffs here for a reading this change did +# NOT move -- the pre-comma restructure v2 has done since 2.0, at +# 1.4.0, and the fork's new report at the 2.x baselines -- while +# 'John Smith, Ma' moves on the COUNT, as its all-caps twin does, and +# has no 1.4.0 diff at all, v1 having read the credential there too. +# Both sit on this rule because the rule is what states which way +# each member of a pair goes. +# +# Literal-anchored to the measured movers, and ONE SET FOR THE FOUR +# 2.x LEDGERS so the rule says the same thing at each of them. The +# 1.4.0 copy differs by exactly one member and holds 'John Prof. MA' +# out: 1.4.0 read a trailing period-marked title as a title itself, +# so that name's diff there belongs to fix(#316)'s rule and not to +# this one. "One set for all five" was written before that carve-out +# existed and was false in these four copies from the day it did +# (2026-09-18). The set is literal because the class is +# wide (every mixed-case name whose trailing piece is a listed +# ambiguous acronym, plus every comma form with two name words before +# the comma) and a regex for the class would claim the one-case +# spellings that do not move. A name here that does not diff at a +# given baseline is simply one the reading already agreed with there. +# `fields` is per-baseline, the union the run at THAT baseline +# measures (#452). +name_regex = "^(?:Davis Royce, Ed|Freiherr von Berg MA|JOHN SMITH, MA|Jack MA|Jack MA\\.|Jack Wei Ma|John Prof\\. MA|John Smith Ma|John Smith, Ed|John Smith, MA|John Smith, Ma|John de Ma|John van der Berg Ma|Smith Jr\\., MA|Smith Jr\\., Ma|Smith, MA|abdul Smith Berg Ma|abdul Smith Jr Ma|abdul Smith Ma|john smith, ma)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#516) an unlisted dotted acronym is read by position" +# A token of two or more period-separated chunks that no vocabulary +# claims joins the ambiguous credential class by SHAPE and is read by +# POSITION, exactly as a bare listed acronym is: a credential where +# the name has words to spare, a name word where it does not, either +# reading reported (rules.md#S3). Case says nothing here -- the +# periods are the evidence -- which is why 'john smith x.y.z.' moves +# with 'John Smith X.Y.Z.'. The same admission reaches the comma form, +# where the NAME-word count decides ('John Smith, A.B.' -> suffix +# 'A.B.'; 'Smith, A.B.' keeps its given and only reports). +# +# Two names here restore a reading a PRIOR bundle's vocabulary change +# took away, by a different route than it lost it: 'John Smith E.S.Q.' +# and 'John Smith C.H.A.' fell to a bare positional read when 'esq' +# and 'cha' left SUFFIX_ACRONYMS +# (decisions.md#suffix-acronym-collisions), and the shape class reads +# them as the credentials they always were with no wordlist entry. +# At the baselines where +# that restoration lands on the baseline's own reading, only the new +# report is left and `fields` says so. +# +# The narrow half of the same change: a chunk match no longer makes a +# dotted token a suffix where EVERY matched chunk is a single ASCII +# character, which reaches only the roman numerals -- 'Jack X.Y.I.' +# moves to family for that reason and not for the shape one. +# +# The shapes the vocabulary settles are untouched and are +# _MUST_NOT_MATCH probes in tests/v2/test_ledger_guards.py: a +# whole-token match ('Jack M.A.', 'John Smith Ph.D.', 'Smith, +# A.B.C.'), a surviving chunk claim ('Doe, John Msc.Ed.'), a leading +# dotted run ('X.Y.Z. Smith', 'J.R.R. Tolkien', 'A.B. Vajpayee'), the +# single trailing period that is no shape at all ('John Smith Xyz.'), +# and a digit chunk, which the alphabetic gate refuses ('Bridge (1.4)'). +# +# Literal-anchored to the measured movers, one set for all five +# ledgers, for the reason the fix(#289) rule above gives: the class is +# a SHAPE, and a regex for the shape would claim every dotted token +# the vocabulary already answers for. +name_regex = "^(?:Jack X\\.Y\\.I\\.|John Smith B\\.Tech\\.|John Smith C\\.H\\.A\\.|John Smith E\\.S\\.Q\\.|John Smith Q\\.W\\.E\\.R\\.T\\.|John Smith X\\.Y\\.Z\\.|John Smith, A\\.B\\.|Smith, E\\.S\\.Q\\.|john smith x\\.y\\.z\\.)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the ambiguous credential class reports at slots that were silent" +# Every decision at the trailing suffix slot, the post-comma given +# slot and a tail segment now reports the EXISTING SUFFIX_OR_NAME -- +# no new kind, and the report tracks the FORK BEING CONSULTED rather +# than which way it went, as the trailing slot always has. These names +# gain it with no role moving at any baseline: the fork was there all +# along and was called silently. It is the first time the comma's +# OWN decision is reported (rules.md#C1); P6's attachment fork has +# reported on a family-comma path since 2.3 ('Berg, Jan vd'). C2's +# structural flag was already the first comma-path report of a shape. +# +# 'Steven Hardman, MD, DO, DDS' goes the other way and LOSES its +# comma-structure flag: its third segment now reads as the credential +# run it is, because 'DO' leans credential in a mixed-case name. That +# is the one place this arc quiets a report rather than adding one. +# +# `_ambiguities` alone, which keeps `_initials` out of these rows as +# always (#484), and the rule cannot absorb a role diff on any of +# these names for the same reason. Literal-anchored, one set for all +# the 2.x ledgers; the 1.4.0 ledger has no twin, `_ambiguities` being +# a v2 surface that cannot enter a diff below baseline 2.0. +# +# 'Steven Hardman, MD, DO, DDS' is held OUT of the set at THIS +# baseline: `fix(#296) a dropped prenominal takes the name position it +# occupies` reaches it through its own comma anchor, declares the +# three roles this rule does not, and stands first -- so the name is +# narrowed out rather than left to file order. The comma-structure +# flag it loses is a 2.2-era report in any case, and the 2.2.0 and +# 2.3.0 ledgers are where the loss is recorded. +name_regex = "^(?:J\\.A\\. K\\.D\\.|Jack X\\.Y\\.Z\\.|John Smith J\\.u\\.n\\.i\\.o\\.r\\.|John Smith R\\.A\\.I\\.|Royce, Ed|Smith Jr\\., A\\.B\\.|Smith, A\\.B\\.|Smith, Ma)$" +fields = ["_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the glued CJK honorific peel reads the case lean" +# The suffix-run predicate the peel uses to decline a post-comma run +# (rules.md#W3, asking C1's own vocabulary question) now reads the +# same written-case evidence rule S2 reads, because `one_case` is +# passed to it. Before, one name written with two credentials divided +# two ways: a Kim-prefixed 김민준씨 input peeled its honorific when the +# credential was 'PhD' and did not when it was 'MA', the only +# difference being that 'MA' is an ambiguous acronym whose case lean +# the predicate was not given. +# +# Every name here is a COMPOSED form -- a Latin wrapper and a family +# comma around a CJK name -- which no East Asian writing system +# produces, so all three are `tolerated` case rows on the radar tier +# (decisions.md#cjk-comma-demotion). They are classified all the same: +# a radar diff this arc INTENDED is one a release note may be written +# from, and an unclassified one is not. +# +# `title` is in `fields` because 'Dr. 김민준씨, MA' had the whole +# string for its family at these baselines, the title included; at +# 2.0.0 and 2.1.0 'Jo김민준씨, DO' moves `title` instead of `given`, +# and the union is what a per-baseline `fields` records (#452). +name_regex = "^(?:Dr\\. 김민준씨, MA|Jo김민준씨, DO|Kim김민준씨, MA)$" +fields = ["title", "given", "family", "suffix", "_ambiguities"] +orders = ["DEFAULT"] diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index a27a0635..fc14719c 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -455,6 +455,24 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # reading nobody wants, and this reading is the one the decision # chose. # +# 2026-09-18 (#516): this rule stopped explaining anything, and the +# reason is not a revert. The by-shape credential class reads a word +# of two or more period-separated chunks that no vocabulary claims by +# POSITION (rules.md#S3), so 'John Smith E.S.Q.' is a credential again +# with no wordlist entry behind it at all -- the reading this baseline +# has, recovered by a different route than the acronym entry gave it. +# The vocabulary decision the `change` tag records still stands and is +# still the history of that entry; what is gone is its ROLE diff. The +# name still diffs here, on the `suffix-or-name` report the positional +# reading brings with it -- `_ambiguities` is a compared surface from +# 2.0 on -- and fix(#516) classifies that, which is why this rule is +# dormant rather than explaining anything: `dormant` means "explained +# no diff", not "the name stopped moving". Only the 1.4.0 copy can say +# the diff is gone outright, that baseline comparing no ambiguities. +# Declared +# `dormant` rather than deleted, so the harness says so out loud if a +# future change takes the positional reading away again. +# # Literal, no alternation, and exactly as wide as the diff: 'esq' # keeps its SUFFIX_WORDS membership, so the acronym entry's only # unique coverage was this one spelling and this one corpus name. @@ -462,6 +480,7 @@ issue = "change(suffix-acronym-collisions) esq leaves the acronym set" # 'John Smith Esq' (the word list still reads it), 'Esq. Smith' # (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone # post-comma credential). +dormant = "'John Smith E.S.Q.' no longer moves any role this rule declares: #516's by-shape credential class reads the multi-dot spelling as a credential by position, with no wordlist entry, which is the reading this baseline already had. Its remaining diff here is the `suffix-or-name` report, which fix(#516) classifies. Kept because the vocabulary change it records is real history and the rule wakes if that reading is lost again" name_regex = "^John Smith E\\.S\\.Q\\.$" fields = ["family", "middle", "suffix"] @@ -520,6 +539,16 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # +# 2026-09-18 (#289): the last clause is retracted. S2's reserve is no +# longer what decides 'John Prof. MA': written capitals in a +# mixed-case name lean credential with nothing to spare, so the name +# reads title 'Prof.', family 'John', suffix 'MA' -- the transparency +# this rule is about is unchanged and now shows a DIFFERENT reading +# through it, which is the point of transparency. This rule keeps the +# name at every baseline where it already had it; the fix(#289) rule +# at the bottom of each file carries the names whose reading moved and +# that nothing else claims. +# # Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and @@ -582,7 +611,18 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +# +# 2026-09-18 (#289/#516): a SEVENTEENTH member, 'JOHN PROF. MA'. It +# arrived in the rules corpus as rules.md#H5's one-case witness that +# the suffix reading is taken over what STANDS once the title chain +# has run, not over what stood -- the reading the mixed-case spelling +# stopped demonstrating when the caps lean started deciding it. The +# name is wholly one case, so no lean fires and nothing about it is +# this arc's: what moves is exactly this rule's trailing-title chain +# ({title, family, suffix}, inside the five roles already declared), +# and it stops diffing at 2.3.0 for the same reason its sixteen +# siblings do. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|JOHN PROF\\. MA|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] [[change]] @@ -801,3 +841,161 @@ issue = "fix(#383/#479) a marked connective letter in a one-case name is reporte name_regex = "^(?:JOSE E MARIA SANTOS|JOHN E SMITH|john e smith|john e jones|jones, john e|e j smith)$" fields = ["_ambiguities"] orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289) a written case contrast decides a bare ambiguous acronym" +# A member of suffix_acronyms_ambiguous written in CAPITALS inside a +# mixed-case name is written the way a credential is written, and is +# taken as one with nothing to spare ('Jack MA' -> given 'Jack', +# suffix 'MA'); one written in any other cased form is written the way +# a surname is written and stays one even with words to spare ('John +# Smith Ma' -> middle 'Smith', family 'Ma'). The same lean reaches the +# post-comma given slot ('Smith, MA' -> family 'Smith', suffix 'MA'; +# 'Smith, Ma' keeps its given), and the comma form's words-to-spare +# test becomes a count of NAME words, which is case-blind -- so +# 'John Smith, MA', 'John Smith, Ed', 'JOHN SMITH, MA' and +# 'john smith, ma' all read the credential where the token count read +# a given name. See rules.md#S2, rules.md#C1 and rules.md#P2. +# +# A name written wholly in one case carries no contrast and does not +# move on the LEAN at all -- 'JOHN SMITH MA', 'ANH DO', 'anh van do' +# and 'Jack Ma' are _MUST_NOT_MATCH probes in +# tests/v2/test_ledger_guards.py. The one-case comma forms in the list +# above are here on the NAME-word count instead, which asks nothing +# about case; that is why they sit beside the lean's own names rather +# than in a rule of their own. +# +# Accepted costs, both recorded rather than repaired: the walk stops +# at a declined pick instead of continuing past it, so a genuine +# suffix standing in front of a name-leaning acronym is never reached +# ('abdul Smith Jr Ma' reads middle 'Jr'), and the lean ends P2's +# particle chain with no words to spare where the count would not +# ('Freiherr von Berg MA', whose own rule further up this file carries +# the reading at 1.4.0). +# +# The Title-case half is witnessed at the comma by two minimal pairs +# rather than argued: 'John Smith, MA'/'John Smith, Ma' both read the +# credential, because two NAME words before the comma decide before +# the case is looked at, and 'Smith Jr., MA'/'Smith Jr., Ma' split, +# because one name word leaves the case to decide and Title-case +# declines. 'Smith Jr., Ma' diffs here for a reading this change did +# NOT move -- the pre-comma restructure v2 has done since 2.0, at +# 1.4.0, and the fork's new report at the 2.x baselines -- while +# 'John Smith, Ma' moves on the COUNT, as its all-caps twin does, and +# has no 1.4.0 diff at all, v1 having read the credential there too. +# Both sit on this rule because the rule is what states which way +# each member of a pair goes. +# +# Literal-anchored to the measured movers, and ONE SET FOR THE FOUR +# 2.x LEDGERS so the rule says the same thing at each of them. The +# 1.4.0 copy differs by exactly one member and holds 'John Prof. MA' +# out: 1.4.0 read a trailing period-marked title as a title itself, +# so that name's diff there belongs to fix(#316)'s rule and not to +# this one. "One set for all five" was written before that carve-out +# existed and was false in these four copies from the day it did +# (2026-09-18). The set is literal because the class is +# wide (every mixed-case name whose trailing piece is a listed +# ambiguous acronym, plus every comma form with two name words before +# the comma) and a regex for the class would claim the one-case +# spellings that do not move. A name here that does not diff at a +# given baseline is simply one the reading already agreed with there. +# `fields` is per-baseline, the union the run at THAT baseline +# measures (#452). +name_regex = "^(?:Davis Royce, Ed|Freiherr von Berg MA|JOHN SMITH, MA|Jack MA|Jack MA\\.|Jack Wei Ma|John Prof\\. MA|John Smith Ma|John Smith, Ed|John Smith, MA|John Smith, Ma|John de Ma|John van der Berg Ma|Smith Jr\\., MA|Smith Jr\\., Ma|Smith, MA|abdul Smith Berg Ma|abdul Smith Jr Ma|abdul Smith Ma|john smith, ma)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#516) an unlisted dotted acronym is read by position" +# A token of two or more period-separated chunks that no vocabulary +# claims joins the ambiguous credential class by SHAPE and is read by +# POSITION, exactly as a bare listed acronym is: a credential where +# the name has words to spare, a name word where it does not, either +# reading reported (rules.md#S3). Case says nothing here -- the +# periods are the evidence -- which is why 'john smith x.y.z.' moves +# with 'John Smith X.Y.Z.'. The same admission reaches the comma form, +# where the NAME-word count decides ('John Smith, A.B.' -> suffix +# 'A.B.'; 'Smith, A.B.' keeps its given and only reports). +# +# Two names here restore a reading a PRIOR bundle's vocabulary change +# took away, by a different route than it lost it: 'John Smith E.S.Q.' +# and 'John Smith C.H.A.' fell to a bare positional read when 'esq' +# and 'cha' left SUFFIX_ACRONYMS +# (decisions.md#suffix-acronym-collisions), and the shape class reads +# them as the credentials they always were with no wordlist entry. +# At the baselines where +# that restoration lands on the baseline's own reading, only the new +# report is left and `fields` says so. +# +# The narrow half of the same change: a chunk match no longer makes a +# dotted token a suffix where EVERY matched chunk is a single ASCII +# character, which reaches only the roman numerals -- 'Jack X.Y.I.' +# moves to family for that reason and not for the shape one. +# +# The shapes the vocabulary settles are untouched and are +# _MUST_NOT_MATCH probes in tests/v2/test_ledger_guards.py: a +# whole-token match ('Jack M.A.', 'John Smith Ph.D.', 'Smith, +# A.B.C.'), a surviving chunk claim ('Doe, John Msc.Ed.'), a leading +# dotted run ('X.Y.Z. Smith', 'J.R.R. Tolkien', 'A.B. Vajpayee'), the +# single trailing period that is no shape at all ('John Smith Xyz.'), +# and a digit chunk, which the alphabetic gate refuses ('Bridge (1.4)'). +# +# Literal-anchored to the measured movers, one set for all five +# ledgers, for the reason the fix(#289) rule above gives: the class is +# a SHAPE, and a regex for the shape would claim every dotted token +# the vocabulary already answers for. +name_regex = "^(?:Jack X\\.Y\\.I\\.|John Smith B\\.Tech\\.|John Smith C\\.H\\.A\\.|John Smith E\\.S\\.Q\\.|John Smith Q\\.W\\.E\\.R\\.T\\.|John Smith X\\.Y\\.Z\\.|John Smith, A\\.B\\.|Smith, E\\.S\\.Q\\.|john smith x\\.y\\.z\\.)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the ambiguous credential class reports at slots that were silent" +# Every decision at the trailing suffix slot, the post-comma given +# slot and a tail segment now reports the EXISTING SUFFIX_OR_NAME -- +# no new kind, and the report tracks the FORK BEING CONSULTED rather +# than which way it went, as the trailing slot always has. These names +# gain it with no role moving at any baseline: the fork was there all +# along and was called silently. It is the first time the comma's +# OWN decision is reported (rules.md#C1); P6's attachment fork has +# reported on a family-comma path since 2.3 ('Berg, Jan vd'). C2's +# structural flag was already the first comma-path report of a shape. +# +# 'Steven Hardman, MD, DO, DDS' goes the other way and LOSES its +# comma-structure flag: its third segment now reads as the credential +# run it is, because 'DO' leans credential in a mixed-case name. That +# is the one place this arc quiets a report rather than adding one. +# +# `_ambiguities` alone, which keeps `_initials` out of these rows as +# always (#484), and the rule cannot absorb a role diff on any of +# these names for the same reason. Literal-anchored, one set for all +# the 2.x ledgers; the 1.4.0 ledger has no twin, `_ambiguities` being +# a v2 surface that cannot enter a diff below baseline 2.0. +name_regex = "^(?:J\\.A\\. K\\.D\\.|Jack X\\.Y\\.Z\\.|John Smith J\\.u\\.n\\.i\\.o\\.r\\.|John Smith R\\.A\\.I\\.|Royce, Ed|Smith Jr\\., A\\.B\\.|Smith, A\\.B\\.|Smith, Ma|Steven Hardman, MD, DO, DDS)$" +fields = ["_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the glued CJK honorific peel reads the case lean" +# The suffix-run predicate the peel uses to decline a post-comma run +# (rules.md#W3, asking C1's own vocabulary question) now reads the +# same written-case evidence rule S2 reads, because `one_case` is +# passed to it. Before, one name written with two credentials divided +# two ways: a Kim-prefixed 김민준씨 input peeled its honorific when the +# credential was 'PhD' and did not when it was 'MA', the only +# difference being that 'MA' is an ambiguous acronym whose case lean +# the predicate was not given. +# +# Every name here is a COMPOSED form -- a Latin wrapper and a family +# comma around a CJK name -- which no East Asian writing system +# produces, so all three are `tolerated` case rows on the radar tier +# (decisions.md#cjk-comma-demotion). They are classified all the same: +# a radar diff this arc INTENDED is one a release note may be written +# from, and an unclassified one is not. +# +# `title` is in `fields` because 'Dr. 김민준씨, MA' had the whole +# string for its family at these baselines, the title included; at +# 2.0.0 and 2.1.0 'Jo김민준씨, DO' moves `title` instead of `given`, +# and the union is what a per-baseline `fields` records (#452). +name_regex = "^(?:Dr\\. 김민준씨, MA|Jo김민준씨, DO|Kim김민준씨, MA)$" +fields = ["title", "given", "family", "suffix", "_ambiguities"] +orders = ["DEFAULT"] diff --git a/tools/differential/expected_since_2.3.0.toml b/tools/differential/expected_since_2.3.0.toml index e4205a8a..8ae60026 100644 --- a/tools/differential/expected_since_2.3.0.toml +++ b/tools/differential/expected_since_2.3.0.toml @@ -173,3 +173,161 @@ issue = "fix(#383/#479) a bare capital connective in an all-upper name stops ini name_regex = "^JUAN Y GARCIA$" fields = ["_initials"] orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289) a written case contrast decides a bare ambiguous acronym" +# A member of suffix_acronyms_ambiguous written in CAPITALS inside a +# mixed-case name is written the way a credential is written, and is +# taken as one with nothing to spare ('Jack MA' -> given 'Jack', +# suffix 'MA'); one written in any other cased form is written the way +# a surname is written and stays one even with words to spare ('John +# Smith Ma' -> middle 'Smith', family 'Ma'). The same lean reaches the +# post-comma given slot ('Smith, MA' -> family 'Smith', suffix 'MA'; +# 'Smith, Ma' keeps its given), and the comma form's words-to-spare +# test becomes a count of NAME words, which is case-blind -- so +# 'John Smith, MA', 'John Smith, Ed', 'JOHN SMITH, MA' and +# 'john smith, ma' all read the credential where the token count read +# a given name. See rules.md#S2, rules.md#C1 and rules.md#P2. +# +# A name written wholly in one case carries no contrast and does not +# move on the LEAN at all -- 'JOHN SMITH MA', 'ANH DO', 'anh van do' +# and 'Jack Ma' are _MUST_NOT_MATCH probes in +# tests/v2/test_ledger_guards.py. The one-case comma forms in the list +# above are here on the NAME-word count instead, which asks nothing +# about case; that is why they sit beside the lean's own names rather +# than in a rule of their own. +# +# Accepted costs, both recorded rather than repaired: the walk stops +# at a declined pick instead of continuing past it, so a genuine +# suffix standing in front of a name-leaning acronym is never reached +# ('abdul Smith Jr Ma' reads middle 'Jr'), and the lean ends P2's +# particle chain with no words to spare where the count would not +# ('Freiherr von Berg MA', whose own rule further up this file carries +# the reading at 1.4.0). +# +# The Title-case half is witnessed at the comma by two minimal pairs +# rather than argued: 'John Smith, MA'/'John Smith, Ma' both read the +# credential, because two NAME words before the comma decide before +# the case is looked at, and 'Smith Jr., MA'/'Smith Jr., Ma' split, +# because one name word leaves the case to decide and Title-case +# declines. 'Smith Jr., Ma' diffs here for a reading this change did +# NOT move -- the pre-comma restructure v2 has done since 2.0, at +# 1.4.0, and the fork's new report at the 2.x baselines -- while +# 'John Smith, Ma' moves on the COUNT, as its all-caps twin does, and +# has no 1.4.0 diff at all, v1 having read the credential there too. +# Both sit on this rule because the rule is what states which way +# each member of a pair goes. +# +# Literal-anchored to the measured movers, and ONE SET FOR THE FOUR +# 2.x LEDGERS so the rule says the same thing at each of them. The +# 1.4.0 copy differs by exactly one member and holds 'John Prof. MA' +# out: 1.4.0 read a trailing period-marked title as a title itself, +# so that name's diff there belongs to fix(#316)'s rule and not to +# this one. "One set for all five" was written before that carve-out +# existed and was false in these four copies from the day it did +# (2026-09-18). The set is literal because the class is +# wide (every mixed-case name whose trailing piece is a listed +# ambiguous acronym, plus every comma form with two name words before +# the comma) and a regex for the class would claim the one-case +# spellings that do not move. A name here that does not diff at a +# given baseline is simply one the reading already agreed with there. +# `fields` is per-baseline, the union the run at THAT baseline +# measures (#452). +name_regex = "^(?:Davis Royce, Ed|Freiherr von Berg MA|JOHN SMITH, MA|Jack MA|Jack MA\\.|Jack Wei Ma|John Prof\\. MA|John Smith Ma|John Smith, Ed|John Smith, MA|John Smith, Ma|John de Ma|John van der Berg Ma|Smith Jr\\., MA|Smith Jr\\., Ma|Smith, MA|abdul Smith Berg Ma|abdul Smith Jr Ma|abdul Smith Ma|john smith, ma)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#516) an unlisted dotted acronym is read by position" +# A token of two or more period-separated chunks that no vocabulary +# claims joins the ambiguous credential class by SHAPE and is read by +# POSITION, exactly as a bare listed acronym is: a credential where +# the name has words to spare, a name word where it does not, either +# reading reported (rules.md#S3). Case says nothing here -- the +# periods are the evidence -- which is why 'john smith x.y.z.' moves +# with 'John Smith X.Y.Z.'. The same admission reaches the comma form, +# where the NAME-word count decides ('John Smith, A.B.' -> suffix +# 'A.B.'; 'Smith, A.B.' keeps its given and only reports). +# +# Two names here restore a reading a PRIOR bundle's vocabulary change +# took away, by a different route than it lost it: 'John Smith E.S.Q.' +# and 'John Smith C.H.A.' fell to a bare positional read when 'esq' +# and 'cha' left SUFFIX_ACRONYMS +# (decisions.md#suffix-acronym-collisions), and the shape class reads +# them as the credentials they always were with no wordlist entry. +# At the baselines where +# that restoration lands on the baseline's own reading, only the new +# report is left and `fields` says so. +# +# The narrow half of the same change: a chunk match no longer makes a +# dotted token a suffix where EVERY matched chunk is a single ASCII +# character, which reaches only the roman numerals -- 'Jack X.Y.I.' +# moves to family for that reason and not for the shape one. +# +# The shapes the vocabulary settles are untouched and are +# _MUST_NOT_MATCH probes in tests/v2/test_ledger_guards.py: a +# whole-token match ('Jack M.A.', 'John Smith Ph.D.', 'Smith, +# A.B.C.'), a surviving chunk claim ('Doe, John Msc.Ed.'), a leading +# dotted run ('X.Y.Z. Smith', 'J.R.R. Tolkien', 'A.B. Vajpayee'), the +# single trailing period that is no shape at all ('John Smith Xyz.'), +# and a digit chunk, which the alphabetic gate refuses ('Bridge (1.4)'). +# +# Literal-anchored to the measured movers, one set for all five +# ledgers, for the reason the fix(#289) rule above gives: the class is +# a SHAPE, and a regex for the shape would claim every dotted token +# the vocabulary already answers for. +name_regex = "^(?:Jack X\\.Y\\.I\\.|John Smith B\\.Tech\\.|John Smith C\\.H\\.A\\.|John Smith E\\.S\\.Q\\.|John Smith Q\\.W\\.E\\.R\\.T\\.|John Smith X\\.Y\\.Z\\.|John Smith, A\\.B\\.|Smith, E\\.S\\.Q\\.|john smith x\\.y\\.z\\.)$" +fields = ["family", "given", "middle", "suffix", "_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the ambiguous credential class reports at slots that were silent" +# Every decision at the trailing suffix slot, the post-comma given +# slot and a tail segment now reports the EXISTING SUFFIX_OR_NAME -- +# no new kind, and the report tracks the FORK BEING CONSULTED rather +# than which way it went, as the trailing slot always has. These names +# gain it with no role moving at any baseline: the fork was there all +# along and was called silently. It is the first time the comma's +# OWN decision is reported (rules.md#C1); P6's attachment fork has +# reported on a family-comma path since 2.3 ('Berg, Jan vd'). C2's +# structural flag was already the first comma-path report of a shape. +# +# 'Steven Hardman, MD, DO, DDS' goes the other way and LOSES its +# comma-structure flag: its third segment now reads as the credential +# run it is, because 'DO' leans credential in a mixed-case name. That +# is the one place this arc quiets a report rather than adding one. +# +# `_ambiguities` alone, which keeps `_initials` out of these rows as +# always (#484), and the rule cannot absorb a role diff on any of +# these names for the same reason. Literal-anchored, one set for all +# the 2.x ledgers; the 1.4.0 ledger has no twin, `_ambiguities` being +# a v2 surface that cannot enter a diff below baseline 2.0. +name_regex = "^(?:J\\.A\\. K\\.D\\.|Jack X\\.Y\\.Z\\.|John Smith J\\.u\\.n\\.i\\.o\\.r\\.|John Smith R\\.A\\.I\\.|Royce, Ed|Smith Jr\\., A\\.B\\.|Smith, A\\.B\\.|Smith, Ma|Steven Hardman, MD, DO, DDS)$" +fields = ["_ambiguities"] +orders = ["DEFAULT"] + +[[change]] +issue = "fix(#289/#516) the glued CJK honorific peel reads the case lean" +# The suffix-run predicate the peel uses to decline a post-comma run +# (rules.md#W3, asking C1's own vocabulary question) now reads the +# same written-case evidence rule S2 reads, because `one_case` is +# passed to it. Before, one name written with two credentials divided +# two ways: a Kim-prefixed 김민준씨 input peeled its honorific when the +# credential was 'PhD' and did not when it was 'MA', the only +# difference being that 'MA' is an ambiguous acronym whose case lean +# the predicate was not given. +# +# Every name here is a COMPOSED form -- a Latin wrapper and a family +# comma around a CJK name -- which no East Asian writing system +# produces, so all three are `tolerated` case rows on the radar tier +# (decisions.md#cjk-comma-demotion). They are classified all the same: +# a radar diff this arc INTENDED is one a release note may be written +# from, and an unclassified one is not. +# +# `title` is in `fields` because 'Dr. 김민준씨, MA' had the whole +# string for its family at these baselines, the title included; at +# 2.0.0 and 2.1.0 'Jo김민준씨, DO' moves `title` instead of `given`, +# and the union is what a per-baseline `fields` records (#452). +name_regex = "^(?:Dr\\. 김민준씨, MA|Jo김민준씨, DO|Kim김민준씨, MA)$" +fields = ["title", "given", "family", "suffix", "_ambiguities"] +orders = ["DEFAULT"]