Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,7 @@ Don't use the bare `python3 -m doctest <file>.rst` CLI (no `optionflags`) to che

**Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100.

**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`), there being no trailing shape rule. The bare word is where "positional" still holds whole, and the reason it must: `TITLES` holds words that are in no suffix set — 746 of them, measured on this tree with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (decisions.md#P5's trailing-position bullet, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration.
**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules.md#S3`). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole, and the reason it must: `TITLES` holds words that are in no suffix set — 746 of them, measured on this tree with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (decisions.md#P5's trailing-position bullet, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration.

### Tests (`tests/`)

Expand Down
69 changes: 60 additions & 9 deletions docs/customize.rst
Original file line number Diff line number Diff line change
Expand Up @@ -202,13 +202,27 @@ they narrow how an existing
entry is read when it appears alone. If you're not sure whether a word
you're adding is one of these ambiguous cases, weigh how often it is a
name against how often it is the credential. Marking it ambiguous is
not free in either direction: an ambiguous acronym counts as a suffix
only when written with its periods, so the bare spelling reads as a
name and the parse reports the fork, and the comma form moves with it
— ``Smith, BA`` reads first ``BA`` rather than suffix ``BA``, and a
bracketed ``John Smith (BA)`` falls through to nickname parsing. A
wrong unambiguous claim takes the credential reading silently and can
lose a real person's surname. For ``particles_ambiguous`` the default
not free in either direction: written with its periods, an ambiguous
acronym counts as a suffix unambiguously; bare, the reading now
depends on the writing itself, so a bracketed ``John Smith (BA)``
falls through to nickname parsing and either bare reading reports the
fork. Written in ALL CAPITALS inside a mixed-case name it counts as
a suffix even with no words to spare (the credential lean); written
Title-case there it stays the surname even WITH words to spare (the
surname lean); lacking either signal — an all-lower spelling in a
mixed-case name, or any spelling in a name written wholly in one
case — the reading falls back to whether the name has two or more
words before it. At a comma the count of NAME words before it decides
FIRST, and the case is read only where the count leaves the word a
name: ``John Smith, Ba`` reads suffix ``Ba`` on the count alone (two
name words before the comma), Title-case or not, while ``Smith, BA``
reads suffix ``BA`` on the CAPITALS lean, one word before the comma
being all the count needs to leave for the lean to promote. What
still reads as the given name is ``Smith, Ba`` (one word, and
Title-case carries no credential lean to promote it) and
``smith, ba`` (one word, one case, no lean at all). A wrong
unambiguous claim takes the credential reading
silently and can lose a real person's surname. For ``particles_ambiguous`` the default
runs the other way: a particle that is not borne as a given name
belongs in the never-given half, which is where ``mc`` and ``ste``
were moved (#360). The other
Expand All @@ -225,15 +239,19 @@ that swallowed it as a title would misparse "Dean Martin" for
everyone.)

``ma`` is a shipped example. It is both a credential and a common
surname, so it is listed in ``suffix_acronyms_ambiguous`` and counts as
a suffix only when written with periods:
surname, so it is listed in ``suffix_acronyms_ambiguous``: written
with periods it counts as a suffix unambiguously, and bare it takes
the case reading above -- Title-case stays the surname, capitals lean
the credential:

.. doctest::

>>> parse("Jack Ma").family
'Ma'
>>> parse("Jack M.A.").suffix
'M.A.'
>>> parse("Jack MA").suffix
'MA'

``particles_ambiguous`` is the same idea for surname particles. A
particle listed there may also be a given name, which is what makes a
Expand Down Expand Up @@ -428,6 +446,39 @@ listed below.
family ``田中さん``, given ``V.`` when ``False`` — though a
comma around a CJK name is tolerated input
(``rules.md#W3``) and this reading can change.
* - ``unlisted_dotted_suffixes``
- ``bool``
- Reads an unlisted token of two or more period-separated chunks
as a credential where the position allows it:
``"John Smith X.Y.Z."`` gives suffix ``X.Y.Z.`` while
``"Jack X.Y.Z."`` keeps family ``X.Y.Z.``, and either reading
is reported. Case is irrelevant — the periods are the signal.
Whole-token vocabulary still wins (``M.A.``, ``Ph.D.``), and a
single trailing period is not this shape
(``"John Smith Xyz."`` keeps family ``Xyz.``). Two further
gates keep it from over-reaching: every chunk must be
alphabetic, so a digit anywhere refuses it
(``"John Smith 1.4"`` keeps family ``1.4``, on or off), and a
script with no period abbreviations of its own refuses it too
(a CJK word glued into periods, ``"John Smith 田.中."``, keeps
family ``田.中.``). Defaults to ``True``; ``False`` reads such a
token as name material everywhere and still reports the fork —
it does NOT revive the pre-2.4 reading of a chunk that is a
single ASCII character — a roman numeral, or the digit ``2`` —
as a credential (``"Jack X.Y.I."`` still keeps family
``X.Y.I.`` either way, and a dotted version string such as
``"John Smith 1.4.2"`` keeps family ``1.4.2``; that retirement
is not behind this switch).
* - ``unlisted_caps_suffixes``
- ``bool``
- Reads an unlisted all-caps word of two or more letters, with no
period in it, in a name written in more than one case as a
credential where the position allows it: ``"John Smith XYZ"``
gives suffix ``XYZ``. Defaults to ``False``, and deliberately:
an all-caps surname is a real writing convention that shape
cannot separate from a credential, so ``"Jean Pierre DUPONT"``
gives family ``Pierre``, suffix ``DUPONT`` with this on. Off,
nothing changes and nothing is reported.
* - ``strip_emoji``
- ``bool``
- Excludes emoji from tokenization — they appear in no field or
Expand Down
Loading
Loading