Skip to content

gh-155742: Optimize json float parsing - #156897

Open
vstinner wants to merge 2 commits into
python:mainfrom
vstinner:json_float
Open

vstinner wants to merge 2 commits into
python:mainfrom
vstinner:json_float

Conversation

@vstinner

@vstinner vstinner commented Sep 3, 2026

Copy link
Copy Markdown
Member

Add internal _PyFloat_FromString(): similar to PyFloat_FromString(),
but don't require a Python object.

Use _PyFloat_FromString() in _json to avoid creating a temporary bytes objects.

_Py_string_to_number_with_underscores() can now be called with NULL object and float_from_string_inner() can now be called with NULL data; if needed they create a temporary bytes objects to format the error message.

Similar to PyFloat_FromString(), but don't require a Python object.

Use _PyFloat_FromString() in _json to avoid creating a temporary
bytes objects.

_Py_string_to_number_with_underscores() can now be called with NULL
object and float_from_string_inner() can now be called with NULL
data; if needed they create a temporary bytes objects to format the
error message.
@vstinner vstinner changed the title gh-155742: Add internal _PyFloat_FromString() gh-155742: Optimize json float parsing Sep 3, 2026
@vstinner

vstinner commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

Benchmark:

import pyperf
import json
import random

NUMBERS = 10 ** 4
ITEMS = [str(random.random()) for _ in range(NUMBERS)]
DATA = '[' + ', '.join(ITEMS) + ']'

runner = pyperf.Runner()
runner.bench_func('json.loads', json.loads, DATA)

Result: [ref] 2.91 ms +- 0.34 ms -> [change] 2.49 ms +- 0.06 ms: 1.17x faster.

Fix the compiler warning:

    {'file': 'Objects/floatobject.c', 'line': '174', 'column': '5',
    'message': 'label followed by a declaration is a C23 extension',
    'option': '-Wc23-extensions'}
@eendebakpt

Copy link
Copy Markdown
Contributor

@vstinner This overlaps with #150639, I still have to loop at the differences.

@sprajs sprajs left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 68ecdf378bb7e403f5a9dcd935933c1dcd45293f against merge base 1620e0f1b59c2078e33d82bef75df3db3c651d1c.

No correctness regression reproduced in 5,252 C-versus-Python-scanner cases, 24 exact-token custom-hook checks, and 400 allocation-failure offsets. These cover 19/20-digit and signed/unsigned boundaries, negative zero, subnormal/overflow cases, exponent extremes, long buffers, 1/2/4-byte Unicode inputs and conversion limits. Exact debug base/head builds pass 328 JSON/float/complex tests (4 skipped); head also passes -R 3:3 on all three suites.

I compared the two overlapping proposals using one PGO/LTO core built from #156897's head. The baseline _json.c sources are identical; each of baseline, #156897 and #150639 was compiled with the same -O3 flags and headers, without module-specific PGO. This isolates the decoder components:

Workload base 156897 150639
float_array_10000 1.96 ms 1.89 ms 1.75 ms
numeric_records_2000 1.65 ms 1.63 ms 1.52 ms
pyperformance_json_loads 20.00 µs 19.88 µs 19.00 µs

Inputs: 10,000 successive random() calls on one Random(156897) instance (the float-array workload from #156897's discussion, with a fixed seed); 2,000 records with integer IDs, two floats and a boolean; and the three-object pyperformance json_loads fixture. The latter is timed with its original 20-call unrolling and normalization. #156897 still allocates the intermediate character buffer; #150639 additionally provides integer and stack-buffer fast paths.

Separately, exact native PGO/LTO builds of this PR's base and head gave:

Workload base head
float_array_10000 1.94 ms 1.92 ms
numeric_records_2000 1.72 ms 1.69 ms
pyperformance_json_loads 22.65 µs 22.45 µs

The native float-array medians give 1.01× here, with overlapping observed ranges (1.91 ms–2.00 ms base, 1.90 ms–1.92 ms head). I did not reproduce the reported 1.17× improvement in this setup.

Timings are medians from seven interleaved fresh processes on Linux x86-64, i9-9900K/GCC 16.2.1, pinned to one logical CPU; GC disabled during timed loops, hash seed 0, calibration to 65 ms followed by a separately timed batch. Builds and tests had finished before timing; the desktop was not frequency-isolated. These are workload-specific measurements, not application-wide speedups.

Review and testing performed with Codex (AI assistance).

@eendebakpt

Copy link
Copy Markdown
Contributor

@vstinner I make an update to #150639. It now also avoids mutation of bytes objects, but is more performant than this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants