fix: preserve exact numeric literals and restore safe AST constant folding

[bkkpro1980]

Summary

This PR resolves IEEE-754 precision loss and representation drift for numeric constants while restoring compile-time AST constant folding for safe expressions.

Background & Problem

1. Precision Loss on Unparsing: unparser.lua previously reconstructed numeric literals using tostring(val), which could lose significant digits when converting parsed numeric values back into source text. This affected exact integers near the 53-bit boundary (2^53 - 1) and high-precision decimal floats, while large hexadecimal constants were re-emitted in lossy or decimal forms.

2. Large Hexadecimal Literal Parsing: Relying on tonumber(source, 16) could parse large hexadecimal literals incorrectly on affected 32-bit environments.

3. Premature / Unsafe Constant Folding: Disabling constant folding globally via simplify = false prevented precision loss, but also disabled standard compile-time simplifications such as 2 + 3, not false, string concatenation, and condition pruning. Conversely, unconstrained folding could evaluate 32-bit cryptographic multiplications such as 0x811C9DC5 * 0x01000193 using host doubles, potentially corrupting the value before 32-bit modulo extraction.

Changes

1. Source Provenance for Number Literals (NumberExpression.raw):

   • Updated Ast.NumberExpression to store the original token slice (raw).
   • Parser:expressionLiteral now passes tk.source to Ast.NumberExpression.
   • unparser.lua emits expression.raw directly when available. For Lua51 targets, digit separators (_) are stripped and binary literals (0b...) fall back to canonical numeric formatting; for LuaU targets, the original literal is emitted verbatim.
   • Synthesized constants have raw = nil and use canonical %.0f formatting for safe integers and %.17g for other numeric values.

2. Tokenizer Hardening (tokenizer.lua):

   • Replaced tonumber with digit-accumulation parsers (parseHex, parseBinary) in Tokenizer:number to avoid platform-dependent integer parsing behavior.
   • Added validation for malformed hexadecimal and binary literals such as 0x and 0b.

3. Safe AST Constant Folding (ast.lua):

   • Restored compile-time constant folding for operations whose operands and results can be represented safely under IEEE-754 integer precision.
   • Arithmetic operations (+, -, *, /, %, unary -, ^) only fold when operands and the result are safe integers.
   • Division only folds for exact integer quotients (lhs % rhs == 0) with non-zero divisors; dyadic fractions and inexact divisions remain as AST expressions.
   • Expressions exceeding the safe integer range remain as runtime AST trees instead of being folded using potentially lossy host arithmetic.
   • Ast.OrExpression and Ast.AndExpression now return the selected operand AST node directly, preserving node identity, .raw provenance, and Lua operand-return semantics.
   • Split constant comparison guards into canCompareEqualityConstants and canCompareOrderConstants to prevent folding comparisons that would error at runtime in Lua (such as 1 < "2") and prevent unsafe equality folds on large numeric values.
   • Restored compile-time constant folding for safe integer arithmetic, pure string concatenations (..), string length (#), and logical not.

4. Regression Test Suite (tests/numeric-precision.lua):

   • Added a standalone test suite covering 32-bit boundaries (0x100000000), maximum safe integers (0x1FFFFFFFFFFFFF), decimal boundary arithmetic at 2^53, safe arithmetic folding, large cryptographic multiplication preservation, dyadic fractions, and/or semantics, boundary comparisons, and 16-digit float preservation.

Verification

Ran the full test suite across all built-in presets (Medium, Minify, Strong, Vmify, Weak) on both lua5.1 and luau inside Docker:

./scripts/run-tests.sh -n 10
# Total: 17 passed, 0 failed, 17 total
# === ALL TESTS PASSED ===

[levno-710]

I found two issues that I think need fixing before this merges:

1. The new parseHex accumulates into a double and can round a large literal differently from Lua itself. For example, print(string.format("%.0f", 0x2944075f548d8d9)) prints 185844359999576288 on both Lua 5.1 and Luau, but the output produced with the Vmify preset prints 185844359999576256. Vmify rebuilds the number node without its raw text, so preserving the spelling in the unparser does not cover this path. Could you fix the conversion and add an end-to-end case through Vmify?

2. Test 7 in tests/numeric-precision.lua has an incorrect expected result. It prints TEST 7 FAILED on both Lua 5.1 and Luau. Please correct the expected result.

[bkkpro1980]

I read your comment just now and I'll look into it. Thank you

[bkkpro1980]

I think the issues have been fixed. I ran tests and it came out fine. I also tested obfuscation on some of my Lua scripts and those seem to work as intended after the fixes.