feat(rest): access table data with the temporary credentials issued by the REST catalog

[lucasfang]

Purpose

Linked issue: close #369

Let the REST catalog drive table-data IO with the per-table temporary credentials it issues (data tokens, e.g. DLF) instead of the static credentials in the catalog options. When data-token.enabled=true, Catalog::GetTableFileSystem(identifier) returns a FileSystem that reloads the table's credentials before they expire by rebuilding its delegate keyed by the issued token. ReadContextBuilder, ScanContextBuilder, WriteContextBuilder and CommitContextBuilder gain WithCatalog(catalog, identifier) to resolve the table's schema and file system from the catalog in one call. A generic CredentialProvider + CredentialProviderFactory extension point is added for callers that bring their own FileSystem; built-in file systems keep signing with the static credentials of their own options. Catalog::Create takes an optional fs_scheme_to_identifier_map so a catalog that resolves several URI schemes keeps routing each scheme to its backend, including the delegate file systems it rebuilds from per-table data tokens.

Tests

Unit tests: rest_credential_provider_test, rest_token_file_system_test, rest_catalog_test, rest_messages_test, resource_paths_test, dlf_auth_test, credential_provider_factory_test. Context and operation coverage: read_context_test, scan_context_test, table_read_test, table_scan_test, file_store_commit_test, file_store_write_test, system_table_test, format_table_test, file_system_catalog_test. Verified locally: paimon-rest-test (150 tests) and paimon-fs-test (69 tests) pass; pre-commit and git diff --check are clean.

API and Format

Additive public API under include/: new headers paimon/fs/credential_provider.h and paimon/fs/credential_provider_factory.h; CredentialProvider::MergeOptionsWithCredentials; Catalog::GetTableFileSystem; an optional trailing fs_scheme_to_identifier_map parameter on Catalog::Create; CatalogOptions::DATA_TOKEN_ENABLED and CatalogOptions::DLF_OSS_ENDPOINT; WithCatalog on the read/scan/write/commit context builders. Every new parameter is optional and defaults preserve the current behavior. No storage format or protocol change. data-token.enabled defaults to false, so existing callers are unaffected.

Documentation

docs/source/user_guide/catalog.rst updated: reading, scanning, writing and committing through the catalog with WithCatalog, and a new "Authenticating with credentials of your own" section covering the CredentialProvider / CredentialProviderFactory path for a caller-supplied file system.

Generative AI tooling

Generated-by: Qoder

[JingsongLi]

Source-level review focused on behavioral parity with Java RESTTokenFileIO (apache/paimon@d77efe0). Local runtime validation could not be completed.

[lucasfang]

Hi @JingsongLi @SteNicholas @lszskye, thank you all for the thorough reviews. I've addressed the review comments in the latest updates, and the branch is ready for another pass. Could you please take a look when you have time and let me know if anything else needs attention? Thanks!

[JingsongLi]

Re-reviewed 19d2189, focusing on consistency with the Java REST implementation. The three findings from my previous review are addressed: token-keyed backend reuse, token-only refresh, and rejecting missing/null token responses. Two table-IO integration gaps remain below. Validation: all 28 changed C++ translation units passed syntax checks using the existing dependency headers, and git diff --check passed. Runtime tests were not run locally.

[JingsongLi]

[P2] Keep external tables on their catalog-configured filesystem

Please preserve the external-table distinction before constructing this wrapper. Java's CatalogUtils.loadTable() selects externalFileIO for external tables, and RESTCatalog.getTable() supplies the catalog-configured filesystem for that path. Here GetTableResponse::IsExternal() is never consulted, so an external native table opened through WithCatalog() with data-token.enabled=true requests /token before accessing storage. If the server only issues tokens for internal tables, the read fails even though the catalog storage credentials can access the external location. Please retain the external flag when resolving table IO and return the configured filesystem for external tables; a test whose external-table token endpoint rejects requests would cover this difference.

[JingsongLi]

[P2] Attach the token filesystem when loading internal format tables

Please also wire the table-scoped filesystem into RestCatalog::LoadFormatTable(). That method still calls FormatTable::Create(fs_, ...), and format-table scans, reads and writes use that stored catalog-wide filesystem. With data-token.enabled=true and storage access available only through the table token, the documented ReadContextBuilder(format_table) / ScanContextBuilder(format_table) paths therefore fail without ever requesting a token; adding WithCatalog() is rejected here. Java selects the internal/external FileIO before dispatching to the format-table loader, so internal format tables receive the refreshing token filesystem as well. Please select and store the appropriate filesystem during format-table loading, respecting the external-table distinction, and cover an internal format table whose storage requires its token.

[JingsongLi]

Approving this change. The external-table credential handling and internal format-table token filesystem integration noted in my latest review can be addressed in follow-up work; they are non-blocking for this PR.

[lszskye]

+1

[JingsongLi]

+1