Skip to content

Go codegen for ClickHouse, DuckDB, Spanner and SQL Server under database/sql - #4621

Open
kyleconroy wants to merge 15 commits into
mainfrom
claude/database-sql-support-research-hbmwsa
Open

Go codegen for ClickHouse, DuckDB, Spanner and SQL Server under database/sql#4621
kyleconroy wants to merge 15 commits into
mainfrom
claude/database-sql-support-research-hbmwsa

Conversation

@kyleconroy

@kyleconroy kyleconroy commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

The Go generator already ran for the four engines the analysis core handles, but knew none of their types: every field came out as any, and the queries passed arguments positionally against @name placeholders. This makes the generated code typed and runnable with each engine's database/sql driver.

Plugin request carries the type expression. codegen.proto gains a recursive TypeExpr message and a type_expr field on Column, populated from the core analyzer's tree. It is only set when the core typed the column; the legacy path leaves it unset and the flat fields stay as they were, so existing plugins keep working. Model columns come from the catalog, which the core path flattened, so catalog.Column carries the tree too. The core path and the live-database analyzer never meet: NewCompiler returns after initCore before any analyzer is built.

A type mapper per engine reads the tree rather than the flat name, which is what lets it see the element of an array, the key and value of a map, the fields of a struct, and the type LowCardinality or SimpleAggregateFunction wraps. Each maps to what the engine's driver hands back:

  • ClickHouse (clickhouse-go/v2): sized integers, *big.Int, shopspring decimal.Decimal, uuid.UUID, net.IP, orb.* geometries; nested nullables as pointers, so Array(Nullable(String)) is []*string and Map(String, Nullable(UInt8)) is map[string]*uint8.
  • DuckDB (duckdb-go/v2): a list column is duckdb.Composite[[]T], which decodes the driver's []any; the same list as a parameter is the plain slice, since a Composite is not accepted as an argument. duckdb.Decimal, duckdb.Interval, duckdb.Map; structs as map[string]any.
  • Spanner (go-sql-spanner): sql.Null* scalars, arrays as slices of the spanner.Null* types the driver decodes by default, big.Rat/spanner.NullNumeric, civil.Date/spanner.NullDate, spanner.NullJSON.
  • SQL Server (go-mssqldb): integers by width, exact numerics as string, temporals as time.Time, mssql.UniqueIdentifier, binary and spatial types as []byte.

Named arguments. When every parameter of a query is bound by name (@name for SQL Server and Spanner, {name:Type} for ClickHouse) each is passed as sql.Named. pq.Array wrapping is limited to PostgreSQL under database/sql, since the other drivers scan slices directly.

Examples in their own module. examples/ is now a Go module that requires sqlc through a replace, so the drivers the examples run against are its dependencies alone (sqlc's go.mod loses them, and lib/pq, which only the examples imported) and cgo stays on for the DuckDB driver. An authors example for ClickHouse, DuckDB, SQL Server and Spanner sits beside the existing ones. Test helpers in examples/internal/local create a database per test on the server named by CLICKHOUSE_SERVER_URI, MSSQL_SERVER_URI or SPANNER_SERVER_URI and skip otherwise.

CI. ClickHouse and SQL Server run as job services. sqlc-test-setup installs Spanner Omni from its standalone server release (pinned SHA-256) and starts a single server on port 15000, so the Spanner example runs too. The end-to-end module builds with cgo for the DuckDB golden, and the examples module is tested from its own directory. docker-compose gains a ClickHouse service beside the SQL Server and Spanner Omni ones.

Verified

  • Full base replay corpus, TestExamples, TestFormat, vet and gofmt pass; no existing golden changed.
  • internal/endtoend/testdata compiles with the drivers in its go.mod, including the DuckDB case under cgo.
  • The DuckDB example runs end to end in-process; the ClickHouse example runs end to end against a local ClickHouse 25.8 server; the SQL Server example ran against the CI service on an earlier push.
  • sqlc's own module cross-compiles for darwin and windows with CGO_ENABLED=0.

Not verified locally: the Spanner example and the Omni install/start in sqlc-test-setup. The Omni server opens IPv6 sockets at startup, which the sandbox this was written in refuses, so CI is its first real run.

Left for later

  • Enums and alias types (DuckDB CREATE TYPE ... AS ENUM, SQL Server CREATE TYPE ... FROM) are not in the core-path catalog and map to any.
  • coreResultCatalog reports public as every engine's default schema.
  • sqlc-gen-go needs the same proto field before it can read the tree.

Cost to note

The Spanner Omni server release is about 276 MB per CI run, and the DuckDB driver's linux/amd64 bindings about 30 MB, paid by the test job.

🤖 Generated with Claude Code

https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT

claude added 15 commits September 17, 2026 05:26
@claude
plugin: carry the analysis core's type expression on Column
5802484 
The core analyzer types every column and parameter as an expression: a
name, whether a value may be null, and nested arguments, so
decimal(10, 2), array(array(integer)) and struct(x: integer) survive
intact. The plugin request flattened that to a single type name plus
not_null, is_array, array_dims, length and unsigned, which is not enough
for a type mapper to tell nvarchar(max) from nvarchar(50) or to see the
element of an array or the fields of a struct.

Add a recursive TypeExpr message to the plugin proto and a type_expr
field on Column, populated from the core's tree in the shim. It is only
set when the analysis core typed the column; the legacy path leaves it
unset and the flat fields stay as they were, so existing plugins keep
working. When a database analyzer overrides a column's flat type in
combineAnalysis, the tree no longer describes the column and is dropped.

A DuckDB codegen_json case pins the shape with decimal arguments, nested
arrays, a labeled struct, a map and a sized array. The existing JSON
request goldens gain the new field, unset, on every column.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
codegen/golang: map ClickHouse, DuckDB, Spanner and SQL Server types
32194c9 
The Go generator ran for the four engines the analysis core handles but
knew none of their types, so every field came out as any. Add a mapper
per engine that reads the column's type expression rather than the flat
name, which is what lets it see the element of an array, the key and
value of a map, the fields of a struct, and the type LowCardinality or
SimpleAggregateFunction wraps. Each maps to the Go types the engine's
database/sql driver hands back: shopspring decimals, orb geometries and
pointer-typed nullable elements for clickhouse-go; duckdb.Composite,
duckdb.Decimal, duckdb.Interval and duckdb.Map for duckdb-go, with a
list parameter typed as the slice itself since a Composite is not
accepted as an argument; the spanner package's Null types for
go-sql-spanner, whose arrays decode into them; and
mssql.UniqueIdentifier for go-mssqldb.

The tree-reading mappers render the whole type, arrays included, so
goType leaves the dimensions to them. Model columns come from the
catalog, which the core path flattened, so catalog.Column now carries the
expression too and the shim forwards it. The queries for these engines
pass slices straight to the driver, so pq.Array wrapping is limited to
PostgreSQL under database/sql, where lib/pq needs it. Import detection
follows a package qualifier into map keys and values and through any
number of slice and pointer prefixes.

A core_types case per engine generates the analyze_types schema and
queries with database/sql and pins the output; the end-to-end module
gains the drivers so the generated code compiles.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
codegen/golang: bind named placeholders with sql.Named, and run the n…
a075a83 
…ew engines' examples

SQL Server and Spanner name a parameter @name in the query text and
ClickHouse's server-side placeholder is {name:Type}; their drivers take
such an argument as sql.Named rather than by position. When every
parameter of a query is named that way, pass each as sql.Named with the
name the query uses. A query written with ? keeps positional arguments.

Add an authors example for ClickHouse, SQL Server and Spanner beside the
existing ones, and one for DuckDB in a module of its own, since its
driver is cgo and the main module's builds keep cgo off. Test helpers
create a database per test on the server CLICKHOUSE_SERVER_URI,
MSSQL_SERVER_URI or SPANNER_SERVER_URI names and skip when none is; the
DuckDB example opens a database in memory. CI runs ClickHouse and SQL
Server as services, builds the end-to-end module with cgo for the DuckDB
driver, and tests the DuckDB module; docker-compose gains a ClickHouse
service beside the SQL Server and Spanner Omni ones for local runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
ci: give the ClickHouse service a password so the example can reach it
298b452 
The ClickHouse image restricts the default user to connections from
localhost unless a password is configured, and the test job reaches the
service through a port mapping, so the ClickHouse example failed to
authenticate. Set a password on the service and in the URI, in
docker-compose too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
examples: make the examples a module of their own, and run Spanner Om…
7cec97c 
…ni in CI

The examples' drivers were becoming dependencies of sqlc itself, and the
DuckDB driver, being cgo, had to live in a module apart. Move every
example into one module under examples/, which requires sqlc through a
replace, so the drivers are its dependencies alone and cgo can stay on
for all of it; sqlc's own go.mod loses them, and lib/pq with them, since
only the examples imported it. The DuckDB example rejoins the authors
example beside the others, and the test helpers for ClickHouse, SQL
Server and Spanner move into the examples module, since the drivers they
need are what kept them out of sqlc. The end-to-end suite skips the
directory under examples that holds those helpers rather than treating
it as an example.

sqlc-test-setup now installs Spanner Omni from its standalone server
release, checked against a pinned SHA-256, and starts a single server in
the background on port 15000, so CI runs the Spanner example against it
alongside the ClickHouse and SQL Server services. The Makefile and CI
run the examples from their module.

Drop the clearing of a column's type expression in combineAnalysis: the
compiler builds a live-database analyzer only on the legacy path and
returns before doing so on the core path, so the two never meet and the
expression is never set where the analyzer runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
docs: say N/A rather than Not implemented in the language support table
e4783bd 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
ci: run the examples in a job of their own
5675ff3 
The examples need every database sqlc supports, while sqlc's own tests
need only PostgreSQL and MySQL. Split the test job in two, and let
sqlc-test-setup take the databases to install and start as arguments so
each job sets up what it uses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
ci: build the end-to-end module with cgo off again
29c7517 
The DuckDB golden imports the DuckDB driver, which is cgo, so it is left
out of that build and checked by the replay suite alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
codegen/json: leave out an unset type_expr; test-setup: install Click…
6fc4352 
…House and SQL Server

A column's type_expr is only set when the analysis core typed it, and the
JSON plugin wrote every other column with an explicit null, which put a
line on every column of every request the legacy path produced. Drop the
member when it is unset instead, so those requests read as they did
before the member existed; the goldens on the legacy path are back to
what they were.

sqlc-test-setup now installs and starts ClickHouse and SQL Server as
well, so the examples need no service containers from CI's host.
ClickHouse comes from the release tarball goldeneye pins, checked
against its SHA-512 and cached where goldeneye caches it, and runs from
a configuration the tool writes that gives the default user the password
the tests use. SQL Server comes from Microsoft's apt repository for
Ubuntu 22.04 and 24.04, set up non-interactively with the EULA accepted,
and starts through systemd where there is one and in the background
otherwise. The examples job in CI installs and starts every database
through the tool.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
codegen/golang: name the engines with constants; ci: build the end-to…
bdb4e57 
…-end module with cgo

The end-to-end module is one of its own, so cgo there is no concern of
sqlc's builds, and the DuckDB golden compiles with the rest again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
test-setup: keep the SQL Server it starts off the tool's own output
4b58b94 
A detached server holding the tool's stderr open kept anything reading
the tool's output waiting after the tool had finished. Its output goes to
a file of its own, the way the ClickHouse and Spanner Omni servers' does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
sqlc-test-setup: accept a successful mssql-conf setup without stat-in…
61ee60d 
…g its conf

On a GitHub runner the tool runs as an unprivileged user, and
/var/opt/mssql is readable by the mssql user only, so stat-ing
/var/opt/mssql/mssql.conf failed with permission denied even though
setup had completed. Treat a successful setup as installed, and when
setup fails check for the configuration through sudo before deciding
whether the failure was only the service start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
codegen/json: write an unset type_expr as null again
456a02b 
The regex that stripped the member from protojson's output was a hack
around EmitUnpopulated, which has no per-field form. A column the
legacy path typed now carries "type_expr": null like every other
unset message field, and the goldens are regenerated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
Bind named placeholders from the compiler, scan DuckDB lists and Clic…
4257811 
…kHouse nullables correctly

A parameter whose placeholder names it, SQL Server's and Spanner's @name
and ClickHouse's {name:Type}, is now marked by the compiler: the mssql
and googlesql converters carry the name on the ParamRef as the ClickHouse
one already did, the analyzer keeps it whichever use of the parameter it
sees first, and the plugin Parameter gains a named field. The Go codegen
reads that instead of searching the query text for "@name", which bound
a CAST-wrapped @A positionally, because the analyzer had named it after
the column it was compared with, and matched "@id" inside "@identity_no".

A row struct is scanned into and a params struct is passed as arguments,
and DuckDB wants different types for the two: a list is duckdb.Composite
when scanned and a plain slice when passed. Row structs used the params
form. The imports for the element of a generic type such as
duckdb.Composite[[]time.Time] were also missing, and sql.Named's
database/sql import reached the querier interface, which does not use it.

clickhouse-go dereferences a Nullable value only when Nullable is the
column's outermost type, so LowCardinality(Nullable(T)) and
SimpleAggregateFunction(f, Nullable(T)) come through as pointers that a
sql.Null wrapper cannot scan; they take the pointer form now.

sqlc-test-setup start skips SQL Server wherever install did, rather than
failing on an Ubuntu release the packages are not published for.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
@claude
plugin: pass the placeholder's name on Parameter rather than a flag
205daf7 
The name a placeholder carries, @name or {name:Type}, is what the
driver binds by, so the plugin Parameter carries it as name, collected
from the query's ParamRef nodes by the compiler rather than taken from
the analyzer, which also names a parameter after the function it is
compared with. The Go codegen passes it to sql.Named instead of the
column's name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kyleconroy @claude