docs(rfc): propose OpenShell testing strategy

[elezar]

Summary

Propose RFC 0016 for OpenShell's testing strategy. Define separate ownership for behavioral contracts, test families, provisioning, installation, execution, and CI policy so contributors have one place to discuss the strategy and clear destinations for its living documentation.

Related Issue

Part of #3954, broadened at the maintainer's request to track the overall testing strategy and assign RFC 0016. Accepting the RFC does not close the implementation tracker.

Changes

• Define general conformance, feature-specific, driver-specific, disruption, load/scale, and lower-level testing boundaries; retain SDK compatibility in companion RFC PR docs(rfc): propose SDK conformance testing #3238.
• Define contracts, test cases, assertions, and suites; treat CLI and SDK access as client interfaces rather than separate test families.
• Specify mandatory versus optional support and distinct result semantics; defer classification of capability-dependent portable tests until a concrete migration example.
• Diagram target preparation separately from behavioral testing, reflecting tmachine environments, installers, and test suites.
• Document existing conformance, Podman driver, and Keycloak provider-refresh suite locations and proposed additions.
• Propose moving the shared conformance library under tests/suites/conformance after test(conformance): run driver suites with cargo #3866 removes the standalone conformance executable.
• Assign clear responsibilities to TESTING.md, proposed tests/CONFORMANCE.md, suite READMEs, CI.md, and contributor skills.
• Define incremental migration by behavioral intent and preserve unanswered versioning, admission, portability, reporting, and CI policy questions for review.

The final PR diff contains only the RFC. Current testing and CI reference documents will be updated through focused follow-ups as the strategy is agreed and implemented.

Testing

• mise run pre-commit passes, including Markdown, Rust workspace/E2E/example lint, formatting, license, Helm, Python, and SDK checks.
• Repository Markdown lint passes.
• git diff --cached --check passes; the final diff against the PR merge base contains only the RFC.
• RFC template sections and issue/PR references checked.
• Unit and E2E tests are not applicable to this proposal-only change.

Checklist

• Follows Conventional Commits.
• Commit is signed off for DCO.
• Living architecture and published documentation changes are deferred to implementation, as required by the RFC workflow.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[johntmyers]

If something is only implemented on one driver, I'd assume that should be part of "driver specific"

[johntmyers]

is this only unit and integration? is everything else an e2e test then?

[matthewgrossman]

is general conformance different than feature-specific where all drivers support it? I'm imagining that there's a set of features w/ venn diagrams including specific drivers, and in a world where all drivers are included, it just becomes a circle and is now deemed a "conformance test"

[matthewgrossman]

effectively wondering if conformance is just a special-case version of feature-specific tests where all drivers are included, or if you see it as something different

[matthewgrossman]

at a practical level I don't see any issues with global-setting changes if we're isolating tests, but I find this statement a little confusing. is it implying that multiple tests are sharing the same environment, so a good test-citizen should restore the env how they found it?

[matthewgrossman]

making sure I understand this section:

Rather than encoding in our test-suite which drivers support features a/b/c, that should be encoded as an API contract that consumers can see. The the test suite just becomes one consumer of that API, and decides which tests are worth running against this given openshell deploy?

[drew]

its a little confusing to put unit and integration in the same sentence. these are usually distinct types of tests and integration has become very overloaded. we call current conformance tests, "conformance integration", and current feature specific tests "feature specific integration"

[drew]

I would suggest the following taxonomy

Test family	Description
Lint	Check formatting, style, and static rules.
Unit	Verify one component in isolation. Place tests inline in its Rust module or in an adjacent test.rs file under src/.
Integration	Verify interactions between components. Place Rust integration tests in the crate’s top-level tests/ directory.
End-to-end	Verify a configured OpenShell target through a client interface. Place suites in the repository-level e2e/ directory; conformance is a class of end-to-end test for portable public contracts.
Benchmark	Measure performance or scale. Place crate benchmarks in benchmarks/ and full-system benchmarks in a root benchmarks/.

Types of e2e tests

End-to-end type	What it verifies
Conformance	Portable public contracts across applicable drivers, including advertised optional capabilities. Includes recovery and continuity after an induced failure. It should be possible to run these tests out of tree.
Feature	Behavior requiring a named external service or special gateway configuration.
Driver	Behavior specific to a driver, its host integration, or its configuration.
Installation	Installing, upgrading, and uninstalling candidate artifacts.

[drew]

This seems unrelated to the section.

[drew]

It would be good if we could start to define what conformance tests we're going to build. I would propose the following suites. Here's a list to get us started

• Sandbox lifecycle: Create, inspect, execute, stop, restart, and delete.
• Policy: Validate, apply, update, and report effective policy.
• Sandbox enforcement: Enforce filesystem, process, and network rules.
• Providers: Manage providers and verify credential delivery, isolation, rotation, and protection from exposure.
• Identity and authorization: Authenticate and enforce access boundaries.
• Middleware: Verify selection, ordering, transformation, and failure handling.
• Interceptors: Verify request transformation, rejection, and preservation of authorization.

[drew]

why is conformance at the top instead of inside conformance/?

[drew]

this whole section is pretty dense. i'm not sure i understand it. conformance tests should

• use public apis
• be parametrized by gateway endpoint
• have the ability to run out of tree so third parties can test for conformance
• run as part of nightly qualification
• run on branch checks when manually triggered.
• it would be great if we can granularly trigger conformance checks by suite on ci

[drew]

can we please detail what tests run where. for example, unit tests always run on ci, e2e tests run when manually triggered or on release qualification, etc. we also need to cleanup the current labelling approach.

i would also like to see us be more efficient in what tests run on branch checks. @SDAChess mentioned work to dynamically figure out what tests to run per pr. i think we should consider this. i've seen interesting approaches that use llms or jev to figure out what tests should be run based on changes. might be interesting to experiment with.

we don't have to build this all at once, but since this rfc is broadly titled "propose OpenShell testing strategy" and talks about execution stragies, i think we should detail where we're headed with things. current branch checks on prs are too slow.

[drew]

i'm assuming this is just for conformance tests. how do we specify a test to be unsupported or skipped?

[drew]

I'm not sure how scalable this is. For example, what if down the road we support loading more than one compute driver. Some compute drivers might enable different capabilities. This is also going to cause options on the compute driver to explode with every product capability a driver might or might not support. We've already started to see this and it creates quite a change amplification that I'd like to avoid.

As part of RFC-0012 we proposed using validation to assert capabilities. For example, if you create a sandbox with a file system policy, and no filesystem policy exists, that sandbox should fail to create will a validation error. I think this is a more scalable approach.

[drew]

We should handle some level of flaky-ness and include infra for retries. If a test is identified as flaky there should be a report that we monitor and fix out of band. If we hard fail on flakes, we're going to be fighting builds and wind up just manually retrying tests anyways (we already see this today).

Down the road, if there's a report we can have some agent iterate on the report to reduce flakes.