diff --git a/docs/agents.md b/docs/agents.md index ea7153c88..c104714b9 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -165,6 +165,21 @@ After a failed run, the agent reads every trace under `output/`, clusters failur If the fix held, the PR goes green. If it didn't, every edit is rolled back with `git checkout` and the report says which patterns the agent couldn't safely handle. No half-applied fixes left behind, no `retries: 3` masking the problem. +## Assertions in plain language + +Some outcomes are hard to pin to a locator: "checkout form has all required fields", "success message is shown". Instead of building a fragile chain of `see*` checks, the agent can write the expected outcome as a statement with the [Decision helper](/assertions#decision-assertions): + +```js +I.decide([ + 'order summary lists the purchased items', + 'success message is shown', +]) +``` + +The agent runs the statement on the live page like any other command and keeps it only if it passes. The test then checks it on every run. + +Decision models like [Jev](https://openrouter.ai/typesafe/jev-1.13) are built for this. They answer in a fraction of a second, cost a fraction of a cent per request, and return a probability instead of free text. That makes them fast and cheap enough to run on every CI build, and predictable enough to keep in a test. + ## Skills bundle Skills teach the agent best practices for using CodeceptJS. Plug them in when you develop tests with agents, and update them regularly to ensure you use CodeceptJS in the most effective way. diff --git a/docs/ai.md b/docs/ai.md index 998511a2d..e156818fc 100644 --- a/docs/ai.md +++ b/docs/ai.md @@ -23,6 +23,7 @@ CodeceptJS AI can do the following: - πŸ‹οΈβ€β™€οΈ **assist writing tests** in `pause()` or interactive shell mode - πŸš‘ **self-heal failing tests** (can be used on CI) +- βš–οΈ **assert statements in plain language** with [decision models](#decision-assertions) ![](/img/fill_form.gif) @@ -338,6 +339,22 @@ Run tests with both AI and analyze enabled: npx codeceptjs run --ai ``` +## Decision Assertions + +Some checks are hard to express with locators: "checkout form has all required fields", "success message is shown". The [Decision helper](/helpers/Decision) asserts them in plain language: + +```js +I.decide([ + 'checkout form has all required fields', + 'submit button enabled', +]) +I.decideVisually('sidebar is shown') +``` + +It uses a [decision model](https://openrouter.ai/models?output_modalities=decisions) like [Jev](https://openrouter.ai/typesafe/jev-1.13) instead of a chat model. A decision model reads the page and returns the probability that a statement is true. It is fast, costs a fraction of a cent per request, and gives a probability instead of free text, so a step passes or fails on a confidence threshold you set. + +Configure it in `ai.decisionModel`. Decisions call the decisions API directly, so they don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage. + ## Advanced Configuration AI prompts and HTML compression can be configured inside `ai` section of `codecept.conf` file: diff --git a/docs/assertions.md b/docs/assertions.md index bb321cb17..d18107b2e 100644 --- a/docs/assertions.md +++ b/docs/assertions.md @@ -397,6 +397,87 @@ If two checks failed, the scenario fails with a single aggregated message like: expected soft assertions '[expected web application to include "You must accept the terms", expected element (.summary-error) to be visible]' to be empty ``` +## Decision Assertions + +Some checks are hard to express with locators: "checkout form has all required fields", "success message is shown", "sidebar is shown". The [Decision helper](/helpers/Decision) asserts such statements in plain language with a decision model. + +A [decision model](https://openrouter.ai/models?output_modalities=decisions), like [Jev](https://openrouter.ai/typesafe/jev-1.13), does not generate text. It reads the page and returns the probability that a statement is true. This makes it a good fit for assertions: + +- **Fast.** One request returns in a fraction of a second, so a decision step runs about as fast as a regular browser step. +- **Cost-efficient.** A request costs a fraction of a cent. You can run decision assertions in every CI build. +- **Reliable.** The answer is a probability, not free text. There is nothing to parse, and you choose how confident the model must be for the step to pass. + +Set `OPENROUTER_API_KEY`, configure the decision model in the `ai` section, and enable the helper next to your browser helper: + +```js +ai: { + decisionModel: { + model: 'typesafe/jev-1.13', + confidence: 0.7, + }, +}, +helpers: { + Playwright: { url: 'http://localhost' }, + Decision: {}, +} +``` + +`ai.decisionModel` accepts: + +| Option | Default | Description | +|---|---|---| +| `provider` | `openrouter` | `openrouter` reads `OPENROUTER_API_KEY`, `typesafe` reads `TYPESAFE_API_KEY` | +| `apiKey` | | API key, overrides the environment variable | +| `model` | `typesafe/jev-1.13` | model for `I.decide` | +| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually`, OpenRouter only | +| `confidence` | `0.7` | minimal probability for a statement to pass | +| `timeout` | `15000` | request timeout in ms | +| `maxLength` | `12000` | maximal length of ARIA snapshot or HTML sent to the model | + +Decisions don't need the `--ai` flag, and `ai.model` is not required. + +Then assert statements about the current page: + +```js +I.decide('top level navigation is available') + +I.decide([ + 'checkout form has all required fields', + 'success message is shown', + 'submit button enabled', + 'cancel button present', +]) + +I.decideVisually('sidebar is shown') +``` + +`I.decide` sends the page URL, title and ARIA snapshot to the model. A statement passes when its probability reaches `confidence`. A list of statements is checked in one request, and every statement must pass. The failure message lists the statements that failed, with their probabilities: + +``` +expected page to satisfy "success message is shown" (12%) with confidence of 70% +``` + +`I.decideVisually` also sends a screenshot, so it needs a model with image input. It uses `visualModel`, which is [Clef](https://openrouter.ai/cloudflare/clef) by default. Visual decisions are experimental. + +To turn decisions off without removing the helper, set its `mode`: + +| Mode | Behavior | +|---|---| +| `assert` | default, fails the step when a statement is not confirmed | +| `report` | requests the model but never fails, even on API errors; results and errors are added to the step as a comment | +| `skip` | does not request the model, every decision passes | + +```js +helpers: { + Playwright: { url: 'http://localhost' }, + Decision: { mode: process.env.CI ? 'assert' : 'skip' }, +} +``` + +A failed decision is not retried by the [retryFailedStep](/plugins/retryFailedStep) plugin: asking again would cost another request and return the same answer. Only connection errors and timeouts are retried. + +Use decision assertions for what a page means, and built-in assertions for exact values. `I.see('Total: $42.00')` is still the right check for a price. + ## Choosing an Approach | You want to check… | Use | @@ -412,4 +493,5 @@ expected soft assertions '[expected web application to include "You must accept | A matcher the above do not cover | `grab*` + `chai` / `jest` / `node:assert` | | A **reusable, project-specific** check | [Custom helper](/custom-helpers) with `see*` method using `codeceptjs/assertions` | | Many independent checks in one run | `hopeThat` from `codeceptjs/effects` | +| A statement that is hard to express with locators | [Decision helper](#decision-assertions) β€” `I.decide`, `I.decideVisually` | | Hiding values from logs | `secret()` | diff --git a/lib/ai.js b/lib/ai.js index a0190db29..dcff6af94 100644 --- a/lib/ai.js +++ b/lib/ai.js @@ -59,7 +59,7 @@ class AiAssistant { debug('Enabling AI assistant') this.isEnabled = true - const { html, prompts, ...aiConfig } = config + const { html, prompts, decisionModel, ...aiConfig } = config this.config = Object.assign(this.config, aiConfig) this.htmlConfig = Object.assign(defaultHtmlConfig, html) @@ -271,4 +271,120 @@ function parseCodeBlocks(response) { return modifiedSnippets.filter(snippet => !!snippet) } +const DECISION_ENDPOINTS = { + openrouter: { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' }, + typesafe: { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' }, +} + +const defaultDecisionConfig = { + provider: 'openrouter', + model: 'typesafe/jev-1.13', + visualModel: 'cloudflare/clef', + confidence: 0.7, + timeout: 15000, + maxLength: 12000, +} + +class DecisionConnectionError extends Error {} + +class DecisionAI { + constructor(config = {}) { + this.config = { ...defaultDecisionConfig, ...config } + + const { provider, confidence } = this.config + this.endpoint = DECISION_ENDPOINTS[provider] + if (!this.endpoint) throw new Error(`Unknown decision provider "${provider}" in ai.decisionModel, use one of: ${Object.keys(DECISION_ENDPOINTS).join(', ')}`) + if (!(confidence > 0 && confidence < 1)) throw new Error(`ai.decisionModel.confidence must be between 0 and 1, got ${confidence}`) + + this.fetchImpl = fetch + } + + checkModel() { + if (this.config.apiKey || process.env[this.endpoint.keyName]) return + + const noKeyErrorMessage = ` + No API key is set for decision model. + + [!] Set ${this.endpoint.keyName} environment variable or apiKey in ai.decisionModel config. + + Example (connect to OpenRouter, default): + + export OPENROUTER_API_KEY=sk-or-... + + ai: { + decisionModel: { + model: 'typesafe/jev-1.13', + confidence: 0.7, + } + } + + Get a key at https://openrouter.ai/settings/keys + + Example (connect to TypeSafe): + + export TYPESAFE_API_KEY=... + + ai: { + decisionModel: { + provider: 'typesafe', + model: 'jev-latest', + } + } + + See https://openrouter.ai/models?output_modalities=decisions for all decision models. + `.trim() + + throw new Error(noKeyErrorMessage) + } + + async decide(model, state, statements) { + this.checkModel() + + const apiKey = this.config.apiKey || process.env[this.endpoint.keyName] + const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'noul', instructions: statement }])) + debug('Decision request', model, statements) + + const controller = new AbortController() + const timer = setTimeout(() => controller.abort(), this.config.timeout) + + let result + try { + let response + try { + response = await this.fetchImpl(this.endpoint.url, { + method: 'POST', + headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, + body: JSON.stringify({ model, state, questions }), + signal: controller.signal, + }) + } catch (err) { + throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`) + } + + if (!response.ok) { + const body = await response.text().catch(() => '') + throw new Error(`Decision model ${model} responded with ${response.status}: ${body}`) + } + + result = await response.json() + } catch (err) { + if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.config.timeout}ms`) + throw err + } finally { + clearTimeout(timer) + } + + debug('Decision response', result?.answers, result?.usage) + + const answers = result?.answers || {} + return statements.map((statement, i) => { + const probability = answers[`q${i}`]?.noul + if (typeof probability !== 'number') throw new Error(`Decision model ${model} returned no answer for "${statement}"`) + return probability + }) + } +} + +export { DecisionAI, DecisionConnectionError } + export default new AiAssistant() diff --git a/lib/helper/Decision.js b/lib/helper/Decision.js new file mode 100644 index 000000000..e67ac2f90 --- /dev/null +++ b/lib/helper/Decision.js @@ -0,0 +1,243 @@ +import fs from 'fs' +import path from 'path' + +import Helper from '@codeceptjs/helper' +import Container from '../container.js' +import Config from '../config.js' +import store from '../store.js' +import output from '../output.js' +import AssertionFailedError from '../assert/error.js' +import { compactAriaSnapshot } from '../aria.js' +import { minifyHtml } from '../html.js' +import { pickActingHelper } from '../utils/trace.js' +import { DecisionAI, DecisionConnectionError } from '../ai.js' + +/** + * Asserts statements about the current page with a decision model. + * + * Decision models (like [Jev](https://openrouter.ai/typesafe/jev-1.13) or [Clef](https://openrouter.ai/cloudflare/clef)) + * do not generate text. They read the page state and return the probability that a statement is true. + * A statement passes when its probability reaches the configured `confidence`. + * + * ```js + * I.decide('top level navigation is available'); + * I.decide([ + * 'checkout form has all required fields', + * 'success message is shown', + * 'submit button enabled', + * 'cancel button present', + * ]); + * I.decideVisually('sidebar is shown'); + * ``` + * + * The model receives the page URL, title and ARIA snapshot. + * Helpers without ARIA snapshots (anything but Playwright) send minified HTML instead. + * `decideVisually` also sends a screenshot, so it requires a model with image input. + * + * This helper must be enabled together with a web helper (Playwright, Puppeteer, WebDriver). + * + * ## Configuration + * + * Decision model is configured in the `ai.decisionModel` section of the config: + * + * ```js + * ai: { + * decisionModel: { + * provider: 'openrouter', + * model: 'typesafe/jev-1.13', + * visualModel: 'cloudflare/clef', + * confidence: 0.7, + * }, + * }, + * helpers: { + * Playwright: { url: 'http://localhost', browser: 'chromium' }, + * Decision: {}, + * } + * ``` + * + * * `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`). + * * `apiKey` (optional) - API key, overrides the environment variable. + * * `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider. + * * `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only. + * * `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass. + * * `timeout` (default: `15000`) - request timeout in ms. + * * `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model. + * + * The helper has one option: + * + * * `mode` (default: `assert`) - how decisions are executed: + * * `assert` - request the model and fail the step when a statement is not confirmed. + * * `report` - request the model but never fail, even on API errors. Results and errors are added to the step as a comment. + * * `skip` - do not request the model, every decision passes. + * + * ```js + * Decision: { mode: process.env.CI ? 'assert' : 'skip' } + * ``` + * + * Decisions work without the `--ai` flag. + * Browse all decision models at [OpenRouter](https://openrouter.ai/models?output_modalities=decisions). + * + * ## Methods + */ +class Decision extends Helper { + constructor(config = {}) { + super(config) + const modes = ['assert', 'report', 'skip'] + this.mode = config.mode || 'assert' + if (!modes.includes(this.mode)) throw new Error(`Unknown Decision helper mode "${this.mode}", use one of: ${modes.join(', ')}`) + this.decisionAI = new DecisionAI(Config.get('ai', {}).decisionModel) + this.options = this.decisionAI.config + } + + get isAsserting() { + return this.mode === 'assert' + } + + get isReporting() { + return this.mode === 'report' + } + + get isSkipping() { + return this.mode === 'skip' + } + + /** + * Asserts that a statement (or each of statements) is true for the current page. + * The decision model receives page URL, title and ARIA snapshot. + * Multiple statements are checked in a single request and all of them must pass. + * + * ```js + * I.decide('user is logged in'); + * I.decide(['submit button enabled', 'cancel button present']); + * const probability = await I.decide('cart is empty'); + * ``` + * + * @param {string|string[]} statements statement or list of statements to verify. + * @returns {Promise} probability of each statement, `undefined` in `skip` mode. + */ + async decide(statements) { + if (this.isSkipping) return this._skip() + return this._run(async () => { + const state = await this._grabState() + return this._assert(statements, this.options.model, state) + }) + } + + /** + * Asserts that a statement (or each of statements) is true for the current page using a screenshot. + * The decision model receives page URL, title, ARIA snapshot and a screenshot. + * Requires `visualModel` with image input. Experimental. + * + * ```js + * I.decideVisually('sidebar is shown'); + * I.decideVisually(['logo is in the header', 'page uses dark theme']); + * ``` + * + * @param {string|string[]} statements statement or list of statements to verify. + * @returns {Promise} probability of each statement, `undefined` in `skip` mode. + */ + async decideVisually(statements) { + if (this.isSkipping) return this._skip() + return this._run(async () => { + const state = await this._grabState() + const screenshot = await this._grabScreenshot() + const content = [ + { type: 'text', text: JSON.stringify(state) }, + { type: 'image_url', image_url: { url: `data:image/png;base64,${screenshot}` } }, + ] + return this._assert(statements, this.options.visualModel, content) + }) + } + + _skip() { + if (store.currentStep) store.currentStep.comment = ' (skipped)' + } + + async _run(fn) { + try { + return await fn() + } catch (err) { + if (this.isReporting) { + if (store.currentStep) store.currentStep.comment = ` (error: ${err.message})` + output.say(` βœ– ${err.message}`, 'yellow') + return + } + if (!(err instanceof DecisionConnectionError)) err.isTerminal = true + throw err + } + } + + async _assert(statements, model, state) { + const list = [statements].flat() + if (!list.length) throw new Error('No statements to decide') + + const probabilities = await this.decisionAI.decide(model, state, list) + + const failed = [] + const results = list.map((statement, i) => { + const passed = probabilities[i] >= this.options.confidence + const result = `${passed ? 'βœ”' : 'βœ–'} ${statement} (${formatProbability(probabilities[i])})` + this.debugSection('Decision', result) + if (!passed) failed.push(`"${statement}" (${formatProbability(probabilities[i])})`) + return result + }) + + if (this.isReporting) { + if (store.currentStep) store.currentStep.comment = `\n${results.join('\n')}` + results.forEach(result => output.say(` ${result}`, result.startsWith('βœ”') ? 'green' : 'yellow')) + return Array.isArray(statements) ? probabilities : probabilities[0] + } + + if (failed.length) { + const err = new AssertionFailedError( + { statements: failed.join(', '), confidence: formatProbability(this.options.confidence) }, + 'expected page to satisfy {{statements}} with confidence of {{confidence}}', + ) + err.showDiff = false + err.message = err.cliMessage() + throw err + } + + return Array.isArray(statements) ? probabilities : probabilities[0] + } + + async _grabState() { + const helper = this._actingHelper() + const state = { + url: await helper.grabCurrentUrl(), + title: await helper.grabTitle(), + } + + if (helper.grabAriaSnapshot) { + state.aria = compactAriaSnapshot(await helper.grabAriaSnapshot()).slice(0, this.options.maxLength) + return state + } + + state.html = (await minifyHtml(await helper.grabSource())).slice(0, this.options.maxLength) + return state + } + + async _grabScreenshot() { + const helper = this._actingHelper() + const file = path.join(store.outputDir, `decision_${Date.now()}.png`) + await helper.saveScreenshot(file) + if (!fs.existsSync(file)) throw new Error('Could not take a screenshot for visual decision') + try { + return fs.readFileSync(file).toString('base64') + } finally { + fs.rmSync(file, { force: true }) + } + } + + _actingHelper() { + const helper = pickActingHelper(Container.helpers()) + if (!helper) throw new Error('Decision helper requires a web helper: Playwright, Puppeteer or WebDriver') + return helper + } +} + +function formatProbability(probability) { + return `${Math.round(probability * 100)}%` +} + +export default Decision diff --git a/test/unit/ai_test.js b/test/unit/ai_test.js index e94a8b942..89b457e13 100644 --- a/test/unit/ai_test.js +++ b/test/unit/ai_test.js @@ -1,5 +1,5 @@ import { expect } from 'chai' -import AiAssistant from '../../lib/ai.js' +import AiAssistant, { DecisionAI } from '../../lib/ai.js' import config from '../../lib/config.js' import { createMockModel, MockResponses } from '../support/mock-ai-provider.js' import fs from 'fs' @@ -297,3 +297,46 @@ describe('AI module with mock provider', () => { expect(mockModel._getCallCount()).to.equal(3) }) }) + +describe('DecisionAI', () => { + it('guides user to set API key', async () => { + const key = process.env.TYPESAFE_API_KEY + delete process.env.TYPESAFE_API_KEY + try { + const decisionAI = new DecisionAI({ provider: 'typesafe' }) + expect(() => decisionAI.checkModel()).to.throw(/TYPESAFE_API_KEY[\s\S]*openrouter\.ai\/settings\/keys/) + const err = await decisionAI.decide('jev-latest', 'state', ['page is loaded']).catch(e => e) + expect(err.message).to.include('No API key is set for decision model') + } finally { + if (key) process.env.TYPESAFE_API_KEY = key + } + }) + + it('accepts API key from config', () => { + expect(() => new DecisionAI({ provider: 'typesafe', apiKey: 'secret' }).checkModel()).not.to.throw() + }) + + it('rejects unknown provider', () => { + expect(() => new DecisionAI({ provider: 'unknown' })).to.throw('Unknown decision provider') + }) + + it('asks all statements in one request', async () => { + const calls = [] + const decisionAI = new DecisionAI({ apiKey: 'secret' }) + decisionAI.fetchImpl = async (url, options) => { + calls.push({ url, body: JSON.parse(options.body) }) + return { ok: true, json: async () => ({ answers: { q0: { type: 'noul', noul: 0.9 }, q1: { type: 'noul', noul: 0.2 } } }) } + } + + const probabilities = await decisionAI.decide('typesafe/jev-1.13', { url: 'http://localhost' }, ['form is shown', 'cart is empty']) + + expect(probabilities).to.eql([0.9, 0.2]) + expect(calls).to.have.length(1) + expect(calls[0].url).to.equal('https://openrouter.ai/api/alpha/decisions') + expect(calls[0].body).to.eql({ + model: 'typesafe/jev-1.13', + state: { url: 'http://localhost' }, + questions: { q0: { type: 'noul', instructions: 'form is shown' }, q1: { type: 'noul', instructions: 'cart is empty' } }, + }) + }) +}) diff --git a/test/unit/helper/Decision_test.js b/test/unit/helper/Decision_test.js new file mode 100644 index 000000000..01693f639 --- /dev/null +++ b/test/unit/helper/Decision_test.js @@ -0,0 +1,275 @@ +import { expect } from 'chai' +import fs from 'fs' +import os from 'os' +import path from 'path' +import Decision from '../../../lib/helper/Decision.js' +import store from '../../../lib/store.js' +import Config from '../../../lib/config.js' + +function createDecision(decisionModel, config = {}) { + Config.create({ ai: { decisionModel } }) + return new Decision(config) +} + +function fakeFetch(answers, calls, status = 200) { + return async (url, options) => { + calls.push({ url, body: JSON.parse(options.body), headers: options.headers }) + return { + ok: status === 200, + status, + text: async () => 'boom', + json: async () => ({ answers }), + } + } +} + +function noul(...values) { + return Object.fromEntries(values.map((v, i) => [`q${i}`, { type: 'noul', noul: v }])) +} + +const browser = { + grabCurrentUrl: async () => 'http://localhost/checkout', + grabTitle: async () => 'Checkout', + grabAriaSnapshot: async () => '- heading "Checkout" [level=1]\n- button "Submit"', + saveScreenshot: async file => fs.writeFileSync(file, 'png-bytes'), +} + +describe('Decision helper', () => { + let decision + let calls + + afterEach(() => Config.reset()) + + beforeEach(() => { + calls = [] + decision = createDecision({ apiKey: 'secret' }) + decision._actingHelper = () => browser + }) + + it('passes when probability reaches confidence', async () => { + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls) + const probability = await decision.decide('submit button is present') + + expect(probability).to.equal(0.9) + expect(calls).to.have.length(1) + expect(calls[0].url).to.equal('https://openrouter.ai/api/alpha/decisions') + expect(calls[0].headers.Authorization).to.equal('Bearer secret') + expect(calls[0].body.model).to.equal('typesafe/jev-1.13') + expect(calls[0].body.questions).to.eql({ q0: { type: 'noul', instructions: 'submit button is present' } }) + expect(calls[0].body.state.url).to.equal('http://localhost/checkout') + expect(calls[0].body.state.title).to.equal('Checkout') + expect(calls[0].body.state.aria).to.include('Submit') + }) + + it('fails when probability is below confidence', async () => { + decision.decisionAI.fetchImpl = fakeFetch(noul(0.4), calls) + const err = await decision.decide('cart is empty').catch(e => e) + + expect(err).to.be.instanceOf(Error) + expect(err.message).to.include('"cart is empty" (40%)') + expect(err.message).to.include('70%') + }) + + it('respects configured confidence', async () => { + decision = createDecision({ apiKey: 'secret', confidence: 0.95 }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls) + + const err = await decision.decide('cart is empty').catch(e => e) + expect(err.message).to.include('95%') + }) + + it('checks all statements in a single request', async () => { + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.99, 0.8), calls) + const probabilities = await decision.decide(['form has fields', 'submit enabled', 'cancel present']) + + expect(probabilities).to.eql([0.9, 0.99, 0.8]) + expect(calls).to.have.length(1) + expect(Object.keys(calls[0].body.questions)).to.eql(['q0', 'q1', 'q2']) + expect(calls[0].body.questions.q2.instructions).to.equal('cancel present') + }) + + it('lists only failed statements', async () => { + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.1, 0.2), calls) + const err = await decision.decide(['form has fields', 'submit enabled', 'cancel present']).catch(e => e) + + expect(err.message).to.include('"submit enabled" (10%)') + expect(err.message).to.include('"cancel present" (20%)') + expect(err.message).not.to.include('form has fields') + }) + + it('sends screenshot to visual model', async () => { + const outputDir = store.outputDir + store.outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'decision-')) + decision.decisionAI.fetchImpl = fakeFetch(noul(0.8), calls) + try { + await decision.decideVisually('sidebar is shown') + expect(fs.readdirSync(store.outputDir)).to.be.empty + } finally { + fs.rmSync(store.outputDir, { recursive: true, force: true }) + store.outputDir = outputDir + } + + const { model, state } = calls[0].body + expect(model).to.equal('cloudflare/clef') + expect(state).to.have.length(2) + expect(JSON.parse(state[0].text).title).to.equal('Checkout') + expect(state[1].type).to.equal('image_url') + expect(state[1].image_url.url).to.equal(`data:image/png;base64,${Buffer.from('png-bytes').toString('base64')}`) + }) + + it('sends html when aria snapshot is not supported', async () => { + decision._actingHelper = () => ({ ...browser, grabAriaSnapshot: undefined, grabSource: async () => '

Checkout

' }) + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls) + await decision.decide('heading is shown') + + expect(calls[0].body.state.aria).to.be.undefined + expect(calls[0].body.state.html).to.include('

Checkout

') + }) + + it('uses typesafe endpoint', async () => { + decision = createDecision({ apiKey: 'secret', provider: 'typesafe', model: 'jev-latest' }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls) + await decision.decide('page is loaded') + + expect(calls[0].url).to.equal('https://api.typesafe.ai/v1/systemone') + expect(calls[0].body.model).to.equal('jev-latest') + }) + + it('reports http errors', async () => { + decision.decisionAI.fetchImpl = fakeFetch({}, calls, 402) + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('402') + }) + + it('reports missing answers', async () => { + decision.decisionAI.fetchImpl = fakeFetch({}, calls) + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('returned no answer for "page is loaded"') + }) + + it('times out hanging requests', async () => { + decision = createDecision({ apiKey: 'secret', timeout: 50 }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = (url, { signal }) => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted')))) + + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('did not respond in 50ms') + expect(err.isTerminal).to.be.undefined + }) + + it('times out stalled response body', async () => { + decision = createDecision({ apiKey: 'secret', timeout: 50 }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = async (url, { signal }) => ({ + ok: true, + status: 200, + json: () => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted')))), + }) + + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('did not respond in 50ms') + expect(err.isTerminal).to.be.undefined + }) + + it('marks failed decisions as not retryable', async () => { + decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls) + const err = await decision.decide('cart is empty').catch(e => e) + expect(err.isTerminal).to.equal(true) + }) + + it('marks http errors as not retryable', async () => { + decision.decisionAI.fetchImpl = fakeFetch({}, calls, 500) + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.isTerminal).to.equal(true) + }) + + it('marks page state errors as not retryable', async () => { + decision._actingHelper = () => ({ ...browser, grabTitle: async () => { throw new Error('page closed') } }) + const err = await decision.decideVisually('sidebar is shown').catch(e => e) + expect(err.message).to.equal('page closed') + expect(err.isTerminal).to.equal(true) + }) + + it('keeps connection errors retryable', async () => { + decision.decisionAI.fetchImpl = async () => { + throw new TypeError('fetch failed') + } + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('request failed: fetch failed') + expect(err.isTerminal).to.be.undefined + }) + + it('reads config from ai.decisionModel', () => { + decision = createDecision({ apiKey: 'secret', model: 'jev-latest', confidence: 0.9 }) + expect(decision.options.model).to.equal('jev-latest') + expect(decision.options.confidence).to.equal(0.9) + expect(decision.options.visualModel).to.equal('cloudflare/clef') + }) + + it('validates config', () => { + expect(() => createDecision({ apiKey: 'secret', provider: 'unknown' })).to.throw('Unknown decision provider') + expect(() => createDecision({ apiKey: 'secret', confidence: 1.5 })).to.throw('between 0 and 1') + }) + + it('requires api key only when deciding', async () => { + const key = process.env.TYPESAFE_API_KEY + delete process.env.TYPESAFE_API_KEY + try { + decision = createDecision({ provider: 'typesafe' }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls) + + const err = await decision.decide('page is loaded').catch(e => e) + expect(err.message).to.include('TYPESAFE_API_KEY') + expect(calls).to.be.empty + } finally { + if (key) process.env.TYPESAFE_API_KEY = key + } + }) + + describe('mode', () => { + afterEach(() => { + store.currentStep = null + }) + + it('skips decisions without requests', async () => { + decision = createDecision({}, { mode: 'skip' }) + decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls) + store.currentStep = { comment: '' } + + expect(await decision.decide('cart is empty')).to.be.undefined + expect(await decision.decideVisually('sidebar is shown')).to.be.undefined + expect(calls).to.be.empty + expect(store.currentStep.comment).to.include('skipped') + }) + + it('reports results in step comment without failing', async () => { + decision = createDecision({ apiKey: 'secret' }, { mode: 'report' }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.1), calls) + store.currentStep = { comment: '' } + + const probabilities = await decision.decide(['form has fields', 'submit enabled']) + expect(probabilities).to.eql([0.9, 0.1]) + expect(calls).to.have.length(1) + expect(store.currentStep.comment).to.include('βœ” form has fields (90%)') + expect(store.currentStep.comment).to.include('βœ– submit enabled (10%)') + }) + + it('reports api errors in step comment without failing', async () => { + decision = createDecision({ apiKey: 'secret' }, { mode: 'report' }) + decision._actingHelper = () => browser + decision.decisionAI.fetchImpl = fakeFetch({}, calls, 500) + store.currentStep = { comment: '' } + + expect(await decision.decide('page is loaded')).to.be.undefined + expect(store.currentStep.comment).to.include('responded with 500') + }) + + it('validates mode', () => { + expect(() => createDecision({}, { mode: 'soft' })).to.throw('Unknown Decision helper mode') + }) + }) +})