Repository navigation
AsyncAPI 3.x: write the test suites - #1809
Draft
LautaroPetaccio wants to merge 20 commits into
Draft
LautaroPetaccio wants to merge 20 commits into
LautaroPetaccio wants to merge 20 commits into
Conversation
A generated test cannot go through the driver: there would be no test without EvoMaster running. It cannot use a client from the core either, since a broker has no universal one and the transport code belongs in drivers. So the driver renders the publish and await as source lines while the search runs, and the writer pastes them, which is what RPC already does for an endpoint invocation. The core asks for them on the action, naming the language and the variable the reply has to land in, so two actions in one test cannot collide. It asks only when a test will be written. Around the pasted lines it adds what the driver cannot know: the operation and its outcome, that a promised reply arrived, and which declared message the classifier recognised it as. A driver that renders nothing produces a test that says so rather than an empty one. With this, --problemType ASYNCAPI no longer refuses to write tests. What replaces that refusal is a narrower one: only Java and Kotlin, because the lines being pasted are the driver's. For the same reason black-box no longer defaults the format to Python for AsyncAPI, which would have made an ordinary run fail at start-up: it has a driver in both modes, so the format comes from the driver as it does in white-box. The suite splitter read a status code off every result, which would have thrown for the first AsyncAPI individual written. It now branches on the result type, counting a message that went out and was answered as a success.
REST writes its own RestAssured calls rather than asking the driver for them, and Kafka can be written the same way: the document names the broker on the server, the topic on the channel or its binding, and the field the correlation id rides in. Nothing else is needed to publish the same message again. So a Kafka test now stands up a producer and a consumer of its own. It seeks to the end of the reply topic before publishing, so a reply older than the test is not taken for an answer to it, mints a fresh id each run and matches it on the way back. Core writes the text and links nothing: the dependency lands on the generated suite, as rest-assured does for REST. The driver's own rendering stays, and wins where it is present. It is what a transport the contract cannot describe needs: one whose correlation id rides inside the service's own message layout, where the document says nothing about where to put it. Neither available, and the test says so rather than being silently empty. What the search resolved is kept on the result, so none of it is worked out a second time here.
The REST writer calls its variables res_0 and body_0, so there is no reason for these to carry an asyncApi prefix on every one. They are now producer_0, consumer_0, record_0, correlationId_0 and reply_0, keyed off the reply variable the core names rather than off the action index, so the locals of one action always agree with the reply it leaves behind. A test name read verb, target, result in REST, and ours was missing the verb: publish_on_<operation>_returns_<message>. The two outcomes that are not a declared reply say what they are rather than naming an enum constant.
…once A REST action is a two-line RestAssured call. Publishing to a broker and correlating a reply needs properties, a producer, a consumer, a seek, a send and a poll loop, and repeating that per action buried the test in scaffolding: eighteen lines to say what a reader wanted in one. It is a method now, written once for the suite, so an action is the call and its assertion. It follows the SSRF assertion helper, which is emitted into the same part of the class and gated the same way, on whether the solution has anything that uses it. A suite that publishes over a transport the contract cannot describe, where the driver renders the lines instead, is not made to carry a Kafka dependency it never calls. The new hook takes the solution, which addExtraStaticVariables does not, because whether a member is needed is a property of the whole suite rather than of one action.
Every other problem type's naming strategy has a unit test asserting the exact strings it produces; this one had none. It does now, including that silence is named the same whether or not the experimental oracles are on. It is named by the outcome when they are off and by the fault's label when they are on, and a test's name must not change with a flag that is only about reporting. Two places deliberately depart from what the other writers do, and now say so where a reader meets them. The reply variable is not minted from the counter that gives REST res_0 and body_0, because the name has to be settled while the search runs, being part of what a driver renders against; the action's index is the only number in hand there. And the emitted Kafka code names its types in full instead of adding imports, because the suite's imports are decided before it is known whether anything will publish over Kafka, and adding them always would make every AsyncAPI suite need the dependency.
What the search published is now kept on the result -- broker, topics, the header the correlation id rode in, the payload -- so the writer has the whole call in hand and never reads the document again. A suite asks the driver where each server is, the way it asks for baseUrlOfSut, and keeps the answer in a variable per server. A broker started on a random port is then reachable from the generated test, which the document's own address would not be. The default keeps the address the document gave, so a black-box suite, which has no driver to ask, is unchanged. The reply is asserted on the way REST asserts on a response: the same enableBasicAssertions, the same fieldsToSkipInAssertions, the same size and collection limits, the same tolerance on doubles. Nothing is predicted from the schema; every assertion is a regression on what came back. Kafka is written in Java and Python as well as Kotlin, so the suite can be generated in any of the three the rest of EvoMaster offers. The fault line of a test's comment block moves up into TestCaseWriter, which is what the TODO beside it asked for, and the AsyncAPI block now lists its calls with their outcome where REST lists them with their status code.
write_driver gains the AsyncAPI section: what a driver has to answer for a message-driven service, and that one is needed even in black-box mode, which is the opposite of every other problem type and so worth saying twice. library_dependencies gains the two a generated Kafka suite compiles against, and options.md is regenerated for outputFormat, which now accepts the languages a Kafka suite can be written in.
This was referenced Sep 30, 2026
LautaroPetaccio
changed the title
LautaroPetaccio
added this pull request to stack #1812
September 30, 2026 17:25
Java has no elvis, and the ternary the Java suite was written with named the call on both sides of it. A driver that works the address out, rather than keeping one, did that work twice every time a suite started. Two statements instead, which is also what the generated code reads like by hand.
The note on outputFormat read as though a driver were involved in the tests as well. It is not: a generated Kafka test publishes with a client of the transport, and the driver renders the publishing lines only for a transport the document cannot describe. What a driver does in either mode is publish while the search runs, which is also why a DEFAULT format is resolved from it rather than from the black-box default. And it could not resolve to Python even so: SutInfoDto.OutputFormat has no entry for it, so a Python suite has to be asked for with the option. Said where the resolution happens, and in the option's own text. The black-box page had the same gap, and the sentence had been put between "--blackBox true" and the "It" that referred to it.
The Python half was only ever checked by py_compile, and three things that parse perfectly were waiting there. assertNotNull is a JUnit name and no Python suite defines one. The blank-reply assertion handed Python Kotlin's isNullOrEmpty. And the server address was declared only for the JVM formats while the call that uses it was written for all three, so a Python suite with a driver named a variable that was never there. The driver is now asked only where a suite can hold one, and the Kafka imports follow the helper rather than every AsyncAPI Python suite, which is what the comment beside them already promised. One writer writes every suite of a run. Clearing the servers it found came after two early returns, so a suite with nothing to publish over Kafka kept the last one's and assigned to fields it had not declared. Cleared first now. Two server names that differ only where an identifier cannot, "a-b" and "a.b", collapsed onto one variable and declared it twice; the first keeps it and the rest fall back to the address the document gave. A field name is a literal too. They went in unescaped, so a reply with a field called "$ref" -- ordinary JSON -- wrote Kotlin that reads it as a template and does not compile. Values already went through the quoting helper; names and server names do now as well. A number outside a double's range parses as an infinity, which is not a literal in any of the three, so it is left to a comment. Kafka is only written when the correlation id rides in a header. Where the document puts it in the payload, the recorded payload is the one the search composed, without it: that test would publish a message nothing could answer and then take whatever was on the reply destination for its answer. Left to the driver, as the class comment always said it should be. The emitted client now closes what it opens, in all three languages, rather than on the happy path only, and a reply destination with no partitions is treated as nothing to wait for instead of being polled, which throws. A format with no client written for it throws where it used to quietly write Kotlin into a file of another language.
The constraint that a suite can only be written in Java, Kotlin or Python was checked while the options were read, and skipped a DEFAULT because the driver had not been asked yet. For AsyncAPI that is the ordinary case in both modes, so a driver naming anything else reached the writer unchecked. The check moved to where it can be run twice, and is run again as soon as the driver's answer lands. Beside it, what the comments there claim. A driver cannot name Python at all, since the enum it answers with has no entry for one, and that is now said both where the format is resolved and in the option's own text. The DTO said the reply variable was null when no script was asked for, when it is set whenever tests are written. The module still said no tests were written yet. And getAsyncApiServerAddress said a generated test asks it, which only a suite that carries a driver can: a plain black-box one publishes to the address the document declares. A driver script of nothing but blank lines is no script, and was taken for one: the test published nothing and then asserted on a variable nothing had assigned.
The driver publishes while the search runs; the tests it leaves behind do not use it. write_driver said a generated test asks the driver where a server is, without saying that only a suite carrying one can. blackbox put its note between a bullet and the "It" that referred to it, where Markdown reads it as part of that bullet anyway. library_dependencies listed what a JVM suite needs but not what a Python one does, which is kafka-python. The section heading now matches its neighbours, as does the name the page uses for the tool.
AsyncApiReplyAssertions had three hundred lines and one payload shape reaching them; it now has a test of its own, driving every branch across the three languages, which is what caught the Python forms. The sorter had none at all, because sorting happens only when a suite is written and no unit test wrote one. The Java half of the compile step had none either, and only a run with a broker exercised it. Two existing tests were not checking what they claimed. The Kafka body test asked for the body before the class members, which is the reverse of what the suite writer does, so it ran with no servers known and pinned the black-box shape while configured white-box; it now runs in the suite writer's order and asserts the variable a white-box suite actually publishes to. And asserting that a body contains "bessj" could not fail, as the fake driver's own script renders the topic names. The name-with-or-without-the-fault test compared a name with itself if the fault stopped being reported, so it now checks the fault is there first. The Python test skips where there is no python3 rather than erroring, and the files both tests write are removed whatever happens.
Four test classes each declared the same resource path, and one of them imported the same name twice. The injector already holds it, so they take it from there. The module test said the writer pastes the driver's lines, which is the fallback rather than what it usually does.
Writing a suite in black-box mode threw: the shared writer asks BlackBoxUtils for the base URL to put in it, and that answered "Black-box testing is currently not supported for ASYNCAPI". Which is the one mode this problem type is unusual in -- a driver publishes while the search runs, because nothing else can, but the suite that run leaves behind has none -- so it is the mode the option text and the black-box page both describe. Nothing had caught it because every writer test drove the test writer directly, and the suite writer is what calls this. A message-driven service has no one base URL to resolve: each server the document names has its own address, and the suite publishes to those, so there is nothing to answer with and nothing in the suite reads it. Both languages now have a test that writes a whole black-box suite and checks what it does not have -- no driver field, nothing started or stopped, nothing asked where a server is -- and that a message goes to the address the document declares.
Seeking to the end of the reply destination is lazy in kafka-python as it is on the JVM: it takes effect on the first poll, by which time the reply has already been published and sits behind the position, so it is never seen. The JVM helpers were fixed for this; the Python one was written from the same shape and inherited the bug without the fix. Every Python Kafka test therefore waited out its whole deadline and then failed on the reply being absent. Found by running one against a real broker and a real service: the service answered -- the replies are on the topic -- and the test saw none of them. Asking for the position drops the same three tests from 16 seconds to 1.4, and the reply arrives. Nothing could have caught this before, because a Python suite was only ever checked by py_compile, and this parses.
…reads right Of the assertions over emitted text, nearly all matched a substring and six matched a line. For a writer whose failures have all been code that parses and then does the wrong thing, that is the wrong ratio, and the measurements said so: line coverage sat at 95% while branches sat at 61 to 78. The JVM output is now compiled. core must not depend on a Kafka client -- the dependency belongs to the suite, not to the fuzzer -- so the helper is replaced by a declaration of the same shape and everything the writer decided is compiled around it: the call, its arguments, the escaping, the literals, the assertions. It earned its place immediately, by catching that a JUnit 4 suite needs a different import than the one assumed. The call every generated test runs on is pinned whole rather than by four substrings that matched in any order; swapping the topic published to with the one awaited would have left all four passing. Its deadline, which decides whether the test waits at all, was asserted nowhere. A second document reaches what the shared one cannot: a message with no reply at all, so the absent argument is seen as None and not null; a correlation header named something other than the default, which NCS's name is identical to; and headers of the message itself, which no document here declared, so the flattening had never run. The Python suite is now the one the run writes, parsed as written. It used to be a module assembled by the test, which supplied by hand the very imports the writer is responsible for emitting -- so the condition that decides whether they appear was checked by nothing. Four tests were not testing what they claimed. Sorting a one-element list never reaches a comparator, so that one passed against a comparator that always threw. Its sibling ordered by a payload the contract does not declare, so the key it meant to check was never the one that decided. The two black-box ones asserted the absence of a type that the shared header imports anyway. And the fix a real broker was needed to find was pinned in Python only, so deleting it from either JVM helper stayed green. Covered besides: what one suite found is no longer left for the next, two server names that collapse onto one identifier, a reply matching no declared message, a format named by the driver after every option has been checked, and which half of a split suite a test lands in -- none of which had a test.
…ackage Every other file under output/service is injected; these two are stateless objects that take what they need as arguments. That is the same reason the reply classifier was moved out of the problem type's service package, so they go the same way, into a package of their own named after the protocol they write for -- which is how the REST naming helpers are already grouped. The writer itself stays: it is bound in the module and the suite writer asks the injector for it. Two tests follow what they test. The one for the writer had been sitting with the AsyncAPI fixtures rather than beside TestSuiteWriterTest, which is where a test for something under output/service belongs.
…nce had to The note on the defaults called the tests here "suites", which is the one word in this area that already means something else -- what a run writes -- and said one test needed createTests on when four call sites do. Neither told a reader the rule: what a caller passes replaces the default of that name, because EMConfig refuses an option given twice. That is now said, on the function rather than inside it. It was also only half true. The match read the name before '=', so the spelling EMConfigTest uses two files away, --createTests true, would have slipped past and failed to parse. Both spellings now count as one option, and a test holds that, since nothing else would notice.
The sampler test built the same three defaults itself, so a fourth added to one list would not have reached the other. They move to argsWith, which the injector already had to compute and which carries the replace-the-default rule with them. Only the list is shared. The sampler test keeps its own module graph, which binds the sampler alone rather than the whole problem type, and its own controller, whose start can be made to fail -- which is what those tests are about and what the driver the others use cannot express.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth of the AsyncAPI 3.x series, on top of #1754. A search over a message-driven service now leaves a test suite behind, as every other problem type does.
What a generated test looks like
The test talks to the broker with an ordinary
kafka-clientsproducer and consumer. It does not go through an EvoMaster driver at run time, the same way a REST test does not: once written, it stands on its own.How the publishing lines are written
Two ways, in this order:
enablePureRPCTestGenerationpath. It is what a transport the document cannot describe needs: one whose correlation id rides inside the service's own message layout, where nothing in core could know where to put it.kafka-clientsout of core: the text is emitted as text, and only the generated suite ever compiles against it.publishAndAwaitReplyis written once per suite, in the companion object, rather than inlined in every test. It seeks to the end of the reply topic before publishing and mints a fresh correlation id per call, so a reply left over from an earlier run is never taken for an answer to this one.Finding the broker
A generated test cannot point at the address in the document: that is where the service was deployed, not where a system started for testing is.
SutHandlergainsgetAsyncApiServerAddress(serverName), defaulting tonull, and the suite fills one variable per server before the tests, exactly asbaseUrlOfSutis filled. When there is no driver, or the driver says nothing, the document's address is used.Assertions
Regression on the observed reply, never a prediction from the schema: the same
enableBasicAssertions,fieldsToSkipInAssertions,maxResponseByteSize,maxAssertionForDataInCollectionand double tolerance the REST writer honours.The same suite as any other
The scaffolding is the shared writer's, and a test asserts that line by line against what the REST writer produces: of the 57 lines an EvoMaster suite opens with, 55 are identical and the other two differ only in the test and target counts. The reply variable is
res_0, as in REST and RPC; the test name comes from the sameLanguageConventionFormatter; the comment block lists calls with their outcome where REST lists them with their status code, and the fault line moved up intoTestCaseWriterso both say it the same way.Java, Kotlin and Python can all be generated for Kafka.
What this changes outside AsyncAPI
Four files the other problem types share. Each is behaviour-preserving for them, and each is here because the alternative was to restate something that already exists:
TestCaseWritergainsaddFaultsCommentLine— the fault line of a test's comment block, lifted out ofRestTestCaseWriter. The//TODO move up when adding test comments to other problem types as wellsitting beside it asked for exactly that, and AsyncAPI is the second problem type to want it; REST's output is unchanged. It also gains anaddExtraClassMembershook, a no-op for any writer that does not override it, so a suite carries a helper only when something in it calls one.TestWriterUtilsgainsisSuitableToPrint, moved fromApiTestCaseWriterwhere it wasprotectedand so unreachable from the AsyncAPI assertion helper. The copy that had been made of it instead had already drifted: its HTML-entity pattern missed the hex spelling and names carrying a digit, so'and½were asserted on verbatim. REST's own pattern is the one that moved, so REST output is unchanged.BlackBoxUtils.targetUrlanswers forASYNCAPIrather than throwing. Without it a black-box run could not write a suite at all — the shared writer asks for a base URL to inline and gotBlack-box testing is currently not supported for ASYNCAPI. A message-driven service has no single base URL, since each server the document names has its own address, so it answers with the empty string and nothing in the suite reads it.TestSuiteSplitterreplaces an unconditionalas HttpWsCallResultwith a branch on the result type. That also stopsWEBFRONTENDhitting aClassCastExceptionon the same line.