fix(ui-dialog,ui-modal): fix flaky Modal and DrawerLayout tests

[matyasf]

Summary

• Dialog reads its props when the focus region activates (one frame after open()), so props that change in between aren't lost. This fixes the flaky DrawerLayout Escape test and the mount-in-overlay case of INSTUI-4401 (not its resize case).
• New Dialog regression test for props that change before the focus region activates.
• Modal Tooltip tests now wait for the tooltip to hide instead of checking once. fixing random test failures

Test Plan

• In the docs app, load a DrawerLayout example narrow enough to start in overlay mode, open the tray, and check that Escape closes it and Tab stays inside it.

check this test code:

<div>
  <Flex gap="large" alignItems="start">
    <div>
      <RadioInputGroup name="fruit1">
        <RadioInput label="Apple" value="apple" />
      </RadioInputGroup>
    </div>
    <div>
      <RadioInputGroup
      name="fruit2"
      messages={[
        { type: 'screenreader-only', text: 'Pick one option' }
      ]}
      >
        <RadioInput label="SR only message" value="apple" />
      </RadioInputGroup>
    </div>
  </Flex>
  <RadioInputGroup
    messages={[{type:'success', text: "asd asdasd asdasd asdas"}, {type:'hint', text: "asd asdasd asdasd asdas"}]}
    name="example2"
    description="inline RadioInputGroup"
    layout="inline">
    <RadioInput value='sdfsdf' label='sdfsdf' />
  </RadioInputGroup>
  <RadioInputGroup
    messages={[{type:'error', text: "asd asdasd asdasd asdas"}, {type:'hint', text: "asd asdasd asdasd asdas"}]}
    name="example2"
    description="stacked RadioInputGroup"
    layout="stacked">
    <RadioInput value='sdfsdf' label='sdfsdf' />
  </RadioInputGroup>
    <FormFieldGroup
    messages={[{type:'error', text: "asd asdasd asdasd asdas"}, {type:'hint', text: "asd asdasd asdasd asdas"}]}
    description="Breakfast"
    rowSpacing="small"
    layout="inline"
    vAlign="middle"
  >
    <TextInput renderLabel="Favorite Breakfast Eatery"/>
    <RadioInputGroup
      name="beverage"
      description="Beverage of Choice"
      defaultValue="coffee"
      layout="columns"
    >
      <RadioInput label="Juice" value="juice" />
      <RadioInput label="Water" value="water" />
    </RadioInputGroup>
  </FormFieldGroup>
  <CheckboxGroup
  name="sports2"
  layout="inline"
  messages={[
    { text: 'Invalid name', type: 'error' }
  ]}
  onChange={function (value) { console.log(value) }}
  defaultValue={['soccer', 'volleyball']}
  description="I wish to receive score alerts for"
>
  <Checkbox label="Football" value="football" variant="toggle" />
  <Checkbox label="Basketball" value="basketball" variant="toggle" />
</CheckboxGroup>
    <CheckboxGroup
  name="sports2"
  layout="stacked"
  messages={[
    { text: 'Invalid name', type: 'error' }
  ]}
  onChange={function (value) { console.log(value) }}
  defaultValue={['soccer', 'volleyball']}
  description="I wish to receive score alerts for"
>
  <Checkbox label="Football" value="football" variant="toggle" />
  <Checkbox label="Basketball" value="basketball" variant="toggle" />
</CheckboxGroup>
</div>

Fixes INSTUI-5201

🤖 Generated with Claude Code

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://instructure.design/pr-preview/pr-2734/
Built to branch gh-pages at 2026-09-30 08:18 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[github-actions]

Visual regression report

Cypress suite: ✅ Passing

Visual diff: ⚠️ Changes detected.

Status	Count
Unchanged	98
Changed	1
New	0
Removed	0

Accessibility (axe): ✅ No violations.

📊 View full report — click a screenshot's ⚠ badge to see each violation boxed on the image, with the offending element named and contrast failures shown as color swatches.

Diff images (1)

badge-canvas.png — 1573 pixels differ

_{Baselines come from the visual-baselines branch. They refresh on every merge to master. The Cypress suite line covers the a11y and console-error assertions — a ❌ there means the suite found real issues even if the visual diff is clean.}