E2E Test Dashboard
Circuitry.ai — automated end-to-end test suite
Home filter only — does not change which tenant runs in the sidebar.
Pass rate trend
Outcomes per run
agents_history_validate or agents_pcmi_validate suites.General Settings
The wall-clock limit before pytest-timeout kills a hung test's worker process (this is what causes the worker
crash/restart behavior above). Each field below applies to every test in that suite — saving a value makes
all of that suite's markers uniform. Stored in config/test_timeouts.yaml.
A test with its own hardcoded @pytest.mark.timeout(N) is never lowered below N by these fields — only raised.
Fallback for any suite without its own marker below. Default: 120s.
Covers advisor list, converse, and edit-shell tests (advisors_* markers).
Covers Power BI embed / vision tests (analysts_* markers) — these need more time for the PBI iframe to load.
Covers agent chat / validation tests (agents_* markers).
Visual-regression timeouts are normally computed from capture/settle/screenshot timing, not a flat number.
Set a value here to force every visual-regression test to this timeout instead.
0 = use the computed defaults. Stored in config/visual_regression_config.py.
A different kind of timeout — how long a single click/wait_for_selector call waits, not how long the whole test runs.
Default: 30 000 ms. Stored in config/tenants.yaml.
How many times a crashed worker is automatically restarted to complete its remaining tests.
Set to 0 to disable restarts entirely.
Example: 5 = worker retries up to 5 times before remaining tests are marked not_executed.
Disabled when "Auto restart until complete" is on.
Maximum time to wait for the Power BI "Loading data…" spinner to clear before taking a vision screenshot.
Default: 60 000 ms (1 min). Stored in config/test_timeouts.yaml.
Extra pause after the spinner disappears so charts finish rendering. Default: 3 000 ms. Set to 0 to skip the settle wait entirely.
Maximum time to wait for a History accordion field's loading spinner to clear before reading its text.
Default: 15 000 ms. Stored in config/test_timeouts.yaml.
Extra pause after every History tab field finishes updating, before its section text is read for validation. Default: 5 000 ms (5 s).
Number of automatic retries for analyst vision and PBI tests when they fail or the PBI iframe is slow to load.
Default: 2. Applies to all analyst test markers. Stored in config/test_timeouts.yaml.
Minimum overall score a GPT-4o vision run must return to mark the test as passed.
Default: 85. Stored in config/visual_regression_config.py.
Minimum content-relevance score. If the screenshot scores below this (e.g. login page, error state) the test fails immediately with score 0 — regardless of pass threshold. Default: 70.
How long to wait for Power BI loading indicators to disappear before starting visual stability checks. Default: 45 000 ms.
How often a low-quality screenshot is taken during chart stability polling. Shorter = more responsive; longer = less CPU pressure on slow PBI pages. Default: 3 000 ms.
The page content hash must be unchanged for this long before the screenshot is accepted. Increase if charts are still animating. Default: 6 000 ms.
Hard cap on the stability-polling loop. If the page never stabilises within this time the screenshot is taken anyway. Default: 45 000 ms.
Fixed pause after the network goes idle so React finishes rendering before the stability poll begins (Dashboard, Advisors, Conversations…). Default: 5 000 ms.
Page content must be visually unchanged for this long on route pages. Default: 4 000 ms.
Hard cap on the stability-polling loop for route pages. Default: 20 000 ms.
JIRA Integration
Tenants
Manage which tenants appear on Home → View by tenant and in the Home sidebar.
Names are copied from config/tenants.yaml when the list is empty. Add or remove tenants here only.
Home tenant list
Active run tenant
URL used for single-worker runs and when you click a tenant on Home.
How parallel runs work
Turn whole features on or off and set a safety cap. Saved to
config/dashboard-features.json (editable on disk).
• Several customers = same tests, many companies (3 customers → 3 runs, or more if you add “browsers per customer”).
• Browsers (one customer) on the test page = same company, tests split across 4 browsers to finish faster.
Counts all running pytest browsers. Example: 3 customers × 2 browsers each = 6 — must be ≤ this cap.
Test Presets
PCMI Config
Credentials for the PCMI portal used in 3-way agent validation tests.
One entry per tenant + environment combination.
Visual Regression
Capture JPEG baselines per environment and tenant. Compare runs use the active tenant pill or each tenant in a multi-customer run.
Several customers
Check each customer that needs baselines. Same idea as Several customers… on the test page — one capture per selected page per customer.
One customer, faster: split the selected baseline pages across several browsers (same URL as the primary tenant above).
Start = missing selected pages only · Re-capture = all selected pages per customer. Counts are tenant-aware (only analysts/advisors on that customer via API).
Browser Settings
Controls how the Playwright browser behaves during test runs. Changes take effect on the next test run.
OFF — a real browser window opens so you can watch each step. Best for debugging.
settings.json. Overrides the defaults in
config/tenants.yaml for each new test run.
Autoheal
When a locator fails after all retries, Autoheal captures the page HTML and asks AI to suggest a better locator. You review suggestions here and apply or reject them individually — or click Fix All to apply every pending suggestion at once.
Only register a candidate after at least this many failed attempts. Example: 3 = triggers on the 3rd failed attempt.
Loading…
Scheduled Runs
Create recurring test jobs that fire automatically. Each schedule uses a preset, tenant, and worker count you define.
Loading…