E2E Test Dashboard
Circuitry.ai — automated end-to-end test suite
Home filter only — does not change which tenant runs in the sidebar.
Pass rate trend
Outcomes per run
agents_history_validate or agents_pcmi_validate suites.Feedback
Advisor conversations volume
Live API analysis (same source as the Streamlit panel): feedback messages per tenant, per day.
Agent runs volume
Live API analysis (same source as the Streamlit app): engaged agent runs per tenant, per day.
Tenant comparison
Tickets created, this range.
By assignee / status
Tickets created in the selected range — assignee / jira_status, live from Jira (refreshed on Apply).
Advisors — tickets by issue type
Tickets linked to this range's advisor feedback, grouped by advisor + triage issue type.
| Advisor | Issue type | Tickets |
|---|---|---|
| No advisor tickets in range. | ||
Agents — tickets per tenant
Tickets linked to this range's agent feedback, per agent per tenant.
| Agent | Tenant | Tickets |
|---|---|---|
| No agent tickets in range. | ||
Recently created tickets — Advisors
Tickets created (or matched as duplicates) in the selected range — same columns as the Streamlit proposed-tickets table.
| # | Action | Parent | Assignee | Title | Items | Advisor | Project | Tenant | Env | Status | Link |
|---|---|---|---|---|---|---|---|---|---|---|---|
| No advisor tickets in range. | |||||||||||
Recently created tickets — Agents
Tickets created (or matched as duplicates) in the selected range — same columns as the Streamlit proposed-tickets table.
| # | Action | Parent | Assignee | Title | Items | Agent | Project | Tenant | Env | Status | Link |
|---|---|---|---|---|---|---|---|---|---|---|---|
| No agent tickets in range. | |||||||||||
Triage
General Settings
The wall-clock limit before pytest-timeout kills a hung test's worker process (this is what causes the worker
crash/restart behavior above). Each field below applies to every test in that suite — saving a value makes
all of that suite's markers uniform. Stored in config/test_timeouts.yaml.
A test with its own hardcoded @pytest.mark.timeout(N) is never lowered below N by these fields — only raised.
Fallback for any suite without its own marker below. Default: 120s.
Covers advisor list, converse, and edit-shell tests (advisors_* markers).
Covers Power BI embed / vision tests (analysts_* markers) — these need more time for the PBI iframe to load.
Covers agent chat / validation tests (agents_* markers).
Visual-regression timeouts are normally computed from capture/settle/screenshot timing, not a flat number.
Set a value here to force every visual-regression test to this timeout instead.
0 = use the computed defaults. Stored in config/visual_regression_config.py.
A different kind of timeout — how long a single click/wait_for_selector call waits, not how long the whole test runs.
Default: 30 000 ms. Stored in config/tenants.yaml.
How many times a crashed worker is automatically restarted to complete its remaining tests.
Set to 0 to disable restarts entirely.
Example: 5 = worker retries up to 5 times before remaining tests are marked not_executed.
Disabled when "Auto restart until complete" is on.
Maximum time to wait for the Power BI "Loading data…" spinner to clear before taking a vision screenshot.
Default: 60 000 ms (1 min). Stored in config/test_timeouts.yaml.
Extra pause after the spinner disappears so charts finish rendering. Default: 3 000 ms. Set to 0 to skip the settle wait entirely.
Maximum time to wait for a History accordion field's loading spinner to clear before reading its text.
Default: 15 000 ms. Stored in config/test_timeouts.yaml.
Extra pause after every History tab field finishes updating, before its section text is read for validation. Default: 5 000 ms (5 s).
How many claims' Claim Adjudicator/Claim Score Generator/dialogue-API lookups run
concurrently in one "Parts + Labor Coverage (API)" run — pure HTTP, no browser, so cheap to
parallelize, but bounded to avoid rate-limiting the tenant's real backend.
Default: 5. Stored in config/test_timeouts.yaml.
How far apart a Claim Score Generator run's timestamp may be from the Claim Adjudicator run's timestamp for the same claim and still be treated as the same adjudication cycle. Default: 3600 s (1 hour).
Number of automatic retries for analyst vision and PBI tests when they fail or the PBI iframe is slow to load.
Default: 2. Applies to all analyst test markers. Stored in config/test_timeouts.yaml.
Minimum overall score a GPT-4o vision run must return to mark the test as passed.
Default: 85. Stored in config/visual_regression_config.py.
Minimum content-relevance score. If the screenshot scores below this (e.g. login page, error state) the test fails immediately with score 0 — regardless of pass threshold. Default: 70.
How long to wait for Power BI loading indicators to disappear before starting visual stability checks. Default: 45 000 ms.
How often a low-quality screenshot is taken during chart stability polling. Shorter = more responsive; longer = less CPU pressure on slow PBI pages. Default: 3 000 ms.
The page content hash must be unchanged for this long before the screenshot is accepted. Increase if charts are still animating. Default: 6 000 ms.
Hard cap on the stability-polling loop. If the page never stabilises within this time the screenshot is taken anyway. Default: 45 000 ms.
Fixed pause after the network goes idle so React finishes rendering before the stability poll begins (Dashboard, Advisors, Conversations…). Default: 5 000 ms.
Page content must be visually unchanged for this long on route pages. Default: 4 000 ms.
Hard cap on the stability-polling loop for route pages. Default: 20 000 ms.
Categories available on the Category dropdown in the Agent test data panel (Agents page). Picking notification on a row there reveals its Expected senders / To email fields — every other category behaves like today's plain chat-agent row. Saved immediately — no need to click Save Settings below.
Model used by Agent Mode's planner, step execution, and test code generation (Ticket Planner, Chat, Test Creator). Sol is the strongest reasoning, best for element identification accuracy; Terra balances cost/speed; Luna is cheapest/fastest for high-volume use. Saved immediately — no need to click Save Settings below.
JIRA Integration
Used by the Ticket Planner's "Browse Azure Test Plans" modal to pull real Test Plan/Suite/Test Case steps. Create a PAT with Test Management (Read) and Work Items (Read) scope.
Tenants
Manage which tenants appear on Home → View by tenant and in the Home sidebar.
Names are copied from config/tenants.yaml when the list is empty. Add or remove tenants here only.
Home tenant list
"Pull tenant IDs" resolves each tenant's tenant_id via the Knowledge API (knowledge_api_client.py) — this is what Internal Apps → Feedback/Triage uses to query Milvus per tenant. Sync now and the scheduled digest also pull IDs automatically at start. Requires CIRCUITRY_PASSWORD_TENANT_<ENV> to be set in .env for each tenant's environment.
Active run tenant
URL used for single-worker runs and when you click a tenant on Home.
How parallel runs work
Turn whole features on or off and set a safety cap. Saved to
config/dashboard-features.json (editable on disk).
• Several customers = same tests, many companies (3 customers → 3 runs, or more if you add “browsers per customer”).
• Browsers (one customer) on the test page = same company, tests split across 4 browsers to finish faster.
Counts all running pytest browsers. Example: 3 customers × 2 browsers each = 6 — must be ≤ this cap.
Test Presets
PCMI Config
Credentials for the PCMI portal used in 3-way agent validation tests.
One entry per tenant + environment combination.
Runs the same login (and, if you give it a claim number, the same claim-search flow) the real tests use, against the credentials already saved above — so you can check a tenant's PCMI login without running a full test.
Visual Regression
Capture JPEG baselines per environment and tenant. Compare runs use the active tenant pill or each tenant in a multi-customer run.
Several customers
Check each customer that needs baselines. Same idea as Several customers… on the test page — one capture per selected page per customer.
One customer, faster: split the selected baseline pages across several browsers (same URL as the primary tenant above).
Start = missing selected pages only · Re-capture = all selected pages per customer. Counts are tenant-aware (only analysts/advisors on that customer via API).
Browser Settings
Controls how the Playwright browser behaves during test runs. Changes take effect on the next test run.
OFF — a real browser window opens so you can watch each step. Best for debugging.
settings.json. Overrides the defaults in
config/tenants.yaml for each new test run.
Autoheal
When a locator fails after all retries, Autoheal captures the page HTML and asks AI to suggest a better locator. You review suggestions here and apply or reject them individually — or click Fix All to apply every pending suggestion at once.
Only register a candidate after at least this many failed attempts. Example: 3 = triggers on the 3rd failed attempt.
Loading…
Scheduled Runs
Create recurring test jobs that fire automatically. Each schedule uses a preset, tenant, and worker count you define.
Loading…
Notification Center
Notify people when a run starts and/or when it finishes — PDF report attached to the finish email, failed-test screenshots inline, plus a summary and chart. Recipients are organized into named groups so different audiences can be notified for different events, instead of one list getting every message. SMTP credentials (host/username/password) are configured via .env, not here, and never displayed in the browser.
notification_from_email directly in settings.json and reload this page.