Reference

Manual vs automated: modes and reason codes

Every case gets a mode, a readiness score from 0 to 100, and named reason codes explaining exactly what is blocking automation.

Applies toPlans:FreeProBusinessEnterpriseRoles:OwnerAdminDeveloperQaWhere:Web appMCP

"Some tests can't be automated" is true and useless. What a team can act on is which tests, why, and what would change it. That is what the rubric produces, and because the reasons are named codes rather than prose, they can be counted, filtered, routed to a human, and quoted verbatim in a client report.

The three execution modes

Mode	What it means
`AUTOMATED`	Runs unattended as a Playwright or Maestro spec; evidence is captured by the runner.
`MANUAL`	Executed by a person, with evidence captured in the extension's Test Run mode.
`HYBRID`	Automated steps up to the manual gate, then a human step with evidence. The case records the step where automation stops.

How a verdict is reached

Detectors run. Deterministic patterns look for blockers in the case text and in the structured signals — source references, recent churn in the files behind the feature.
Fixtures override. A registered fixture that solves a blocker cancels it. An OTP bypass turns an OTP case back into a spec.
Surviving blockers decide the mode. None left means AUTOMATED. All of them behind one step gate means HYBRID, with the step split recorded. Otherwise MANUAL.
Readiness is scored. 100 minus penalties plus bonuses, every term itemised so a reviewer can see why the number is what it is.
A justification is written deterministically. An LLM may rewrite that sentence for readability; it can never change the mode, the score or the reasons.

The automation-readiness score

Readiness starts at 100 and moves from there. Blockers subtract — the heaviest reason counts in full and each further reason counts at a reduced weight, because blockers overlap. Determinism problems subtract too: no steps, no expected result, vague language, a known-flaky history, or a module that has been failing recently.

Bonuses go the other way: an existing spec, a per-step expected result, selector hints in project memory, and how often the case runs. The breakdown lists every term that fired, so the score is an argument you can disagree with rather than a black box.

The 14 reason codes

These are the complete set. They are shared by the app, the MCP tools, the client report and the assignment engine, so a code means the same thing everywhere it appears.

Code	Label	What it means	What clears it
`OTP_2FA`	OTP / 2FA	OTP, 2FA, magic link or SSO with no test bypass registered as a fixture.	Register an `otp_bypass` fixture.
`CAPTCHA`	CAPTCHA	CAPTCHA or Turnstile challenge without test keys.	Register the provider's test keys as a fixture.
`REAL_PAYMENT`	Real payment	Real money movement through a live payment provider.	Register a `psp_sandbox` fixture.
`EMAIL_SMS_DELIVERY`	Email / SMS delivery	Verifies that an email, SMS or push notification actually arrived.	A mail-catcher or delivery-webhook fixture.
`VISUAL_JUDGMENT`	Visual judgment	"Looks right" — brand, layout or aesthetic judgment only a human can make.	Nothing; split the objective half into a HYBRID case.
`ACCESSIBILITY_JUDGMENT`	Accessibility judgment	Screen-reader or assistive-technology experience beyond automated axe rules.	Nothing; automated rules cover a different question.
`HARDWARE`	Hardware	Needs a physical device: printer, NFC, camera, Bluetooth or sensors.	A device farm, where one applies.
`THIRD_PARTY_NO_SANDBOX`	No third-party sandbox	Depends on an external system that offers no sandbox or test mode.	A vendor sandbox, if they ship one.
`NON_DETERMINISTIC_AI`	Non-deterministic AI	Asserts on LLM or AI-generated output that varies between runs.	Assert on structure or invariants instead of exact text.
`LEGAL_SIGNOFF`	Legal sign-off	Includes a legal or compliance sign-off step a person must perform.	Nothing; this one is the point.
`ONE_OFF`	One-off	Migration or one-time data fix; not worth automating.	Nothing; correctly stays manual.
`UNSTABLE_UI`	Unstable UI	UI area changed in too many recent commits to automate reliably yet.	Time — the signal clears as churn drops.
`DATA_SETUP_INFEASIBLE`	Data setup infeasible	Fixture or test-data setup cannot be reproduced automatically.	Register a `seed_data` or `test_account` fixture.
`EXPLORATORY`	Exploratory	Explicit exploratory charter; the tester decides what to try.	Nothing; an exploratory charter is deliberately unautomated.

Re-classifying a case

Open the case in the case editor and choose Re-classify.
BugMojo previews the rubric's verdict with the detector evidence behind each reason. Nothing is saved yet — the preview is read-only.
Choose Apply to fill the mode, readiness and reason controls, adjust anything you disagree with, then save. The save is an ordinary edit, so it writes revision N+1.

Common questions

Is the classification done by an AI model?

The verdict is deterministic — detectors, fixtures and a scoring rubric, no model. An LLM may only rewrite the justification sentence, never change the mode or the score.

Can an AUTOMATED case carry manual reasons?

No. The reasons are exactly why a case is <em>not</em> automated, so AUTOMATED with reasons is rejected as invalid input.

Can I override the verdict?

Yes. Re-classifying only previews the rubric's answer; you apply it or set the mode yourself, and your save is what persists.

How do I get a case out of MANUAL?

Clear the blocker it names. Registering a fixture — an OTP bypass, a sandbox key, seeded data — overrides the reason it solves and pushes the case back towards automatable.

Where do these codes show up outside the app?

In the review queue, on the case's mode chip, over MCP from <code>classify_test_case</code>, and in the client report's honest list of what was manual and why.

Was this helpful?

Updated Sep 19, 2026 · Still stuck? Contact support