AI Agent for Test Automation

I built AI Agents to handle my test automation, and they immediately tried to unionise.

I’ve been doing test automation for long enough that I can spot a flaky test from far away, with the sixth sense of a person who has been personally betrayed by “WaitForTimeout(5000)” more times than they’ll admit in polite company.

So, when AI agents started looking like the next logical step – “What if my framework could think?”, I did what any sensible automation architect would do:

I built a small army of agents to help with test automation tasks, and then spent a week debugging why one of them kept confidently “fixing” things that were never broken.

This is the story of how I built them (LangChain + Python + Playwright + MCP vibes), the problems I ran into, and how I got the agents to stop acting like extremely confident interns with admin access.

The Dream: A Tiny QA CoE in My Laptop

The vision was simple:

• I feed user stories, BRDs, URLs and API specs
• Agents generate:
  • Page models / POMs (Playwright-based)
  • Manual test cases
  • Gherkin scenarios
  • Automated tests using the POM and business methods
  • API test matrices
  • Utility code (DB, PDF, mail, logs)
  • Execution + reporting
  • Log analysis and “why did this fail” summaries

Basically: a “Test Automation Factory” where I do the thinking once and agents do the repetitive work 100 times, without getting bored, asking for coffee, or resigning after a sprint retrospective.

The Cast: My Agent Sitcom Ensemble

Here’s the squad I ended up with:

1. Framework Creator Agent
   “Give me your stack, I’ll scaffold the repo, structure, configs, runners, CI templates.”
2. POM / Page Model Agent (Playwright + MCP-style browser control)
   “I’ll open the page, inspect elements, and generate page objects.”
3. Manual Test Creation Agent (RAG-powered)
   “I’ll read requirements, past defects, domain rules, and produce sensible manual tests.”
4. Page-level Test Creator Agent
   “I’ll create test cases per screen/component with positive/negative/edge coverage.”
5. Gherkin Test Creation Agent
   “I speak fluent Given-When-Then, occasionally with an accent.”
6. Utility Builder Agent
   “Need RDS helpers, PDF checks, Slack alerts, Jira hooks, Excel readers? Done.”
7. Execution Agent
   “I’ll run suites, collect artifacts, and package reports like a delivery service.”
8. Log Analysis Agent
   “I’ll parse the failure, correlate traces, and tell you what actually broke.”

And of course…

9. The ‘RAG Somewhere’ Agent
   This one doesn’t do much, but it makes sure we say “RAG” often enough to sound expensive.

Problem #1: The Agents Were Overconfident (A.K.A. “Hallucination with Swagger”)

Early version output looked like this:

“I fixed the selectors and improved stability.”
(No you didn’t. You renamed everything and broke three pages.)

The agent wasn’t lying maliciously. It just had what I call the Energy of a golden retriever:
happy, helpful and certain it brought you the correct stick. Even if the stick is your neighbour’s Wi-Fi router.

How I fixed it

I stopped letting agents “free-write” code and switched to structured outputs:

• JSON schemas for actions (create_file, update_file, add_test, explain_failure)
• Strict tool boundaries (agents can propose; only tools can write)
• “Confidence + evidence required” rule:
  • If you claim an element exists, show how you found it (selector strategy / DOM hint).
  • If you claim a fix, reference the failing log line/stack trace.

Also: I permitted agents to say “I don’t know” (shockingly effective).

Problem #2: Playwright + Real Apps = “Welcome to Authentication Hell”

In demos, everything is sunshine:

• Go to the URL
• Click login
• Done

In reality, enterprise apps go:

• redirect to SSO
• 2FA
• session tokens
• “Your password expired”
• “Your admin has changed your policy”
• a captcha appears like a final boss

One agent tried to “solve” MFA by recommending I “disable MFA in production.”
The agent is no longer invited to architecture meetings.

How I fixed it

I implemented three practical strategies:

1. Storage State + Known Test Accounts
   Log in once manually, save storageState, reuse it for agent runs.
2. Auth Adapters
   A plug-in style auth handler:
   • SSO cookie mode
   • API token mode
   • pre-seeded user session mode
3. Fallback Mode
   If login gets weird, agent switches to:
   • generate page models from accessible pages
   • or generate tests based on screenshots + DOM snapshots
   • or ask for a “post-login URL” and continue from there

No drama, no false heroics.

Problem #3: Flaky Selectors and the “Div Soup” Disaster

The first time my POM agent saw a modern UI framework page, it produced selectors like:

• div > div > div:nth-child(2) > div > div > button

This is not a selector. This is a cry for help.

How I fixed it

I added a selector policy:

Priority order

1. data-testid, data-test, data-qa
2. accessibility (getByRole, getByLabel, getByText with constraints)
3. stable attributes (name, aria-* when consistent)
4. last resort: CSS/XPath, but only with anchoring + uniqueness check

And I made the agent prove uniqueness:

• highlight count
• strict mode checks
• “does it still match after re-render?”

The agent now chooses selectors like an adult.

Problem #4: “It Works on My Machine” (Now Featuring Python Environments)

Agent runs started failing with classics:

• ModuleNotFoundError
• Playwright browsers not installed
• version mismatch between pydantic, langchain, and the universe
• Windows paths behaving like they’re haunted

How I fixed it

I treated the agent system like a product, not a pile of scripts:

• One command bootstrap
  • create venv
  • install requirements
  • install playwright browsers
  • validate with a health check
• A dedicated doctor command:
  • prints versions
  • validates tool access
  • confirms browser launch
  • checks vector store connectivity

The “doctor” command saved my sanity and possibly my hairline.

Problem #5: RAG Didn’t Magically Make My Agents Smart (It Made Them Confidently Misled)

I plugged in RAG expecting enlightenment.

Instead, the agent started quoting random paragraphs like:

“As per the document…”
…and then referenced a section about holiday policies while generating OTP API tests.

How I fixed it

RAG got disciplined:

• Better chunking (by headings/meaning, not arbitrary token slices)
• Metadata tags (module, app, version, date, source)
• Retrieval filters (“only pull docs for this app + this flow”)
• Top-k tuning + reranking
• And the golden rule:

If the retrieved context doesn’t contain the answer, the agent must say so.

RAG became a librarian, not a gossip.

Problem #6: Agents Loved Doing Extra Work (Especially the Wrong Extra Work)

You ask:

“Generate tests for Feature A”

Agent replies:

“Sure! I also refactored your entire framework, migrated to a new folder structure, renamed your utilities, and added blockchain.”

This is what I call Scope CreepGPT.

How I fixed it

I introduced “job contracts” per agent:

• clear input/output boundaries
• hard stop conditions
• max file change limit per run
• “Only touch these folders” rules

If the agent wants to refactor the universe, it must open a “proposal” first.
Like a civilized being.

Problem #7: When Agents Fail, They Fail Like Magicians (Smoke, Mirrors, and Zero Clues)

The most annoying failure mode was:

• agent runs a tool
• tool fails
• agent says “Done- All Good”

Sir. Nothing is done. The repo is on fire.

How I fixed it

I added:

• structured tool error returns
• mandatory “execution summary”
• logging with correlation IDs
• artifacts every run (inputs, outputs, tool calls, decision notes)

Now when something fails, I can trace it like a detective instead of guessing like a philosopher.

The Payoff: What Actually Worked (And Made Me Keep This System)

Once the chaos settled, the benefits were real:

• Page models in minutes for new screens
• Consistent test case templates across manual + automation
• Faster coverage expansion (UI + API + orchestrator style flows)
• Better failure triage (log analysis agent is my new favorite coworker)
• Reusable utilities generated in a standard style (DB, PDF, mail, etc.)
• A system that scales with the team because the “how” is encoded

It didn’t replace engineers.
It replaced the boring parts that drain engineers.

And that’s the only kind of replacement I’m interested in.

Lessons Learned (From the School of Hard Knocks and Harder Stack Traces)

1. Agents need guardrails more than they need “intelligence.”
2. RAG is not magic; it’s a retrieval system that needs design.
3. Browser automation for agents must respect real-world auth.
4. Tooling must be observable, testable, and boring (boring = reliable).
5. If an agent can change code, it must also explain exactly what it changed and why.

Closing: My Agents and I Have an Understanding Now

I don’t call them “AI Agents” anymore.

I call them:

• “Junior Engineers with infinite enthusiasm”
• “Automation interns who never sleep”
• “Highly productive chaos, contained by policies”

They still surprise me sometimes.
But now it’s the good kind of surprise, like opening a report and seeing:

tests generated
selectors stable

logs explained
and nothing renamed to “final_final_v7_fixed2”

Which, in automation terms, is basically a Disney ending.

Also, click the link below to check out more about our frameworks and AI-enhanced tools

To understand more about how we at Testhouse supercharge test automation with controlled automation agents, visit https://www.testhouse.net/solution/frameium-autotest/