An AI agent that says "tested and passing" has told you almost nothing until you know whether its test could ever have failed. After a year of building publishing software with AI coding agents, my team has a core rule: each check must catch a broken build, processes must fail loudly when something goes wrong, and synthetic users must walk the app before I ever see it. Here is why that discipline matters and how we apply it.
The testing discipline behind every tool we build with AI agents starts with the core understanding that this is human-directed agentic programming. If you don't stand on the shoulders of decades of good software development and project management discipline then you are destined to fail.
- AI-built software usually works on the first try and breaks later, in the places nobody looked.
- A check that can't fail isn't evidence, so prove each one catches a known-broken version before you trust it.
- Processes should stop and report when their input is wrong, instead of finishing the job with corrupted data.
- Synthetic users in Playwright and Claude in Chrome find broken buttons, typos, accessibility gaps and wasted steps before a human reviews anything.
- This discipline is the difference between vibe coding and software you can run a business on.
Why Vibe-Coded Tools Often Work on Day One and Break Once You Use Them In Production
When I started, my projects were closer to what people think of as vibe coding: "Hey ChatGPT, create me a plugin that builds an FAQ box at the bottom of every article, pulling data from the post it appears on." It would do that, and do it nicely, and it would almost always have a problem we didn't discover until we started using it more. That sucked.
As the tools got more complicated, the failures got harder to see. When software sends thank-you emails to contributors, I'm not going to know whether that worked. I'd just assume it did since the debugging round said everything was ready to ship. We hit the same thing in FlowAds, our ad and affiliate placement tool, when links weren't rendering the affiliate code correctly, and in a dozen other situations. Each one looked fine from the outside.
Not Just "Intelligence" - Why Building Tools To Unlock The Power Of AI Is Essential To Success
Why small businesses should build their own AI tools: the publishing stack behind a travel network, what it costs and why your subject matter experts must drive it.
How I'm Transforming My Blogging Strategy To Develop, Currate and Publish Original High-Quality Content
How FlowContent replaced AI slop with interviews, surveys and roundups, and how any small publisher can start gathering original, expert-sourced content this week.
FlowGate: The Release Process That Turns AI-Built Apps Into Real Products
How FlowGate takes AI-built Joomla extensions from internal tools to public releases: Joomla 5 and 6 testing, a demo site with fictional data and a human final call.
Promoted by FlowAds
This is one reason most vibe-coded software sucks, and why I resist calling the work I do vibe coding. It's also why software engineers and project managers still have jobs and COOs need to spend time cultivating a culture of excellence that focuses on maintaining the right processes, not just outsourcing everything to teams of agents while laying off expensive senior staff.
A Check That Can't Fail Isn't Evidence
The core rule is simple to say and easy to skip. Before a test, probe or deploy check counts as evidence, it has to be shown to fail when something is wrong. The cleanest way is to run it against a known-broken version, such as an older release that had the bug. If the check passes the broken version too, the check is the defect.
A test that only confirms a page loaded, a script that hides its own errors or a comparison against the wrong baseline will all report success every time. Now we require the design to tell us when something fails, and to be robust enough to anticipate the failure in the first place.
Make It Fail Loudly When the Input Is Wrong
The same thinking applies inside the tools. Sometimes, when I'm testing and something doesn't feel right, the fix is as easy as adding debug output so we can see what's happening. Other times it means requiring that a process doesn't run at all if it didn't read its source document properly.
A tool that stops and says "I couldn't read the source, nothing was written" costs you a minute. A tool that pushes ahead with a corrupted payload can publish a broken article, email the wrong people or quietly drop affiliate codes for weeks.
Most of this design language lives in the standing instructions every agent on my team works from. I still remind the agents during development, and again while debugging.
My Synthetic User Testing Finds What Code Reviews Can't See
The newest layer in my testing process is using synthetic users. We developed Playwright scripts and Claude in Chrome prompts to walk through an entire app the way a person would, trying to accomplish real goals like building a meme or creating a roundup. Then the agent reports back with findings.
Sometimes the findings are basic: a broken button, a language file issue, a typo or colors that fail the WCAG contrast guidelines (AA calls for a 4.5:1 ratio for normal text) that keep pages readable for people with low vision. Other times they are fundamental. On one walk, the agent found that a task took 24 steps and came back with a plan to cut it to 14. That's a MASSIVE improvement.
These walks have caught real bugs that automated tests and code review missed. One audit of our promotion tools found confirmation dialogs that didn't work on the Dispatch, Post Now and Delete buttons, and an Instagram dispatch that went live on a single click. We fixed the broken dialogs in a follow-up release.
How To Build This Discipline Into Your Own AI Projects
You don't need a custom framework to start. Add three rules to the instructions you give your coding agent:
- Prove every test can fail - show it catching a deliberately broken version before it counts.
- Stop on bad input - if the process can't read what it needs, it halts and reports instead of guessing.
- Walk it like a user - before you review a feature, have an agent complete a real task in a browser and report every step, error and slowdown.
Then keep the human at the end of the line, reviewing the agent's report before anything ships.
Human Intuition Is Still Critical To The Testing Process - We're Illogical and Like To Break Stuff!
The checks catch most of what's broken, but the best signal is still a person using the software and sensing that something doesn't feel right. When you get that feeling, don't accept "tested and passing." Ask the agent to show you the test failing on purpose. If it can't, you've found your next bug.
Thanks for reading. We hope this was helpful!