Agentic development, dark factory or spec driven development is clearly a thing now. In fact, giving a spec to a non deterministic agent is nothing new, we just used to call them developers, junior developers and interns. The issue is accountability. In order for a manager or senior developer to take accountability for the work of the non deterministic agent, they need to have agency, they need to have the ability to determine that the job done meets the specification, the explicit parts in the spec and the non explicit parts like cyber security and architecture. There are three dominant approaches to this, testing, reviewing, and automated quality tools like SonarQube.
When writing code was slower, senior developers would achieve agency through the review process. The process of reviewing code was significantly faster than the process of writing code which meant senior developers has agency over junior developers. Agents write lots of code, and they write it fast. Senior developers can no longer review the code fast enough which means they do not have agency over the agents and cannot take accountability for the code they write. Further, the cognitive load associated with reviewing code is significantly higher than that of writing it, it is also much less fun. Senior developers focused purely on reviewing code are giving up on the review process for agent generated code, losing their agency and can therefore not take accountability for it.
When the review process loses its effectiveness, the testing process needs to step up to provide the development team with agency.
Before AI agents, testers, business analysts and business subject matter experts would prepare and execute tests. This testing, whether manual or automated would provide them with the agency needed to take accountability for releasing systems into production. In the pursuit of speed and cost savings, many teams adopt the anti pattern of using Agents to generate tests. Once again, too many tests are produced for the testers, business analysts and business subject matter experts to review effectively. Whereas writing code has always been non deterministic, testing code has to be deterministic. This means the process of testing has to be rigorous and deterministic for the testers, business analysts and business subject matter experts that take accountability for the system to be released into production. This means that Agents cannot be used as a “bag of magic beans” but rather need to follow a deterministic process for generating effective test suites.
In the pre-agentic world, testing was the bottleneck or constraint that prevented the deployment of code to production.
In the agentic world, testing IS STILL the bottleneck or constraint that prevented the deployment of code to production.
I would love to hear from Agentic practitioners in large corporations about how they are addressing the accountability issue.
(Also posted to LinkedIn)
Leave a Reply