Agent Frameworks and Code Quality: Reducing the Risks of Nondeterminism and Vibe Coding

AI-generated code often looks correct — and that is exactly why it can be dangerous. Incorrect model assumptions may remain invisible for a long time. The result? Technical debt that is difficult to trace back to a single design decision. As agent frameworks gain traction, IT teams need to answer one practical question: how can they use nondeterministic AI models without losing control over code quality, costs, and system stability?

Key Takeaways

• Vibe coding can accelerate a team’s work, but without safeguards it can increase technical debt just as quickly.
• The problem is not a single model hallucination, but the repeated introduction of inconsistent or incorrect assumptions into the repository.
• The more tools and integrations an agent can access, the harder it becomes to keep its behavior predictable — especially when the agent chooses tools on its own.
• In enterprise organizations, the engineer’s role is shifting from simply generating code to governing the process by which code is created.

What Is AI Nondeterminism in Code Generation?

In the context of AI, nondeterminism means that the same model can generate different solutions for identical or very similar tasks. During prototyping, this is not always a problem. In production environments, however, it makes it harder to predict code quality, reproduce bugs, and maintain consistent standards across the team. For this reason, generating code that merely looks correct is not enough. Teams need mechanisms that reduce model variability and help control the consequences of AI-driven decisions.

From Vibe Coding to Supervised Code Generation

If agent frameworks are to support code quality in a meaningful way, teams cannot treat them as autonomous senior developers. Without the right context, a model does not understand local architectural trade-offs, historical dependencies, or the reasons why a given solution was implemented in the first place.

This is where the problem known as vibe coding begins. Vibe coding is an approach to software development based mainly on dialogue with AI, limited analysis of the generated code, and the assumption that the model “knows what it is doing.” Similar phenomena are also described as AI-first coding, prompt-driven development, or conversational coding, although these terms are not fully interchangeable. The code may be syntactically correct, but it may fail to follow design patterns, architectural decisions, or local repository rules. During prototyping, this can speed up work. In production environments, it becomes risky.

One of the most commonly overlooked sources of this risk is tool sprawl. It appears when an agent is given access to too many tools, functions, and integrations. In theory, this expands what the agent can do. In practice, the risk of incorrect routing increases — especially when the agent chooses from dozens of similar tools on its own.

Tool Sprawl in Practice

A larger tool catalog does not always improve agent effectiveness. It can increase the number of incorrect calls, retry loops, and costly operations that bring no real business value.

Why Governance Matters More Than the Model

Tool sprawl reveals a broader issue: the quality of AI-generated code is not determined by the model alone. Code rules, review processes, cost control, tests, and the way architectural decisions are documented are just as important. In enterprise environments, the advantage does not come from using AI itself, but from the organization’s ability to define clear boundaries for model behavior.

An AI agent does not create technical debt simply because it makes mistakes. It creates technical debt when those mistakes look credible and when subsequent changes become structurally inconsistent. For developers, vibe coding can shorten the path from idea to pull request. For IT managers, the risk begins when the team loses track of why the code looks the way it does. In practice, this means harder debugging, more regressions, inconsistent implementation patterns, and higher maintenance costs.

Vibe Coding vs. Enterprise Governance

AI Is Not Always the Best Tool

Code formatting, style enforcement, and rule-based quality checks are still better handled by classic linters and formatters. With consistent configuration, they are cheaper, faster, and deterministic. Using language models for tasks based on explicit rules often increases cost without delivering proportional value.

How Can Agent Frameworks Help Keep Legacy Code Stable?

The real test for agent frameworks begins when the code is no longer modular or well documented. In legacy systems, the model has to reconstruct the history of decisions, dependencies between modules, and undocumented business rules. That is why simply increasing the context window does not solve the problem. More tokens mean more information, but they do not guarantee a correct understanding of the relationships between those pieces of information.

A good example is the source code of Telegram for Android, analyzed during Talk4Devs.

One of the elements discussed was the ChatActivity.java class, which contains 45,569 lines of code and takes up around 2.43 MB. A file of that size is a challenge not only for humans, but also for agentic systems. Without prior decomposition of the repository, the model may miss important pieces of logic, map dependencies incorrectly, and generate changes that are inconsistent with the application architecture. In legacy code, splitting the repository into smaller contexts is often more effective than continuing to increase the number of tokens.

Why a Larger Context Window Is Not Enough

A larger context window allows the model to see more data, but it does not guarantee correct interpretation. That is why modern AI tools for code increasingly use semantic indexing, embeddings, and contextual search instead of simply “throwing” the entire repository into the prompt.

Expert Voice

“Effective work with AI systems on legacy code requires a strict separation between conceptual sessions and coding sessions, as well as mandatory, independent pull request review by the author before changes are passed on to the team.”
— Jarosław Michalik, Fractional CTO and Google Developer Expert (Android)

Human-in-the-Loop Governance in Practice

In mature organizations, AI does not eliminate the programmer’s role. It changes where human expertise creates the most value. Increasingly, teams talk about moving from the execution layer to the governance layer: engineers spend less time manually writing code and more time controlling the process of generation, validation, and deployment. These safeguards do not eliminate model nondeterminism, but they limit its impact on the production system: they help narrow the AI’s context, enforce quality rules, and stop risky modifications before deployment.

Characterization tests are particularly important when working with legacy code. These tests describe the current behavior of the system. They help the team check whether a change generated by AI violates existing business logic, even when the original application logic is not well documented.

Governance Mechanisms and Their Role

It is worth remembering, however, that governance is not a free layer of protection. ADRs, AI rules, and characterization tests require maintenance, updates, and clear team ownership. The lack of such mechanisms usually becomes visible only during system maintenance. The potential consequences? More difficult onboarding, higher risk after team turnover, slower debugging, lower team velocity, and greater dependence on the knowledge of individual people. When technical decisions do not make it into documentation or the review process, the IT manager is left with code whose behavior has to be reconstructed later — during a regression, a production incident, or the next release.

What Should AI Handle, and What Should Still Be Left to Classic Tools?

AI works well for:

• analyzing dependencies between modules,
• generating tests,
• preparing prototypes,
• automating multi-step processes.

Classic tools still win at:

• code formatting,
• style control,
• static analysis,
• repeatable deterministic operations.

This is where many organizations make a costly mistake: they replace proven tools with AI-based solutions that do not offer a real business advantage.

FAQ

What is the difference between vibe coding and supervised code generation?

Vibe coding is based on intuitive collaboration with AI and quick acceptance of generated changes. Supervised code generation requires safeguards such as review, tests, and checkpointing.

Why don’t large context windows solve legacy code problems?

A larger context window allows the model to see more code, but it does not guarantee correct understanding of dependencies between modules, the history of architectural decisions, or undocumented business rules.

How can teams reduce the risk of hallucinations during code generation?

The most effective approach is governance: code review, checkpointing, ADRs, characterization tests, and monitoring agent behavior.

Conclusion

Agent frameworks can improve team productivity, but they do not solve the problem of model nondeterminism on their own. The risk can only be reduced through architecture, process, context control, tests, review, and high-quality oversight. The goal of governance is not to limit AI productivity, but to preserve the benefits of automation without shifting the costs to future system maintenance.

These are the areas we work on at j‑labs in Data & AI projects delivered for organizations of different sizes and levels of technological maturity. If you want to validate your assumptions or discuss the direction of your solution, contact us.