AI-assisted software development has become standard practice in many engineering organizations, but the security implications of generated code remain highly uneven. New research from Secure Code Warrior, developed in collaboration with RMIT University, attempts to quantify these risks through a new AI Trust Index. The study examined 1,760 complete codebases generated by 16 frontier large language models from vendors including OpenAI, Anthropic, Google, and others. The results show that AI-generated code is not uniformly risky, nor is any single model universally safer than its peers. Instead, the security profile of generated code is shaped most strongly by the framework it is paired with.

What the AI Trust Index Measures

The AI Trust Index is a web-based body of data designed to help organizations understand the security risks introduced by LLM-powered coding tools. It was built on a methodology created in partnership with RMIT University and published by Secure Code Warrior. Researchers evaluated 1,760 complete codebases produced by 16 frontier models across 11 development frameworks. The goal was not to rank models for raw coding ability but to assess how much security risk each model generates relative to others in specific contexts.

The findings are sobering. AI-generated code contained an average of 15 confirmed vulnerabilities per codebase, including 4.3 vulnerabilities rated critical or high severity. In total, more than 27,000 vulnerabilities were identified across all codebases. These numbers highlight the scale of the challenge facing development teams that rely heavily on AI assistants. While efficiency gains are real, they come with a significant security burden that must be managed deliberately.

No Universal AI Coding Winner

Researchers assigned each model a trust score on a scale of 0 to 100, with 100 indicating the fewest vulnerabilities found in testing its generated code relative to other models. Notably, there was no clear winner among major AI vendors. Two Anthropic Claude models, Sonnet 5 and Fable 5, scored best with trust scores of 80.4 and 76.4 respectively. OpenAI's GPT 5 Mini finished in last place with a score of 21.6. Overall, models from OpenAI, Anthropic, and Google appeared near both the top and bottom of the list, demonstrating that vendor reputation alone is not a reliable indicator of coding security.

More importantly, the research found that every AI model has a distinct security fingerprint. Across all evaluated frameworks, models consistently exhibited recurring vulnerability patterns rather than random failures. The report states that each model produces a repeatable mix of OWASP vulnerability categories, enabling organizations to anticipate where security weaknesses are most likely to occur. This predictability is a double-edged sword: it means attackers who know a model's patterns can exploit them, but it also means defenders can prepare targeted mitigations.

Framework Pairing Matters More Than Model Choice

One of the most actionable insights from the research is that a model's security performance varies significantly across frameworks. Secure Code Warrior found that a model that performs well in one framework, such as Django or React, may deliver mediocre performance in another or even miss security controls entirely. For example, Claude Opus 4.8 earned a perfect 100 trust score when tested against Django, but dropped to 57.3 in C-Basic and 28.5 in C-Basic.

This variation means that organizations cannot simply adopt a single 'safest' AI model and assume uniform protection. Instead, they must evaluate how specific models interact with the frameworks their developers actually use. The zoomed-out overall trust score matters less than the model-framework pairing. The research identified 11 development frameworks in all, and the differences between the safest and riskiest combinations were dramatic. According to Secure Code Warrior CEO and co-founder Pieter Danhieux, the highest-scoring frameworks were 40 times less risky than the lowest-scoring ones.

JavaScript and Java EE/JSP carried much of the exposure, while C and Java Spring barely registered on the risk scale. This is a critical finding for organizations that have standardized on JavaScript for web development or Java EE for enterprise systems. It suggests that teams working in these environments need extra security controls, code review, and automated scanning when using AI-generated code.

Predictable Failures and Common Weaknesses

The research also revealed that the most common security failures resulted from things models forgot to do, rather than from actively dangerous code. Authentication checks and input validation were frequently missing, creating vulnerabilities that an attentive developer might have caught. These are not new threat vectors; they are classic weaknesses that application security teams have been addressing for years. But AI-generated code tends to produce them at scale and with consistent patterns.

Across all models tested, 17 Common Weakness Enumerations (CWEs) appeared in every one of the 16 models. This near-universal presence indicates a systematic gap in current AI training and code generation. The report highlighted five weaknesses that carry the most volume: sensitive data leaking into logs, cross-site scripting, hard-coded credentials, predictable session tokens and reset codes, and path traversal. These five categories alone account for a large share of the vulnerabilities found in AI-generated code.

Because these weaknesses are predictable, organizations can build targeted defenses. The report emphasizes that risk is overall predictive rather than random; the specific model and framework combination made it more or less predictable how many vulnerabilities and which types of vulnerabilities researchers would find. This means security teams can profile the AI tools they use and integrate that knowledge into their development pipelines.

Cost Is Not a Security Signal

The study also examined whether more expensive AI coding tools offer better security. Researchers found little to no correlation between model usage cost and security. Spending more on a premium model does not automatically reduce the number of vulnerabilities in generated code. The security outcome is driven more by the model's architecture, training data, and its interaction with the framework than by its price tag.

This is an important consideration for organizations budgeting for AI development tools. The most expensive option may not be the safest choice for a given environment. Conversely, a lower-cost model could perform well in one framework while needing additional oversight in another. The data reinforces the need to evaluate AI coding tools in the specific contexts where they will be used, rather than relying on benchmarks that treat all frameworks the same.

Implications and Guidance for Security Teams

The findings do not mean organizations should avoid AI coding tools entirely. Instead, they suggest a more disciplined approach. Pieter Danhieux told that organizations cannot personally audit every team's framework choice, but they can require that any team building in a high-risk environment have a security gate and training requirements in place before code ships. The same way organizations demand extra scrutiny for other high-risk systems, AI-assisted development in JavaScript or Java EE should trigger additional review.

Danhieux also recommends prioritizing training investments around actual universal situations rather than trying to cover every scenario a developer might encounter. With 17 CWEs appearing in every model tested, it is impractical to mandate remediation for all of them at once. Instead, he advises CISOs to prioritize the five that carry the most volume: sensitive data leaking into logs, cross-site scripting, hard-coded credentials, predictable session tokens and reset codes, and path traversal. None of these is an insurmountable threat vector, and AppSec teams have had playbooks for all five for years. The job now is making sure those playbooks get pointed at AI-generated code with the same rigor as anything else.

For organizations looking to operationalize these insights, several steps are essential. First, security teams should map the AI models their developers use and assess which frameworks those models handle best. Second, they should integrate automated security scanning into CI/CD pipelines specifically for AI-generated code, looking for the recurring vulnerability patterns identified by the study. Third, training programs should focus on the five most common weaknesses, ensuring developers understand how to spot and fix them in generated code. Fourth, high-risk frameworks like JavaScript and Java EE/JSP should have mandatory security gates before deployment.

The emergence of AI coding assistants has transformed software development, but it has also introduced new levels of risk that are not evenly distributed. As this research demonstrates, the choice of AI model matters, but the choice of framework pairing matters more. By understanding the security fingerprint of their AI tools and taking targeted action, organizations can enjoy the productivity gains of AI-generated code without accepting unmanageable vulnerability levels. The data gives security leaders a roadmap: focus on the weakest combinations, enforce security gates in high-risk environments, and train developers on the most common mistakes that AI models make.

Source: Dark Reading News