Title: AI Hallucination: A New Vulnerability in Code Generation
Recent developments in artificial intelligence have unveiled a concerning phenomenon known as "package hallucination." This term refers to instances where large language models (LLMs) generate outputs that include factually incorrect or entirely irrelevant information. These inaccuracies have been a persistent issue for LLMs, undermining their reliability and effectiveness, and proving challenging to predict and address. A forthcoming study, to be presented at the 2025 USENIX Security Symposium, sheds light on this issue.
The researchers conducted extensive tests, utilizing 30 different scenarios—16 in Python and 14 in JavaScript—generating a staggering 576,000 code samples in total. Among the 2.23 million package references contained within these samples, 440,445 (approximately 19.7 percent) were links to non-existent packages. Notably, of these hallucinations, 205,474 had unique package names, which raises significant concerns.
One of the critical findings of this study is the frequency of repeated hallucinations. Approximately 43 percent of these non-existent package references were generated multiple times across various queries. The researchers indicated that 58 percent of hallucinations occurred more than once in a series of ten iterations. This pattern suggests that many hallucinations are not random anomalies but rather predictable occurrences that persist over multiple tries, thus posing a heightened risk for potential exploitation by malicious actors.
Malicious individuals could leverage this predictable behavior by identifying regularly hallucinated package names, then deploying malware using these fictitious identifiers. This strategy could lead to significant risks, particularly as these non-existent packages could be inadvertently accessed by numerous developers.
The research also highlighted differences in the frequency of package hallucinations across various LLMs and programming languages. Open-source models such as CodeLlama and DeepSeek exhibited an average hallucination rate of nearly 22 percent, while commercial models were considerably lower, with just over 5 percent. Furthermore, code written in Python had a lower average hallucination rate of nearly 16 percent compared to more than 21 percent for JavaScript.
As businesses increasingly rely on AI-generated code for streamlined development processes, understanding these hallucinations is crucial. The MITRE ATT&CK framework provides a lens through which one can analyze how these vulnerabilities could be exploited. Potential adversary tactics include initial access and persistence, as attackers may integrate these hallucinations into their strategies to access sensitive systems and data.
In summary, the emergence of package hallucination signifies a new route of vulnerability that numerous businesses in the tech sector need to address proactively. The implications for supply-chain security are substantial, as the very tools intended to enhance efficiency may also introduce significant risks if not carefully monitored and managed.
