In 2023, it was reported that security researchers at Trend Micro successfully utilized ChatGPT to generate malicious code through engagement with the AI as if it were a security analyst and penetration tester. This process enabled the model to produce sophisticated PowerShell scripts based on extensive databases of known malicious code.
According to expert Moussouris, this capability raises serious concerns about malware creation. She highlighted that the simplest way to bypass existing safeguards on AI models involves framing the request as a part of a capture-the-flag exercise, prompting the AI to generate harmful scripts.
The cybersecurity landscape has long grappled with the issue of unsophisticated attackers, often referred to as “script kiddies,” and AI technology may exacerbate this problem by lowering the entry barriers to cybercrime. Hayley Benedict, a Cyber Intelligence Analyst at RANE, emphasized that while novice hackers could exploit AI, the real threat may come from well-established hacking groups that can leverage AI to amplify their existing capabilities.
“The true risk lies with the hackers who already possess significant skills and operational experience,” Benedict remarked. This allows them to significantly enhance their cybercriminal activities, allowing for faster generation of complex malware. Moussouris echoed this sentiment, asserting that the rapid advancements facilitated by AI pose a substantial challenge to control cyber threats effectively.
Hunted Labs’ Smith expressed a similar viewpoint, underscoring that the real danger of AI-generated code arises when it is in the hands of highly knowledgeable individuals. Such expertise allows for the crafting of systems capable of bypassing multiple security layers, evolving malicious payloads adaptively during attacks. “This scenario would be exceedingly challenging to manage and mitigate,” he stated.
Smith envisioned a landscape where multiple zero-day vulnerabilities could be exploited simultaneously, amplifying the urgency of the cybersecurity threat. Moussouris confirmed that the tools needed to execute such sophisticated attacks are currently available, but cautioned that AI has not yet advanced enough for a neophyte hacker to deploy effectively without human oversight.
She noted, “We are still at a stage where AI cannot fully replicate the critical decision-making of human operators in offensive security.” The underlying anxiety regarding AI-generated code is the potential for widespread misuse; however, the reality is that it is highly skilled individuals who pose the most significant risk. Currently recognized as potentially the closest example of an autonomous “AI hacker” is XBOW, a product developed by a team of over 20 experts with extensive backgrounds in major tech firms.
Moreover, the ongoing duel of capabilities between malicious actors and cybersecurity professionals has been noted as an evolving dynamic in the industry. “The best defense against a malicious actor using AI is a skilled defender equipped with AI,” Benedict concluded, reiterating the importance of fortifying defenses against evolving threats.
Moussouris framed this advancement as the next iteration in the longstanding arms race in cybersecurity. She pointed out that the shift from manual hacking to the development of automated tools has fundamentally changed the tactics employed in cyberattacks. “AI is simply another instrument in the toolkit for those who can effectively harness its potential for protective or offensive measures,” she stated.
This ongoing evolution highlights the necessity for business owners to stay informed about the changing cybersecurity landscape, especially as it relates to emerging technologies and their implications for future attacks.
