Recent investigations have revealed the presence of up to 100 malicious artificial intelligence and machine learning models hosted on the Hugging Face platform. These models pose significant risks, as they can execute unauthorized code through the loading of specific pickle files, according to software supply chain security firm JFrog.
Senior security researcher David Cohen highlighted that these models can create a backdoor into users’ systems, granting attackers a shell on the compromised device. This vulnerability enables full control over the victim’s machine, allowing for access to critical internal systems and potentially leading to large-scale data breaches and corporate espionage.
Notably, one rogue model was found to establish a reverse shell connection to an IP address linked to the Korea Research Environment Open Network, also known as KREONET, illustrating the global nature of this threat. Other models with similar payloads have been reported connecting to additional IP addresses.
In an unusual twist, the creators of one of these malevolent models advised users against downloading it, suggesting this publication may be linked to researchers rather than malicious actors. However, JFrog emphasizes that ethical guidelines in security research prohibit the dissemination of active exploits or harmful code, a principle that was ignored in this instance when the malicious code attempted to connect to a legitimate IP address.
These findings highlight the emerging threat landscape within open-source repositories, where harmful code can be hidden amid legitimate resources. As researchers develop increasingly sophisticated prompt injection techniques, these vulnerabilities are becoming more pronounced.
For instance, a new generative AI worm, named Morris II, has been created to steal data and propagate malware across systems. This worm builds upon the concept of the original Morris worm by exploiting adversarial prompts embedded in various media like images and text, which, when processed by generative AI models, can trigger self-replication and malicious actions.
The ComPromptMized attack technique takes this a step further, resembling traditional exploits such as buffer overflows and SQL injections by embedding commands within a query. This method effectively targets applications dependent on generative AI services and those utilizing retrieval-augmented generation for enhanced responses.
The broader implications of these developments raise concerns regarding the robustness of generative AI systems against such adversarial tactics. Previous studies have indicated vulnerabilities to adversarial perturbations in multi-modal LLMs, which can distort outputs based on unrecognized inputs, whether through images or audio files. Researchers continue to explore these vulnerabilities, emphasizing the critical need for enhanced security measures in AI deployments.
As the threat landscape evolves, business owners must remain vigilant and proactive in safeguarding their systems against potential intrusions that exploit these sophisticated techniques. Recognizing that initial access, persistence, and privilege escalation tactics may be utilized in these attacks is essential for implementing effective defenses and minimizing their impact.
