Microsoft’s ambitious integration of generative AI, particularly through its Copilot feature within Microsoft 365 applications, promises a new era of productivity by allowing users to ask questions and retrieve relevant information from emails, Teams chats, and files. However, this innovative capability raises significant security concerns, particularly as malicious actors seek to exploit these very processes.
At the recent Black Hat security conference in Las Vegas, researcher Michael Bargury demonstrated several alarming proof-of-concept vulnerabilities within Copilot that could be manipulated by hackers. These vulnerabilities include using Copilot to generate false references to files, exfiltrate sensitive private data, and circumvent Microsoft’s security protections. One of the most striking examples presented was Bargury’s creation of “LOLCopilot,” a tool that can transform the AI into an automated spear-phishing machine.
When a hacker gains access to an individual’s work email, LOLCopilot can analyze the user’s communication patterns to identify frequent contacts. It can then draft personalized messages that mimic the user’s writing style, even incorporating specific emojis. This enables attackers to send out hundreds of tailored phishing emails within minutes—far more efficient than the traditional method of painstakingly crafting a single email to deceive a target. Bargury noted, “I can do this with everyone you have ever spoken to, and I can send hundreds of emails on your behalf,” underscoring the ease with which an attacker can exploit these capabilities.
The method underlying this attack showcases how large language models (LLMs), like those powering Copilot, can be manipulated. By issuing written prompts designed to extract data, attackers can exploit the AI’s inherent functionality while embedding malicious instructions. This highlights the challenges organizations face when integrating AI systems with corporate data, especially when untrusted or harmful information is introduced into the AI’s decision-making process. The AI may respond with what appears to be legitimate results, making it difficult for users to discern between accurate and malicious outputs.
In addition to spear-phishing, Bargury demonstrated another vulnerability in which a hacker who has hijacked an email account can discreetly request sensitive information, such as employee salaries, without triggering Microsoft’s protective measures. By carefully crafting the prompt to omit references to the source files, the hacker can extract this information without raising alarms. Bargury remarked that “a bit of bullying does help,” implying that a manipulative prompt can yield the desired results from the AI.
Other demonstrations revealed further avenues for exploitation. For instance, a hacker who lacks access to email can still poison the AI’s knowledge base by sending a malicious email. This can lead the AI to provide inaccurate banking information, potentially endangering financial transactions. Bargury cautioned, “Every time you give AI access to data, that is a way for an attacker to get in,” highlighting the ongoing risks associated with AI data access. Bargury’s research also indicated that an external hacker could extract information about the potential outcomes of upcoming company earnings calls. In a final demonstration, he showcased how attackers could leverage Copilot to function as a “malicious insider,” directing users to phishing websites and further compromising organizational security.
In response to these findings, Phillip Misner, head of AI incident detection and response at Microsoft, acknowledged the importance of identifying such vulnerabilities and confirmed that the company is actively assessing the implications of Bargury’s research. Misner stated, “The risks of post-compromise abuse of AI are similar to other post-compromise techniques,” emphasizing the necessity of implementing robust security measures across various environments and identities.
As generative AI systems, including Microsoft’s Copilot, OpenAI’s ChatGPT, and Google’s Gemini, evolve, they are increasingly tasked with completing complex operations, such as scheduling meetings and managing online transactions. However, security experts have consistently raised concerns about the integration of external data into these systems, as it can lead to indirect prompt injection and data poisoning attacks. Johann Rehberger, a security researcher and red team director, remarked, “I think it’s not that well understood how much more effective an attacker can actually become now,” emphasizing the need for vigilance regarding what LLMs produce and communicate to users.
Despite Microsoft’s significant efforts to safeguard Copilot from prompt injection attacks, Bargury discovered methods to exploit the system by understanding its underlying architecture. This involved extracting the internal system prompt and analyzing how it accesses enterprise resources. He observed that while Microsoft has implemented various controls, specific prompts can bypass these restrictions, allowing an attacker to exploit the system for their purposes. “You talk to Copilot and it’s a limited conversation because Microsoft has put a lot of controls,” he explained. “But once you use a few magic words, it opens up and you can do whatever you want.”
Both Rehberger and Bargury underscored the critical importance of monitoring AI outputs and their interactions with sensitive data. Rehberger pointed out that many security issues stem from organizations granting excessive access to files without proper permissions, a long-standing challenge in corporate security. He warned, “Now imagine you put Copilot on top of that problem,” suggesting that the integration of AI could exacerbate existing vulnerabilities.
To effectively mitigate these risks, organizations must adopt a proactive approach to security that includes continuous monitoring of AI-generated outputs and a comprehensive understanding of how AI systems operate within their environments. Bargury emphasized, “The risk is about how AI interacts with your environment, how it interacts with your data, how it performs operations on your behalf.” Organizations need to ensure clarity regarding what AI agents do on behalf of users and whether those actions align with user intent.
In summary, while Microsoft’s Copilot offers significant potential for enhancing productivity, it also introduces critical security vulnerabilities that must be addressed. As AI systems continue to advance and become more integrated into daily workflows, organizations must prioritize security measures and remain vigilant against the evolving tactics employed by cybercriminals.