Researchers from the University of Texas have uncovered significant security vulnerabilities in the backend technology used by tools like Microsoft Copilot. These issues, found in retrieval-augmented generation (RAG) systems, could lead to serious data leaks and other security risks for enterprise users.
The group of five researchers identified a class of vulnerabilities they call “ConfusedPilot,” which they believe could “confuse” Microsoft Copilot for Microsoft 365, leading to breaches of confidentiality.
According to the researchers, RAG models are prone to a problem known as the “confused deputy” issue. This occurs when an entity within an organization, which does not have permission to perform a certain action, tricks a more privileged entity into doing it on its behalf.
The team demonstrated two main types of vulnerabilities. The first involves embedding malicious text into a prompt, which then corrupts the responses generated by the large language model (LLM) that powers Copilot.
The second vulnerability concerns the “leaking of secret data” by exploiting the caching mechanism during data retrieval. The researchers also warned that these vulnerabilities could be used together to spread misinformation within an organization.
One of the most concerning aspects of these vulnerabilities is the threat posed by insiders. The report highlighted the risk of an employee using these flaws to access information they shouldn’t be able to see. For example, a malicious actor could create a fake sales report with false information, which would then influence Copilot’s decision-making. The fake report could even contain instructions that cause Copilot to behave differently when it accesses the data.
The researchers emphasized the risks associated with RAG systems and raised serious concerns for enterprises that rely on popular AI tools like Copilot.
“While RAG-based systems like Copilot offer significant benefits to enterprises in terms of efficiency in their everyday tasks, they also introduce new layers of risk that must be managed,” the team stated.
Andrew Bolster, a senior research and development manager of data science at Synopsys, echoed these concerns. He pointed out that the vulnerabilities discovered in Microsoft Copilot could potentially affect all RAG systems.
“Copilot for Microsoft 365 is the demonstrated target for this attack, but it’s not alone in this threat model. These same attacks apply to many enterprise RAG systems where there is permissive internal access to data,” Bolster said.
Bolster also stressed the importance of data governance in preventing such security issues.
He noted that the adoption of generative AI must be accompanied by “thoughtful and well-structured” data governance strategies. This would ensure proper “separations” exist when it comes to the data accessible to RAG systems, thereby minimizing the risk of corruption or misuse.
“Much the same way that leaders establish verification and approval chains for public marketing publications or technology documentation, internal knowledge bases should maintain mechanisms for persisting data lineage and approval status for being included in global RAG,” Bolster advised.