Attack Vectors on RAG Applications

In the previous posts, I talked extensively about RAG (Retrieval Augmented Generation) applications, and some of the design patterns to build a fast and cheap RAG app. In today’s issue, I will walk you through some security attacks that RAG apps are particularly vulnerable to.

These attack types are varied in feasibility and effectiveness, but one must be aware of all of them and put effort into avoiding them, especially while building critical RAG application.

Attack Vectors on RAG applications

Data poisoning

Occurs when an attacker influences a LLM response by inserting harmful content into the RAG data sourced from external, untrustworthy sources. This action alters the model's parameters and introduces biases. An attacker could similarly tamper with trusted data (for example, a corporation's internal documents) or untrusted data (such as information from the internet) if the model uses this data in real-time to answer queries (like when integrating an OpenAI plugin or Azure Cognitive Service index).

Example

Data poisoning of a web RAG data. In this scenario, the attacker can inject malicious links in web pages that are susceptible to be used by RAG applications. The LLM then queries these web pages and returns malicious links to the end user.

Prompt injection

Prompt injection is a method where an attacker, having access to the LLM, crafts a specific prompt to compromise users, steal confidential data, or misuse the model. In certain scenarios, this could also lead to executing code or revealing information to other users, highlighting the need for robust security measures.

Hallucination abuse

An attacker can prompt an LLM repeatedly to identify if it mentions any malicious links. If so, the attacker can lead users to the same prompts that lead to malicious links being mentioned by the LLM. This attack is particularly unlikely, but not impossible, especially against less powerful LLMs.

Security considerations

To mitigate the security risks we outlined above, we suggest you pay close attention to these implementation details. These suggestions follow our experience building HeyCloud, a secure RAG system for devops use cases https://www.heycloud.ai/.

Prompt Handling

Handling prompts well means making sure that the questions or commands given to the system are safe and won't cause any problems. Before the system starts working on a prompt, it should look at it carefully to check for anything weird or signs that someone might be trying to trick it. The system should be smart enough to tell if a prompt looks suspicious and either fix it or not use it. This helps stop troublemakers from making the system do something it shouldn't, like giving away secrets or doing something harmful.

Llama Guard is a great starting point to do exactly this. https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/

Data handling

RAG apps use external and internal data to answer questions or perform search. Just like input prompts, this data must be sanitised to ensure no harm to end user. One way to do this is: during retrieval, once the closest chunk is retrieved, the prompt composer should verify the integrity and safety of data.

Isolation

Keeping users' data separate from each other is really important. This way, one user's information can't accidentally or purposely get mixed up with someone else's. This can be achieved by using a multi-tenant server that authenticates users, grants access to data and only uses the LLM with a prompt which is composed following access rules.

Observability

Being able to see what's going on inside the system is key to catching and fixing problems fast. This means keeping an eye on how the system is working and being on the lookout for anything unusual. If something odd pops up, like the system starting to act in a way it shouldn't, noticing it quickly helps fix things before they get worse.

Conclusion

HeyCloud serves clients with non-negotiable security requirements, so we put a lot of effort into making every bit of our infrastructure as secure as possible.

We are building the ultimate AI copilot for DevOps teams, so we face a particular set of security threats that are not common in traditional applications.

However, since these threats concern most LLM-RAG systems, we thought it may be helpful to write about our learnings. Hope today’s issue provided some useful insights for your LLM-RAG journey!