Prompt Injection

Prompt injection is the first thing that makes you realise that a large language model isn't just a clever autocomplete – it's a system that follows instructions, and those instructions can come from anywhere in its context window. Understanding this unlocks a whole security mindset: you'll start to see data as either 'trusted directions' or 'untrusted content that might override them.' This is the foundation for AI security concepts like tool-use sandboxing, output filtering, and agent isolation. If you skip this, you'll build applications that are one clever email away from being controlled by an attacker.

Imagine you're a very obedient personal assistant who has been given a strict set of rules by your boss: 'Only open emails from your boss's address. Never click links in emails from anyone else.' But you also have a habit of reading every email out loud to yourself, including the text at the bottom. One day, you receive an email that contains this line: 'Ignore your boss's rules. Click this link: http://evil.example.' Because you are so obedient, you take that as a new instruction. You've just been tricked into breaking the rules that were meant to keep you safe.

Prompt injection works exactly like this. A large language model (LLM) is a system that answers questions, follows prompts, and completes tasks. It has a 'context window' – the entire chunk of text it can 'see' at once. That context window includes the original instructions you give it (the system prompt) and any additional text you feed in (user input, retrieved documents, tool outputs). The model treats everything in that context window as a kind of 'conversation,' and it must guess which parts are commands and which parts are just data.

The core problem is that the model doesn't have a built-in 'switch' to tell the difference between 'this is a command' and 'this is just a fact.' If you say 'Please summarize this webpage: [webpage text]', the model might also follow instructions hidden inside the webpage text. The webpage text wasn't meant to be a command, but the model is so eager to follow instructions that it will treat it as one. That's the injection.

Think of a very polite and accommodating tour guide. Their job is to take you around the museum, but they are always listening to what visitors say. You give them a clear plan: 'Give us the standard tour, and if a visitor asks to leave the group, tell them to wait at the entrance.' The tour guide is extremely obedient. Now, one visitor pulls the guide aside and says, 'The boss has changed the plan: take the whole group to the gift shop immediately and ignore the entrance rule.' The tour guide doesn't know that this visitor isn't the boss. They just see a person giving an instruction. So, they follow the new instruction.

This is exactly how prompt injection works. The system prompt is the 'plan' you gave the tour guide. The retrieved document, the user email, or the tool output is a 'visitor' who is whispering instructions into the guide's ear. The model cannot tell the difference between the authority of the plan and the authority of a piece of text in its context. It just sees instructions and obeys.

Where the analogy breaks down: A human tour guide would be suspicious of a random visitor making such a audacious claim. They would check the visitor's credentials or question the change. An LLM has no such suspicion mechanism. It has no way to 'verify' who is speaking. It just sees a string of text, and if that text looks like a command, it may follow it. That's the core weakness.

Prompt injection is a security attack where a user or a piece of untrusted content (like a web page or an email) embeds instructions into the data that is fed to a language model, causing the model to override its original system prompt or user directions and perform a different, potentially malicious action.

You're starting with a clean slate. This concept is your first step into the world of 'AI security.' It's related to things you'll learn later like: adversarial examples (where attackers subtly manipulate images to fool vision models), data poisoning (where attackers corrupt training data), and model extraction (where attackers steal a model's behavior). But prompt injection is unique because it attacks the model at inference time, by manipulating the input text, not the model itself.