AI red teaming is structured adversarial testing of an AI system. The goal is to find failures in security, safety and responsible AI before attackers or users do. It borrows the mindset of traditional red teaming but changes the targets and the techniques.
How It Differs from Traditional Pentesting
- Targets include model behaviour, not only code and infrastructure
- Results are probabilistic: an attack may work on one attempt in ten, so success rates matter
- It tests safety and fairness harms as well as security flaws
- Fixes often involve prompts, filters, grounding and architecture rather than a single patch
The Three Categories
A useful way to organise a test is by three categories: security (can an attacker steal data or take actions?), safety (can the system be made to produce harmful content?) and responsible AI (does it behave fairly, accurately and transparently?).
Stages of an Engagement
- Scope and objectives: what harm matters to the business?
- Threat model: use MITRE ATLAS and the OWASP LLM Top 10 to choose tests.
- Attack: manual prompts plus automation.
- Score: measure how often attacks work and how severe the results are.
- Report: findings, evidence and specific fixes.
- Re-test after fixes.
garak: An Open-Source LLM Vulnerability Scanner
garak runs probes against a model, covering prompt injection, jailbreaks, toxicity and data leakage, and produces a report of how often each probe succeeded. It is a good way to get a baseline quickly.
PyRIT: Microsoft's Python Risk Identification Tool
PyRIT is the framework Microsoft's AI Red Team uses to automate attacks. It uses orchestrators to drive conversations, converters to transform prompts, targets to represent the system under test and scorers to judge responses. It scales manual ideas into repeatable experiments.
Automation Is Not Enough
Tools find known weaknesses fast. Creative, context-specific attacks still need a human who understands the application. Good engagements use both.
Our AI Security course teaches garak and PyRIT in dedicated modules, and our AI red teaming service applies the method to your systems.
