Announcing Prompt Security’s Automated AI Red Teaming

Stress-test homegrown LLM and agentic applications before attackers do.

‍

AI applications are moving from experimentation to production - and with that shift comes a fundamentally new attack surface.

Traditional application security still matters, but it was not designed for systems that reason over untrusted input, take autonomous actions, and interact with tools, data, and users in unpredictable ways. Large language models (LLMs) and agentic workflows can be manipulated through adversarial prompts, unexpected inputs, and tool misuse, creating failures that span security, safety, and operational risk all at once.

The Prompt Security team has spent the last couple of years protecting homegrown AI applications at some of the world’s largest companies. As the scope and impact of AI systems continue to grow, it became clear that runtime protection alone is not enough. Teams need a way to proactively break their AI systems before attackers do.

Today, we’re announcing the beta release of Prompt Security’s Automated AI Red Teaming for homegrown AI applications, available to selected customers.

We built Automated AI Red Teaming to give product and security teams a structured, repeatable way to simulate real-world attacks, identify the vulnerabilities that actually matter, and validate fixes over time - so AI applications can be shipped with confidence, not guesswork.

What you can do with Prompt Security’s Automated AI Red Teaming

With Automated AI Red Teaming, you can:

• Test against real-world AI risks, including prompt injection and jailbreaks, sensitive data exposure, system prompt disclosure, harmful or policy-violating content, and resource abuse.
• Stress-test agents and tool-enabled workflows, where models can take actions, call tools, and trigger downstream impact - not just generate text.
• Get actionable results, including evidence, a clear risk score, and remediation guidance that helps teams move from “what broke” to “how to reduce risk.”
• Remediate and prevent discovered vulnerabilities using Prompt Security for Homegrown AI Applications, where you can quickly configure guardrails to block any vulnerabilities found during red teaming - closing the loop between testing and enforcement.
• Deploy flexibly in SaaS or on-premise environments.

What AI red teaming is (and is not)

AI red teaming applies a familiar security principle: adopt an attacker’s mindset, probe for weaknesses, and document what breaks. In AI systems, these probes often take the form of malicious prompts and crafted inputs designed to override instructions, extract data, or steer behavior into unsafe states.

It is important to distinguish model testing from application red teaming:

• Model testing and benchmarks evaluate a model’s behavior on predefined, generic tasks. They are useful for comparing models and understanding baseline capabilities, but they operate outside of your real business and technical environment.
• Application-level red teaming is context-specific. It focuses on how your AI application behaves when deployed with real prompts, real data sources, real tools, real guardrails, and real users. This is where vulnerabilities emerge - from application logic, prompt design, integrations, permissions, and deployment choices - not from the model in isolation.

In other words, a model may perform well on benchmarks and still fail dangerously once embedded in a real product. AI Red teaming is how you discover what your actual application is vulnerable to, in your specific context.

The threats you should be testing

AI applications can fail across security, safety, and reliability dimensions. Common red teaming categories include:

• Prompt injection and jailbreaks: attempts to override system instructions and guardrails.
• Sensitive data exposure: extracting secrets, PII, or internal context that should not be revealed.
• System prompt disclosure: coercing the model to reveal hidden instructions that enable follow-on attacks.
• Harmful content and misuse: generating disallowed or dangerous content, including harassment, hate, self-harm, or wrongdoing enablement.
• Resource abuse: triggering excessive tool calls, runaway loops, or high compute usage.

If your system includes tools and actions - such as agents calling APIs, accessing data stores, sending messages, or executing workflows - the risk increases. The model is no longer only producing text; it can take actions in the world. This makes testing for unsafe tool use, unexpected execution paths, and “rogue agent” behavior essential.

How Automated AI Red Teaming works

Prompt Security’s red teaming approach is built around a few core building blocks:

• Target: the system you are testing, such as a chat UI, API endpoint, RAG workflow, or agent.
• Attack: a single malicious prompt designed to provoke a specific failure mode.
• Scan: an orchestrated run of many attacks against a target, with results recorded and scored.

A typical workflow looks like this:

1. Define the target
   Specify what you are testing, such as a chat interface, RAG endpoint, or agent workflow.
   
   
2. Select threat coverage
   Choose the categories you want to simulate, such as prompt injection, sensitive data exposure, or system prompt disclosure.
   
   
3. Run an automated scan
   The platform executes a structured set of adversarial tests and captures outputs and supporting evidence.
   
   
4. Get actionable guidance
   Each scan includes specific remediation guidance to help teams fix issues effectively.
   
   
5. Retest continuously
   Rerun scans after changes to validate fixes and monitor drift, regressions, and newly introduced vulnerabilities.
   
   

Why we built this (and why runtime enforcement still matters)

AI red teaming is not a checkbox exercise. When done well, it becomes a core product discipline - a way to deliberately break systems before users or attackers do, harden real behavior, and reduce risk before it reaches production.

But pre-production testing on its own is not enough. AI systems change continuously once they are live. Inputs shift, usage patterns evolve, and new attack techniques emerge. Runtime enforcement is what catches drift, novel abuse, and unintended misalignment as they happen. Red teaming helps teams find and fix the most important issues early; runtime controls are what keep systems safe after launch.

For existing Prompt Security customers, Automated AI Red Teaming extends the same security mindset you already apply in production upstream into development and testing - using familiar deployment models, reporting workflows, and a clear line of sight between red teaming findings and the runtime controls you already rely on.

For organizations not yet using Prompt Security, Automated AI Red Teaming provides a focused entry point: a way to start stress-testing your most critical homegrown AI workflows before they reach production. As your AI footprint grows, coverage can expand alongside it - and when you’re ready to enforce controls at runtime, the same platform scales to deliver continuous protection across your entire AI attack surface.

‍