Processing submissions that resist prompt injection
Protecting the integrity of a government agent that reads, summarizes, clusters, or ranks user submissions from adversarial content inside those submissions, through four structural controls that hold regardless of how the injection is worded.
The impact of agents
As agencies use their own agents to process submissions at scale, the content of a submission becomes a channel for manipulating the agent that reads it. Indirect prompt injection (instructions planted in the very text an agent ingests) is the top-ranked risk for language-model applications, and national cyber agencies warn it may never be fully mitigated, because a model cannot reliably tell the instructions it was given from the data it was asked to read. For policymakers this lands squarely on the duty to consider what was submitted and to defend the resulting decision: an agency must be able to stand behind a reading produced, in part, by an agent that ingested content specifically crafted to steer it.
What must be verified
When an agency uses its own agent to read, summarize, cluster, or rank user submissions, it needs confidence that the agent's reading reflects what the submissions say and not instructions hidden inside them. Content a submitter controls can carry directions that alter how the processing agent behaves, and by the standards bodies' own account this cannot be fully filtered out. The confidence government needs is therefore structural and procedural, not a claim of immunity.
Protecting access
The defense can exclude the honest submitter if it is drawn too tightly. A genuine submission whose unusual phrasing trips an adversarial-content filter is redirected or denied consideration on a false signal. This is more likely for someone writing in a second language, with low literacy, or using assistive technology. Over-reliance on the agent's own ranking compounds the risk: a submission the agent scores as low-relevance because of unfamiliar phrasing is ranked below submissions it reads easily, without anyone deciding to exclude it.
Keep the path open by never rejecting a submission on a suspected-injection signal alone: treat the signal as data for human follow-up rather than grounds for exclusion, route anything the agent flags or deprioritizes to human review, and keep a standing route for a submitter to have their submission considered by a person.
Response surface
Submissions are untrusted data the agent may read but never obey, the agent holds no authority to act, a person owns the consequential step, and everything is logged — a submission that trips a suspected-injection signal is routed to a human, never excluded on the flag alone.
The embedded instruction had no effect. The submission still counts, and a person decides how to treat it.
Prompt injection can’t be fully filtered, so the pipeline limits the damage instead: untrusted data, no agent authority, a human on the consequential step, and a log. A flag routes a submission to a person, never out of the record — unusual phrasing from a second-language writer trips the same signal.
Maturity
- Emerging
For the secure-development scaffolding, where treating ingested content as untrusted, least privilege, and human oversight are settled guidance from national cyber agencies.
- Frontier Headline
For a government submission-processing pipeline hardened against adversarial content, where no published deployment documents a control specific to injected submissions.
Precedents
The risk is defined and ranked at the standards level. Prompt injection is the top-ranked language-model application risk, split into direct (the user's own input) and indirect (instructions planted in content the agent later reads), and its canonical treatment states plainly that fool-proof prevention is unclear (OWASP LLM01:2025). The adversarial-ML taxonomy of the US standards body adds indirect prompt injection as a defined subcategory (NIST AI 100-2e2025), and the seminal research showed a language-model application can be compromised by injecting prompts into data it is likely to retrieve (Greshake et al., peer-reviewed at AISec '23). For an agency, that retrieved data is the user's submission.
A national cyber agency frames every control as risk-reduction, not a solution. The UK NCSC states prompt injection "may never be totally mitigated" the way SQL injection can be, and counsels secure design over silver-bullet solutions. The joint international Guidelines for Secure AI System Development set the surrounding frame: secure design, development, deployment, and operation on a secure-by-default basis. The mitigations that follow — segregate and mark external content as untrusted, validate output against an expected shape, enforce least privilege, keep a human on consequential actions, and log inputs, outputs, and tool calls — each reduce the risk; none removes it.
A government tool already reads user submissions with a language model — and its safeguard is human oversight. The UK's Consult analyzes large volumes of consultation responses and presents themes to human reviewers rather than deciding, aligned to the principles governing lawful consultation. Its published safeguard is human-in-the-loop for accuracy and legitimacy; no published control specific to adversarial or injected submissions is documented, which marks the exact gap this pattern names.
What carries over to agent use
The mitigations transfer directly to a submission-processing pipeline, and the honest posture is that they harden it rather than seal it: the four controls carry over unchanged, alongside validating the agent's output against the shape a triage step expects. A real-world study of a ranking pipeline found a measurable fraction of inputs already carried hidden injections, most not phrased as obvious commands, so the assumption to design on is that some injected content will arrive and some will get through.
The unresolved part is that no published government deployment documents a control specific to injected submissions, and the standards bodies agree no control makes the risk go away. That is why the hardened pipeline is frontier even though its component practices are settled secure-development guidance. It is also why the durable answer leans on keeping a human on the consequential decision rather than on any claim to have filtered the adversarial content out.
Where things go wrong
The failure mode is an agency acting on a reading of the submissions that a submission covertly steered. A crafted entry in the pile can direct the processing agent to misrepresent the weight of opinion, drop a category of response, or emit content the agent was never meant to produce. Because the manipulation rides inside ordinary-looking text, it is invisible in the summary the agency relies on. Without a log of what the agent did, it stays invisible afterward too, leaving a decision defended on a reading no one can reconstruct. The four controls are what keep a manipulated reading from becoming a manipulated decision.
Sources
6 references
- OWASP LLM01:2025 — Prompt Injection
- NIST AI 100-2e2025 — Adversarial Machine Learning: A Taxonomy and Terminology
- UK NCSC — prompt injection may never be fully mitigated
- Guidelines for Secure AI System Development (NCSC, CISA and partners)
- UK Consult — evaluating an AI tool for public consultation analysis
Reporting and analysis about the subject rather than the primary framework. Treat it as a lead and corroborate before relying on it.