LLM Security & Data Privacy Compliance Middleware: a gateway for safe AI apps

The moment an organization plugs an LLM into its workflow, that LLM inherits every prompt your users type — and everything those prompts might leak. Employee data in a chat tool, customer details pasted into a support bot, a prompt that tells the model to ignore its instructions. Most applications have no centralized layer that enforces safety and compliance controls on user prompts before they reach the model.

This project — my second of two for Round 2 at Shentinelix Sphere — is that layer: the LLM Security & Data Privacy Compliance Middleware, a security gateway that sits between client requests and the LLM.

What it checks

Every incoming prompt passes through a security analysis engine that detects:

• PII — names, emails, phones, financial identifiers — and redacts them
• Toxicity — scored to decide prompt acceptance
• Prompt injection & jailbreak attempts — via heuristics and regex
• Malicious URLs and policy violations buried in the prompt

The decision engine then resolves each prompt to one of three outcomes: Allow, Block, or Redact — safe prompts are forwarded to the LLM (the LLM API), everything else is stopped or scrubbed first.

The compliance copilot — every prompt is assessed, and the verdict and risk score are shown before anything reaches the model.

The gateway architecture

Client Requests
  → Authentication & RBAC layer (JWT)
  → Security Analysis Engine
       · PII detection (Presidio NER)
       · Toxicity scoring (Detoxify)
       · Prompt injection detection
       · Threat-intelligence checks
  → Decision Engine (Allow / Block / Redact)
  → LLM API call
  → Audit-logging database

A prompt-injection attempt caught at the gate — blocked before the model ever sees it.

Two details in that stack are worth calling out:

1. Presidio (NER) for PII — instead of a fragile regex list, named-entity recognition finds sensitive data even when it's phrased unpredictably.
2. Immutable audit logs — every request is traceable end to end, which is what turns a filter into a compliance tool.

Implementation details

• Prompts validated with Pydantic schemas before anything else runs
• PII entities detected with NER models and redacted before LLM processing
• Toxicity scores determine prompt acceptance
• Injection patterns caught with heuristics and regex
• Only safe prompts are forwarded to the model

Security features

• JWT-based authentication with role enforcement (RBAC) — different roles get different capabilities
• Rate limiting to prevent abuse and runaway token cost
• Secure HTTP headers
• Encrypted storage of API keys
• Immutable audit logs for compliance

Compliance & privacy

The middleware was designed with data-protection principles in mind: PII is redacted before the LLM ever sees it, full request traceability is preserved, and security thresholds are configurable — so a stricter profile can be applied to compliance-sensitive deployments without code changes.

Tech stack

Python 3.10+ · FastAPI · Presidio (PII) · Detoxify (toxicity) · LLM API · JWT · RBAC · Rate limiting · SQLite · Docker / Docker Compose · Pytest

Testing

• Comprehensive unit test coverage
• Security-feature validation tests
• Performance benchmarking
• Docker-based deployment testing

Use cases

Enterprise AI applications · chatbot platforms · internal LLM tools · compliance-sensitive AI systems

Repo: github.com/SabarishR08/llm-guardian

(This middleware has since been consolidated into LLM Guardian — the compliance modes, URL-guard and MCP tool-call defenses all live in that one repo now.)

Lessons that stuck

• Defense in depth beats one clever layer. Regex catches the classics, NER and ML catch the paraphrases; neither alone is enough.
• Rate limits are a security control, not just a cost control.
• Keep a battery of attack payloads as regression tests — the only way prompt defenses stay alive is by re-running the attacks you fear most after every change.