Exposed AI Tooling: Censys Found 294,000 Open LLM Endpoints in Nine Months

A number worth pausing on

The Censys 2026 State of the Internet report counted more than 294,000 IP addresses running exposed AI and LLM tooling, spread across 43 distinct products. That figure grew by over 60% in nine months. Langflow and LiteLLM topped the list by volume.

Nobody deployed 294,000 public AI gateways on purpose. That number is the sum of a great many small decisions that each made sense at the time: a proof of concept on a cloud VM with a public IP, a docker-compose file whose port mapping was 0.0.0.0:3000 instead of 127.0.0.1:3000, a Kubernetes service that got a LoadBalancer because ClusterIP was harder to reach from a laptop. The tooling in this category is designed to be easy to stand up, which is exactly why so much of it is standing up in places nobody is watching.

The products that keep turning up

Forty-three products is a long tail, and the head of the distribution is where the work is. A handful of names account for most of the count, and those are the ones to check first, because they are the ones your teams are most likely to have installed on a whim.

Langflow instances grew 169% over the nine-month window, the fastest rise in the set. That growth sits next to a difficult security record. Censys counted 18 vulnerabilities in the project since 2024, 14 of them rated high severity, with four seeing observed exploitation. CVE-2025-3248 made the pattern concrete: an unauthenticated remote code execution flaw in the /api/v1/validate/code endpoint, scored 9.8, which parsed and executed user-supplied Python before checking whether the caller had any right to be there. CISA added it to the Known Exploited Vulnerabilities catalogue in May 2025 after attackers used it to install the Flodrix botnet, a Linux-targeting variant of the LeetHozer family. Version 1.3.0 closed it by requiring a valid token before the parser runs.

LiteLLM grew 97% across the same period. Ollama leads on raw volume: Censys recorded 7.23 million observations from 175,108 unique hosts in 130 countries over 293 days.

Different products fail differently, and the table below is the one to hand to whoever triages your scan results. The port column tells you what to scan for. The final column tells you what to put in the ticket.

Product	Common port	Authentication out of the box	What an unauthenticated instance hands over
Ollama	11434	None by design	Free inference on your GPU, the model list, and the ability to pull arbitrary models onto your disk
LiteLLM proxy	4000	A master key exists, frequently unset in quickstarts	Provider API keys, spend budgets, every downstream key the proxy issued
Langflow	7860	Enforced only in later versions	Flow definitions with embedded connection strings, plus code execution
Flowise	3000	Optional username and password via environment variables	The same class of exposure as Langflow: flows, credentials, custom code nodes
ComfyUI	8188	None	Workflow graphs, generated outputs, custom node execution, filesystem paths
Gradio demos	7860	None unless auth= is passed at launch	Whatever the demo function does, called by anyone who finds it
LangServe	8000	None	The chain, its prompts, and every tool the chain can reach
Open WebUI	8080 or 3000	The first account created becomes admin	Chat history, connected model backends, user accounts
vLLM and other OpenAI-compatible servers	8000	Optional --api-key	Free inference plus precise model identification
Jupyter	8888	Token, often disabled for convenience	Shell access as the notebook user
Qdrant, Chroma, and similar vector stores	6333, 8000	Off in common configurations	The embedded corpus, which is usually your internal documents
MCP servers	Varies	Transport-dependent, frequently none	Every tool the server exposes, executed with the server’s privileges

The pattern across those rows is that the products with the weakest authentication tend to have the most direct route to either compute or data. No exposure in this category is benign.

What an exposed instance actually gives away

The severity varies by product, and lumping them together as “exposed AI tools” hides the part that matters. Three distinct categories of loss show up.

The first is credential theft. An orchestration tool like LiteLLM exists to hold provider API keys and route requests to them. An unauthenticated instance is a free proxy to somebody else’s paid account, and the usual outcome is a five-figure bill from a model provider before anyone notices. The keys themselves are frequently readable through the same interface, which extends the problem beyond one runaway invoice.

The second is data. Workflow builders like Langflow keep prompts, retrieved documents, connection strings for vector stores, and often the contents of whatever knowledge base the team wired up. A retrieval pipeline built over an internal wiki turns into a search interface onto that wiki, available to anyone who finds the port.

The third is execution. Several tools in this category run code as a designed capability, whether that is a Python node in a flow, a tool-calling agent with shell access, or a plugin system. Reaching one of those without authentication is remote code execution with extra steps, and the steps are documented in the product’s own tutorial.

What the first hour of a compromise looks like

Walking the attacker’s path makes the priority ordering obvious. Discovery is automated and continuous, so the clock starts within hours of the port opening, not whenever a human happens to look.

An unauthenticated gateway gets probed for a model list, which confirms it is live and reveals which providers are wired up. The next request enumerates configured keys or simply starts issuing completions, because a working proxy is valuable whether or not the keys are extractable. If the tool exposes a flow editor or a node graph, the attacker reads the flows, and flows contain connection strings: a Postgres URL for the metadata store, an S3 bucket for documents, a vector database endpoint. Those secondary credentials are usually broader than the AI tooling itself, since somebody granted them during setup and never narrowed them.

Execution comes last and only when it is needed, because by that point the attacker often has database access without touching the code-execution path. The lesson from that ordering is that hardening the code-execution node while leaving the flow definitions readable solves the wrong half of the problem.

The reconnaissance is already running

That ordering is not speculative, and the traffic behind it has been measured. Researchers at Zenity Labs stood up honeypot backends imitating self-hosted LLM runtimes and logged what arrived between February and June 2026.

Across that window they recorded more than 1,400 distinct source addresses probing four runtimes: Ollama on 11434, LiteLLM on 4000, LangServe on 8000, and OpenClaw on 18789. Around 60,000 of the requests were liveness and version fingerprinting, from 235 addresses. Roughly 31,000 were model catalogue enumeration, from about a thousand addresses. A smaller and stranger set, 446 requests from 57 addresses, simply asked the model to identify itself, in Russian, Chinese, and English.

The paths involved make a short and stable list, which is what makes them useful as detection content:

/api/version          version fingerprinting
/api/tags             Ollama model enumeration
/v1/models            OpenAI-compatible model enumeration
/api/chat             Ollama native chat
/api/generate         Ollama native completion
/v1/chat/completions  OpenAI-compatible inference
/v1/messages          Anthropic-style inference

Two individual sources show what a discovered instance actually gets used for. One address sent the single word “Hi” to each of eight discovered Ollama models, 2,619 times per model, which reads as a capacity test rather than curiosity. Another cycled 40 API keys across 31 models on a LiteLLM instance, roughly 23,000 requests, which is credential validation at industrial scale.

Put those paths into your detection rules. A request for /api/tags arriving from an address outside your network is a high-confidence signal, because nobody inside your organisation reaches your inference stack from the public internet by accident.

Why this class of software ends up public

Traditional infrastructure got safer partly because the defaults got better. Databases stopped listening on all interfaces out of the box. Cloud storage buckets went private by default after enough public ones made the news.

AI tooling in 2026 sits roughly where those categories sat a decade ago. The products are young, the competitive pressure is toward a five-minute quickstart, and authentication is often a feature of the paid tier or a configuration flag rather than the default state. A quickstart that says docker run -p 3000:3000 is teaching every reader to bind to all interfaces, and most readers will paste it onto a cloud host without editing.

There is a second driver that has nothing to do with defaults. AI projects frequently start outside the normal path. A team gets budget for an experiment, spins something up in a personal cloud account or a corner of the org nobody inventories, demonstrates value, and the prototype quietly becomes load-bearing without ever passing through the review that a production service would face. Asset inventories built around known application teams do not see it, and vulnerability scanners scoped to known IP ranges do not scan it.

The path from quickstart to indexed host

Drawing the sequence out helps, because no single step in it is negligent. Each one is the shortest path to a working demo, and the exposure is the accumulated total.

Any one of those arrows breaks the chain. Changing the port mapping to 127.0.0.1:7860:7860 breaks it at B. A subnet with map_public_ip_on_launch set to false breaks it at C. A policy that refuses to attach a public address outside an approved account breaks it at D. Reviewing prototypes on a schedule breaks it at F. Nothing breaks it at H or I, which is the argument for spending your effort on the left half of the diagram.

The step that deserves more attention than it gets is E. A prototype that works is a prototype that stops being reviewed, because attention moves to the next thing. Whatever process you have for promoting a demo into a supported service is the process that has to carry the network question, and in most organisations that process does not exist for AI projects yet.

Shadow AI is the delivery mechanism

Exposed instances do not appear at random across an organisation. They cluster in the parts of it that build with AI outside the path a normal service would take, which currently means most of the organisation.

Survey numbers vary by how the question was asked, and the spread is instructive. PagerDuty’s 2026 shadow AI survey found that two-thirds of office professionals had used AI tools at work while believing company policy did not permit it. Verizon’s 2026 Data Breach Investigations Report put regular AI use on corporate devices at 45%. Salesforce’s 2026 workforce survey landed at 67% using AI tools at work. Whichever figure you prefer, the governance side of the ledger reads consistently across all of them: reporting through 2026 has repeatedly put the share of organisations with a formal AI security policy at under a fifth.

Most of that activity is somebody pasting a document into a chat interface, which is a data-loss problem rather than an attack-surface problem. The subset that matters here is smaller and worse: engineers self-hosting. A team that pulls the Ollama image onto a GPU box, or stands up Langflow to prototype an agent, has created an asset with a network identity, a set of credentials, and no owner in your configuration management database.

Finding that subset means looking for the traces self-hosting leaves rather than surveying people about their habits:

• GPU instance types appearing in cloud billing under cost centres with no machine learning history
• Container image pulls of ollama, langflow, flowise, comfyui, or open-webui in your registry proxy logs
• Egress from unfamiliar subnets to api.openai.com, api.anthropic.com, or generativelanguage.googleapis.com
• Corporate cards carrying recurring charges from model providers, which finance can pull faster than security can scan
• DNS records under your own domains resolving to addresses outside your known ranges

Billing is the highest-yield signal on that list and the one security teams ask for last. A monthly export of GPU spend and model provider charges, joined against your asset inventory, finds shadow AI infrastructure in an afternoon. The teams running it are rarely hiding anything; they simply never had a reason to tell you.

Finding yours

Attacker-side discovery is trivial and you should assume it has already happened. Search engines for internet-connected devices index these products by their HTTP response signatures, and a query for a default page title returns thousands of live hosts. Doing the same search against your own domains and IP ranges costs nothing and is the fastest way to find out where you stand.

Start from the outside, because that is the view that matters:

# Enumerate your public ranges for the ports these tools commonly use.
# Adjust the range; 7860 (Gradio), 8188 (ComfyUI), 3000 (many), 4000 (LiteLLM),
# 7474/8000 (assorted), 11434 (Ollama).
nmap -Pn -sS -p 3000,4000,5000,7860,8000,8188,11434 \
  --open -oG - 203.0.113.0/24

Then verify what responds. An HTTP 200 with no authentication challenge is the finding:

# A bare 200 on the API root means no auth in front of it.
curl -s -o /dev/null -w '%{http_code}\n' http://203.0.113.42:4000/models
curl -s http://203.0.113.42:11434/api/tags   # Ollama: lists loaded models

The Ollama check is worth running specifically. It is one of the most commonly exposed items in this category, it has no authentication of its own by design, and a response listing your loaded models tells an attacker they have found a free inference endpoint attached to your hardware.

Inside cloud accounts, the query is about public addressing rather than ports:

# AWS: every running instance that has a public IP, with its security groups.
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[?PublicIpAddress!=`null`].
           [InstanceId,PublicIpAddress,Tags[?Key==`Name`].Value|[0]]' \
  --output table

Cross-reference the result against your asset inventory. The rows that do not appear in the inventory are where the AI experiments live.

Discovery across three clouds

The AWS query above finds instances that hold a public address, and it misses several other routes to the internet. Load balancers, managed container services, and Kubernetes services of type LoadBalancer all publish without any instance ever holding a public IP. Run the equivalent queries in every cloud you have, including the ones you believe you do not use.

# Azure: every public IP resource and what it is attached to.
az graph query -q "
  Resources
  | where type =~ 'microsoft.network/publicipaddresses'
  | project name, resourceGroup, subscriptionId,
            ip = properties.ipAddress,
            attachedTo = properties.ipConfiguration.id
" --output table

# GCP: instances holding an external NAT address.
gcloud compute instances list \
  --filter="networkInterfaces[].accessConfigs[].natIP:*" \
  --format="table(name, zone, networkInterfaces[0].accessConfigs[0].natIP)"

# GCP: forwarding rules, which is how most managed exposure actually happens.
gcloud compute forwarding-rules list \
  --format="table(name, region, IPAddress, portRange, target)"

Kubernetes deserves a pass of its own, because one line in a manifest provisions a public load balancer in every managed cluster worth naming:

kubectl get svc --all-namespaces -o json \
  | jq -r '.items[]
      | select(.spec.type=="LoadBalancer")
      | [.metadata.namespace, .metadata.name,
         (.status.loadBalancer.ingress[0].ip // "pending"),
         (.spec.ports | map(.port|tostring) | join(","))]
      | @tsv'

Once you have a candidate list, fingerprint it rather than eyeballing it. The script below never authenticates and never sends a payload, which keeps it defensible to run across your own estate without a long conversation first:

#!/usr/bin/env bash
# usage: ./fingerprint.sh 203.0.113.42
host="$1"
probe() {
  code=$(curl -s -o /dev/null -m 5 -w '%{http_code}' "$1" || echo 000)
  printf '%-48s %s\n' "$1" "$code"
}
probe "http://$host:11434/api/tags"          # Ollama
probe "http://$host:4000/v1/models"          # LiteLLM
probe "http://$host:7860/api/v1/version"     # Langflow
probe "http://$host:3000/api/v1/chatflows"   # Flowise
probe "http://$host:8188/system_stats"       # ComfyUI
probe "http://$host:8000/v1/models"          # vLLM, LangServe
probe "http://$host:8888/api"                # Jupyter

Anything returning 200 is a finding. A 401 or 403 counts as a partial pass and still deserves a second look, because several products in this category answer 403 on one path and 200 on another. Keep the response body as evidence and attach it to the ticket, since a screenshot of your own model list ends the argument about whether the instance was genuinely open far faster than a port number does.

A scheduled external scan you can actually keep

A one-time sweep tells you about today. The reason this category grew 60% in nine months is that new instances arrive faster than sweeps remove them, so the scan has to run on a schedule and report only what changed.

Diffing is what makes it survivable as a habit. A weekly scan that prints 400 open ports gets ignored by the third week. A weekly scan that prints “two new listeners since last Tuesday” gets read every time.

#!/usr/bin/env bash
# /usr/local/bin/ai-surface-scan
set -euo pipefail

RANGES=/etc/asm/ranges.txt        # one CIDR per line, your public space
STATE=/var/lib/asm
PORTS=3000,4000,5000,5678,6333,7860,8000,8080,8188,8888,11434,18789
mkdir -p "$STATE"

today="$STATE/$(date +%F).txt"
nmap -Pn -sS -p "$PORTS" --open -oG - -iL "$RANGES" \
  | awk '/Ports:/ {print $2, $5}' \
  | sort -u > "$today"

if [[ -f "$STATE/baseline.txt" ]]; then
  new=$(comm -13 "$STATE/baseline.txt" "$today" || true)
  if [[ -n "$new" ]]; then
    jq -Rs '{text: ("New AI-adjacent listeners on public ranges:\n" + .)}' <<<"$new" \
      | curl -sS -X POST -H 'Content-Type: application/json' -d @- "$SLACK_WEBHOOK"
  fi
fi
cp "$today" "$STATE/baseline.txt"

Run it from outside your own network. A scan launched inside a VPC follows the internal path and will cheerfully report that everything is fine while the internet sees something different. A small VM at another provider, or a hosted CI runner, gives you the correct vantage point:

# .github/workflows/external-surface.yml
name: external-surface
on:
  schedule:
    - cron: '17 6 * * 1' # 06:17 UTC every Monday
  workflow_dispatch:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install scanner
        run: sudo apt-get update && sudo apt-get install -y nmap jq
      - name: Scan and diff
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
        run: sudo -E ./scripts/ai-surface-scan.sh
      - name: Persist baseline
        uses: actions/upload-artifact@v4
        with:
          name: surface-baseline
          path: /var/lib/asm/baseline.txt

Two operational notes. First, scan only address space you own or hold written authorisation for, and keep that authorisation somewhere you can produce it, because your provider’s abuse desk will eventually ask. Second, -sS needs raw sockets, so on a runner without privileges use -sT and accept the slower run. If your ranges are large enough that nmap takes hours, put naabu or masscan in front of it and reserve nmap for the hosts that answered.

Proving it is actually unreachable

Fixing the binding and closing the security group produces a comfortable feeling that is frequently wrong. Verification is a separate step from remediation, and it has to happen from the outside, because every mistake in this category looks correct from the inside.

Start on the host, where the bind address tells you what the process is willing to accept:

# Anything on 0.0.0.0 or [::] is listening on every interface.
ss -ltnp | awk 'NR==1 || $4 ~ /0\.0\.0\.0|\[::\]/'

# Docker's own view, which is the one that decides reachability for containers.
docker ps --format 'table {{.Names}}\t{{.Ports}}'

A published port shown as 0.0.0.0:7860->7860/tcp is open regardless of what your host firewall reports. Docker writes its forwarding rules into the nat table ahead of the filter rules that ufw manages, so a container port published without an address prefix stays reachable while ufw status shows a deny-by-default policy. Teams rediscover this every few months, usually during an incident.

Then test from a vantage point with no relationship to your network. A cloud shell at a different provider, a hosted CI runner, or a phone on mobile data all qualify. A VPN-connected laptop does not.

# From outside. 000 means the connection never completed, which is the goal.
for p in 3000 4000 7860 8000 8188 11434; do
  printf '%-6s %s\n' "$p" \
    "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "http://203.0.113.42:$p/" || echo 000)"
done

# IPv6 separately. Rules written for v4 routinely leave v6 wide open.
curl -6 -s -o /dev/null -m 5 -w '%{http_code}\n' 'http://[2001:db8::1]:11434/api/tags'

The IPv6 check catches a genuine and common failure. Cloud security groups treat 0.0.0.0/0 and ::/0 as separate rules, dual-stack instances pick up a routable v6 address automatically in several default configurations, and a team that tightened the v4 rule and stopped there remains fully exposed to anyone who resolves the AAAA record.

Once the manual check passes, turn it into a test that runs without you:

# CI job on an external runner. A reachable service fails the build.
fail=0
for target in 203.0.113.42:7860 203.0.113.43:11434; do
  code=$(curl -s -o /dev/null -m 5 -w '%{http_code}' "http://$target/" || echo 000)
  if [[ "$code" != "000" ]]; then
    echo "FAIL: $target answered $code from outside the network" >&2
    fail=1
  fi
done
exit "$fail"

That job earns its keep six months later, on the day somebody recreates the load balancer during an outage and forgets to reapply the restriction. Write the assertion while the remediation is fresh, because that is the only moment when anyone knows exactly which addresses and ports are supposed to be silent.

Fixing it without breaking the work

The instinct is to shut everything down, which generates a fight with the team whose project it is and usually ends with the service reappearing somewhere less visible. A better sequence keeps the capability and removes the exposure.

Bind to localhost and reach it over a tunnel. Most of these tools are used by a handful of people, and SSH port forwarding or a mesh VPN covers that need completely:

# docker-compose.yml
services:
  langflow:
    image: langflowai/langflow:latest
    ports:
      - '127.0.0.1:7860:7860' # the prefix is the entire control

Where a service genuinely needs to be reachable by other services, put it behind an authenticating proxy rather than adding an API key inside the tool. An identity-aware proxy in front of the container gives you single sign-on, logging, and revocation without depending on the product’s own authentication maturing.

For provider credentials, stop putting long-lived keys in the tool at all. Route through a gateway that holds the real keys and issues scoped, short-lived tokens to each internal consumer. A leaked token that expires in an hour and is capped at a spending limit turns a crisis into a ticket.

The conversation with the team that owns it

Remediation fails on politics more often than on technique, so it helps to arrive with something other than a shutdown notice. The team running the exposed instance is usually delivering something the business asked for, on a deadline, with no security engineer attached.

Two framings work better than a policy citation. The first is cost: an open proxy to a paid model API is a direct financial exposure with a number attached, and finance understands a five-figure unplanned bill faster than they understand data classification. The second is offering the replacement rather than the removal. Turning up with a working tunnel configuration, a proxy that gives them single sign-on, or a gateway that hands out scoped tokens converts the conversation from “stop doing your project” into “here is a better version of your setup”. Teams accept that trade readily, and the instance stays fixed because the fixed version is more pleasant to use than the broken one.

The bill is often the first alert

Worth calling out separately: for a large share of these incidents, the detection channel is finance rather than security.

An exposed model gateway with working provider keys is a free inference endpoint, and the people who find them use them at volume. Consumption climbs from a few dollars a day to hundreds, then thousands. Somebody in finance queries an unexpected line item three weeks later, and the security team learns about the exposure from an expense review.

That is a slow and expensive detection path, and it is trivially improvable. Every model provider offers spend alerting and hard spending caps. Setting a cap slightly above expected usage on every provider account converts a five-figure surprise into a service interruption that pages someone the same hour. For non-production accounts, which is where most exposed instances live, the cap can be low enough that abuse becomes immediately obvious.

Rate limiting at the gateway serves the same function with better granularity. A per-key request ceiling means a stolen key stops being useful quickly, and a sudden ceiling breach is a clean signal.

An authenticating proxy in front of everything

Adding authentication inside each tool means depending on the security maturity of a dozen young projects, each with its own login screen, session handling, and patch cadence. Put one proxy in front of all of them instead, and the products behind it never need to be directly reachable at all.

The pattern is a reverse proxy that terminates TLS, checks an identity provider, and forwards only authenticated requests to a container bound to localhost or a private network. Caddy plus oauth2-proxy covers it in about fifteen lines:

# Caddyfile
langflow.internal.example.com {
    forward_auth oauth2-proxy:4180 {
        uri /oauth2/auth
        copy_headers X-Auth-Request-User X-Auth-Request-Email

        @unauthorised status 401
        handle_response @unauthorised {
            redir * /oauth2/start?rd={scheme}://{host}{uri}
        }
    }

    handle_path /oauth2/* {
        reverse_proxy oauth2-proxy:4180
    }

    reverse_proxy langflow:7860 {
        flush_interval -1        # streaming responses must not be buffered
        transport http {
            read_timeout 0       # long generations outlive default timeouts
        }
    }
}

Two settings in that block are the ones people miss, and both produce the same outcome when skipped: somebody concludes the proxy broke the tool and removes it. flush_interval -1 disables response buffering, without which token-by-token streaming arrives as a single lump at the end. The zero read timeout stops a long generation being cut off mid-response. The nginx equivalents are proxy_buffering off and a proxy_read_timeout measured in minutes, and the same trap exists there.

The oauth2-proxy side is ordinary configuration with one decision worth making deliberately:

# oauth2-proxy.cfg
provider          = "oidc"
oidc_issuer_url   = "https://login.example.com/"
client_id         = "ai-tooling"
email_domains     = ["example.com"]
upstreams         = ["static://202"]
cookie_secure     = true
cookie_expire     = "8h"
# Restrict to a group rather than the whole company.
allowed_groups    = ["ai-platform-users"]

allowed_groups is the line that converts “anyone with a company login” into “the fourteen people on this project”. Skipping it hands every employee, every contractor, and every compromised account in the tenant a route to the flow editor and its code execution nodes.

Where a service is machine-to-machine rather than human-facing, mutual TLS fits better than an identity provider round trip. Where the tool is used by three people who all have SSH access anyway, a tunnel remains the cheapest correct answer and needs no proxy. Pick the lightest option that matches the real access pattern, then bind the container to localhost so no other option remains available.

Standing up a gateway that holds the keys

The earlier advice to stop putting long-lived provider keys inside tools needs a concrete shape, because “use a gateway” is easy to agree with and hard to act on. The shape is a single service that holds the real provider credentials, sits on a private network, and issues scoped keys to everything else.

Nothing in the private column holds a public address. The gateway is the only component with a provider key, which makes it the component you patch first and monitor hardest.

LiteLLM is the common choice for this role, so take it as the worked example. The configuration keeps real credentials in the environment and never in the file:

# config.yaml
model_list:
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: chat-alternate
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  enforce_user_param: true # every request names a user, for attribution

litellm_settings:
  max_budget: 500 # hard ceiling across the whole gateway, in USD
  budget_duration: 30d
  drop_params: true

Consumers then receive keys that expire and cost a bounded amount:

# Issue a key for one team, one model, one hour, five dollars.
curl -sS -X POST http://gateway.internal:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
        "models": ["chat-default"],
        "duration": "1h",
        "max_budget": 5,
        "rpm_limit": 60,
        "metadata": {"team": "search-relevance", "issued_by": "ci"}
      }'

A key with those properties changes the shape of a leak entirely. A stolen long-lived provider key is an open-ended liability you find out about on an invoice. A stolen gateway key is worth five dollars, dies within the hour, names the team it belonged to in every log line, and can be revoked with one call without disturbing anyone else’s access.

The gateway becomes the thing you defend, and it earns that attention. CVE-2026-42208 is the reminder: a pre-authentication SQL injection in LiteLLM’s own authentication path, scored 9.3, present in versions 1.81.16 through 1.83.6 and fixed in 1.83.7. The injection ran inside the auth check itself, so no rate limit, allow list, or key requirement stood in front of it, and anything reachable on port 4000 was exploitable. Exploitation in the wild was reported roughly 36 hours after the advisory, and researchers counted more than 2,000 exposed instances in April 2026. Centralising your keys behind a gateway is the right design; putting that gateway on a public address undoes the entire benefit in a single request.

Logging that makes abuse visible

Every control described so far reduces the chance of an exposure. None of them tells you that a perfectly valid key is being used by the wrong person, which is the failure mode left over once the ports are shut. The proxy and the gateway are the two places where that becomes observable, because the tools themselves log almost nothing you would want during an investigation.

Four fields carry most of the investigative value, and all four have to be captured at issue time rather than reconstructed afterwards:

• The gateway key identifier and the metadata attached when it was created, meaning team, issuer, and purpose
• The source address, plus the authenticated identity where a proxy sits in front
• Model name, prompt and completion token counts, and computed cost for each request
• Response status and latency, which is what separates abuse from a deployment that simply broke

With those recorded, the alerts write themselves. Spend rate per key is the highest-signal one, because abuse of a stolen key looks nothing like normal usage inside fifteen minutes:

# Alert when a key burns budget far faster than its own baseline.
groups:
  - name: llm-gateway
    rules:
      - alert: GatewayKeySpendSpike
        expr: |
          sum by (api_key_alias) (rate(litellm_spend_metric_total[15m]))
            > 5 * sum by (api_key_alias) (rate(litellm_spend_metric_total[24h] offset 24h))
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'Key {{ $labels.api_key_alias }} is spending 5x its usual rate'

A handful of other signals earn their alert budget, and each has a known false-positive source worth writing into the runbook so the first responder does not chase it twice.

Signal	What it catches	Where the noise comes from
Key used from an autonomous system it has never appeared in before	A key that left your network	Engineers travelling, and CI runners on rotating provider addresses
Requests to /v1/models or /api/tags from outside your ranges	Reconnaissance against something still reachable	Your own external scan, which should be allowlisted by source
A CI-issued key active outside its pipeline’s schedule	Credential reuse by a person or an attacker	Manual pipeline reruns during incidents
Sustained requests with identical short prompts	The capacity testing described earlier	Health checks, if somebody pointed one at a completion endpoint
Token counts that jump an order of magnitude per request	Data being pushed through your account	A genuine new feature nobody told you about

Keep this telemetry for ninety days rather than the fourteen that comes as default in most log pipelines. The exposure window in these incidents routinely turns out to be weeks long, the invoice that triggers the investigation arrives a month after the fact, and a retention period shorter than your billing cycle guarantees the answer has already been deleted by the time anyone asks the question.

Denying public exposure in code

Sweeps and proxies both depend on somebody doing the right thing after the fact. Policy in code moves the decision earlier, to the point where creating the exposure fails instead of succeeding, and it is the only control in this article that scales without adding headcount.

Start where the accident happens most often, in a subnet that assigns public addresses automatically:

# Terraform: private by default, with no silent opt-in.
resource "aws_subnet" "ai_workloads" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.40.16.0/20"
  map_public_ip_on_launch = false

  tags = {
    Name    = "ai-workloads-private"
    Purpose = "self-hosted inference and orchestration"
  }
}

Above that, an organisation-level guardrail closes the workaround where somebody attaches an address by hand:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "DenyPublicIpOnLaunch",
			"Effect": "Deny",
			"Action": "ec2:RunInstances",
			"Resource": "arn:aws:ec2:*:*:network-interface/*",
			"Condition": {
				"Bool": { "ec2:AssociatePublicIpAddress": "true" }
			}
		}
	]
}

Service control policies cannot inspect the CIDR inside a security group rule, so the wide-open ingress case has to be caught somewhere else. Catch it at plan time, where the feedback reaches the person who wrote the line:

# policy/security_groups.rego, run with conftest against a terraform plan
package main

deny[msg] {
  change := input.resource_changes[_]
  change.type == "aws_security_group_rule"
  change.change.after.type == "ingress"
  cidr := change.change.after.cidr_blocks[_]
  cidr == "0.0.0.0/0"
  msg := sprintf(
    "%s exposes ports %v to %v to the whole internet",
    [change.address, change.change.after.from_port, change.change.after.to_port]
  )
}

The other clouds have direct equivalents. GCP’s constraints/compute.vmExternalIpAccess organisation policy denies external addresses across a folder or an entire organisation, with an explicit allow list for the handful of machines that genuinely need one. Azure ships a built-in policy definition that denies network interfaces with public IPs, which you assign at the management group and relax by documented exception.

Kubernetes needs a rule of its own, because a LoadBalancer service walks straight past every VM-level control above:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: no-public-loadbalancers
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-loadbalancer-services
      match:
        any:
          - resources:
              kinds: ['Service']
      validate:
        message: 'Expose through the shared ingress; LoadBalancer services are blocked.'
        pattern:
          spec:
            type: '!LoadBalancer'

Ship all of these with an exemption mechanism from day one. A guardrail with no documented route to an exception gets switched off during the first outage it complicates, and switching it back on is nobody’s Monday morning priority.

Remediation options, ranked by effort

Nobody gets to do all of the above at once, so the ordering matters more than the list. Rank by what each option stops per hour of work invested, then start at the top and stop when the budget runs out.

Option	Effort	Stops	Leaves open
Change the port binding to 127.0.0.1	Minutes	All external access to that instance	Everything else on the host, and every other instance
SSH tunnel or mesh VPN for the people who need access	An hour	External access while keeping the tool usable	Machine-to-machine access patterns
Hard spend cap and alert on every provider account	An hour	Unbounded financial damage from any exposure, found or unfound	The data and execution consequences entirely
Authenticating reverse proxy in front of the tool	A day	Anonymous access, and it gives you an audit trail	Abuse through a compromised employee account
Gateway holding provider keys and issuing scoped tokens	Days	Provider credential theft becoming an open-ended bill	Exposure of the gateway itself, now your highest-value target
Weekly external scan with diffing and alerting	Days, then ongoing	Nothing directly, but it shortens time-to-discovery for everything	Anything on ranges you did not know you owned
Policy as code denying public IPs and LoadBalancer services	Weeks	The next accident, across every team, permanently	Assets in accounts outside your organisation boundary
Asset inventory that actually covers AI projects	Months	The structural cause	Nothing, assuming it is complete, which it will not be

Two rows are worth arguing about. The spend cap is the highest-value hour in the table, because it bounds the worst financial outcome of every exposure you have yet to find, and one person can apply it across all your provider accounts in an afternoon. The external scan buys the least on the day you build it and the most across the following year, which makes it the hardest item to get funded and the one to keep pushing for anyway.

Anti-patterns that keep showing up

Remediation in this area goes wrong in a small number of repeatable ways. Each of the following has appeared often enough to be predictable, and most of them feel like progress at the time somebody implements them.

Setting an API key inside the tool and calling it finished. Many of these products accept a static key as a query parameter or a header, and the key then lands in access logs, browser history, and the Slack message where somebody shared the URL. A static key in front of a code execution endpoint is a speed bump with good intentions.

Moving the service to an unusual port. Internet-wide scanners enumerate all 65,535 of them, and the reconnaissance data shows probes against 18789 alongside the obvious candidates. Changing the port removes the service from your own runbooks and leaves it exactly as findable as before.

Trusting ufw in front of Docker. Described above, and worth repeating because it produces a firewall that reports success while forwarding traffic anyway. Bind the published port to 127.0.0.1 and the whole question disappears.

Allowlisting the office address and forgetting about it. Office ranges change, remote staff arrive from residential addresses, and the exception grows to cover a commercial VPN exit range shared with thousands of unrelated customers. Allowlists in this category decay into open access over roughly eighteen months.

Using one provider key everywhere. A single key shared between production, staging, and three notebooks makes rotation an outage, so nobody rotates, so the key in the exposed prototype is also the key in production.

Treating the model list endpoint as harmless. /v1/models and /api/tags return no secrets and appear as the first request in every recorded reconnaissance sequence. Leaving them reachable while claiming the instance is protected means the protection sits on the wrong endpoint.

Deleting the instance without rotating the credentials. Terminating a compromised VM removes the evidence and none of the risk, because whatever keys it held left the building days earlier.

Scanning only the ranges you already know about. The instances that matter live in the account nobody inventories, at a provider procurement never approved, paid for on somebody’s personal card. Start from billing and DNS rather than from the range list, which describes yesterday’s estate.

Believing the authentication flag without testing it. Several products in this category have shipped configuration options honoured on one code path and ignored on another. Send an unauthenticated request and read the response code yourself. That takes ten seconds and produces the only evidence that counts.

Underneath all of these sits one assumption worth naming: that an obscure service is an undiscovered service. Discovery in 2026 is continuous, automated, and cheap, and the gap between deploying something and having it probed is measured in hours.

When you find one that has already been abused

Some of these findings are worse than near-misses. The instance answered, the model list came back, and the provider dashboard shows usage from three continents overnight. Response ordering for that case differs from a clean exposure, because money is leaving the building while you work.

Containment comes before investigation here, and it starts at the provider rather than the host. Setting a hard spend limit takes about a minute in every major provider console and stops the bleeding even if the attacker holds keys you have not located yet. Do that first, preserve evidence second, cut the network third. Stopping the instance destroys memory-resident evidence and signals to the attacker that you noticed, while a deny-all security group achieves containment and leaves the machine available for forensics.

Rotation order matters, because rotating in the wrong sequence hands the attacker a window to re-establish access using credentials you have not reached yet.

Order	Credential	Why it sits here	Typical mistake
1	Model provider API keys	Actively costing money and visible in the tool’s own configuration	Rotating one provider and missing the second one wired up months earlier
2	Database and object storage credentials embedded in flows	Broader access than the AI tool itself, and rarely scoped down	Assuming the attacker never opened the flow editor
3	Cloud instance role sessions	Temporary credentials pulled from the metadata service outlive the key rotation	Detaching the role while issued session tokens remain valid
4	Keys the tool issued downstream	Every consumer the gateway or tool handed a key to	Treating internal keys as low risk because they are internal
5	SSO and service accounts the tool touched	Slowest to abuse, longest-lived, easiest to forget	Skipping it because the connection looked read-only

Step three is the one teams get wrong most often. Detaching an instance profile stops new credentials being issued and leaves already-issued session tokens working until they expire on their own. On AWS, invalidate them explicitly:

# Deny everything issued before now for this role. Existing sessions die immediately.
aws iam put-role-policy \
  --role-name langflow-prototype-role \
  --policy-name RevokeOlderSessions \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Deny",
      "Action": ["*"],
      "Resource": ["*"],
      "Condition": {
        "DateLessThan": {"aws:TokenIssueTime": "2026-07-26T00:00:00Z"}
      }
    }]
  }'

On the cost side, pull the usage export from each provider and find the first request that was not yours. That timestamp is your exposure window, and it usually predates the invoice by weeks. Contact provider support early with a written timeline. Several providers will credit clearly fraudulent usage when the abuse is documented and reported promptly, and the ones that will not are considerably less receptive once the invoice has already been settled.

Then answer the data question honestly, because the temptation to close the ticket when the spend stops is strong. Work out what the flows could reach, which retrieval corpora were connected, and whether any of it was personal or regulated. An exposed workflow builder with a live connection to an internal document store is a data incident with notification obligations attached, whatever the invoice eventually says.

Making the next one visible

One-off cleanups decay. The reason this category grew 60% in nine months is that new instances appear faster than sweeps remove them, so the durable fix has to be continuous.

Three controls carry most of the weight, in rough order of effort. Continuous external scanning of your own ranges, running weekly and alerting on new open ports, catches the exposure regardless of who created it or which account it lives in. Cloud policy that denies public IP assignment outside an approved account or subnet stops a category of accident before it happens. An egress and spend alarm on model provider accounts catches the case you missed, because an exposed proxy shows up as a cost anomaly long before anyone reports it as a breach.

None of this is AI-specific, which is the point worth ending on. The security problem is a young product category with weak defaults, deployed by people working outside the normal review path, inside organisations whose discipline of knowing what they have on the internet was never extended to cover it. The tools are new. The exposure is the oldest finding in the field.