The LiteLLM Package Was Poisoned for 40 Minutes. The Keys Lasted for Months

For roughly 40 minutes on 24 March 2026, two poisoned versions of a popular Python package sat in the place developers normally trust to install it. A build job that fetched LiteLLM 1.82.7 or 1.82.8 could run a credential stealer before the application had even started. The malicious code looked through the build machine for cloud keys, source-control tokens, SSH keys, Kubernetes credentials, database passwords, and other secrets, then sent what it found away. (LiteLLM incident report, Ars Technica)

The infected packages were removed that day. The incident did not end that day.

On 12 and 13 August 2026, new analyses described about 434,000 captured files tied to close to 2,500 organizations. Hudson Rock said the archive it examined contained 153GB and 433,909 files, while Ars Technica reported a much larger figure for material obtained by Hudson Rock. That size discrepancy is unresolved in the public reporting, so it should not be treated as a settled fact. The reports do agree on the part that matters operationally: the stolen material includes real credentials from a very large number of build environments, and an independent researcher said that keys from one major technology company still worked months after the original compromise. (Help Net Security, CloudSEK, Ars Technica)

A brief package compromise became a months-long identity problem because build systems held credentials that outlived the jobs using them. If your response to a poisoned dependency ends with deleting the package or rebuilding a clean image, you may have removed the burglar while leaving every copied key in service.

The new evidence changed the response clock

LiteLLM is an open-source gateway that gives applications one interface for calling different AI models. Teams use it in local programs, backend services, model gateways, agent frameworks, and build pipelines. That placement makes it useful. It also puts the package near exactly the material a credential stealer wants: model-provider keys, cloud identities, deployment tokens, repository access, and the environment variables that connect a software build to the rest of a company.

LiteLLM’s own March incident report established the narrow facts early. Versions 1.82.7 and 1.82.8 published to the Python Package Index, or PyPI, were malicious. PyPI is the public package service from which Python tools commonly install software. The packages were available from 10:39 UTC for about 40 minutes before quarantine, and LiteLLM said its official proxy Docker image was not affected because that route pinned dependencies and did not rely on the poisoned packages. (LiteLLM incident report)

The malicious versions did more than wait for an application to import a suspicious module. Version 1.82.8 included a .pth file, a Python startup hook that can execute when the interpreter starts. A developer did not need to call LiteLLM for that hook to run. Starting Python in an affected environment was enough. Community analysis recorded the same behavior on 24 March and warned teams to rotate every credential that had been present on an exposed machine. (GitHub issue 24518)

A 40-minute publishing window sounds small until it meets automation. Continuous integration and continuous delivery, usually shortened to CI/CD, rebuild software without waiting for a person to type each command. A floating dependency can be resolved hundreds of times across branches, pull requests, scheduled jobs, test matrices, architecture variants, and ephemeral runners. One bad release can therefore touch far more machines than its clock time suggests.

The August analysis gives that multiplier a shape. CloudSEK said its reconstructed dataset included roughly 434,000 captured files and mapped potential exposure to more than 2,500 organizations. Hudson Rock attributed 118,829 CI runner dumps in its archive to 2,488 corporate domains. Both firms cautioned that exposure is not the same as proof that every named company suffered a later intrusion. A file can contain an expired key, a test value, or infrastructure operated by a subsidiary or supplier rather than the company suggested by an email address. (CloudSEK, Help Net Security)

That caution should narrow the claim, not soften the response. Screenshots and reporting describe AWS secret keys, Salesforce client secrets, Slack signing secrets, Azure environment variables, AI-provider keys, repository tokens, and Kubernetes credentials in the captured material. Independent security researcher Kevin Beaumont said he checked keys under one affected organization’s responsible-disclosure policy after the company told him they had been rotated. According to his account, almost every tested key still worked, and he filed a report. (Help Net Security, Ars Technica)

As of 14 August 2026, there is no public evidence that every exposed credential remains active or that every listed organization was breached through it. There is enough evidence to reject passive reassurance. The incident date is nearly five months old, the stolen data has surfaced in research hands, and at least one spot check reportedly found live access. Teams that may have run those versions should move from “were we vulnerable?” to “which identities could that job see, and have we proved each one dead?”

How one poisoned scanner reached an AI package

The LiteLLM compromise did not begin with a developer choosing an obscure download mirror. It began farther upstream, in a security tool already trusted by the package’s release process. That chain is why the event deserves attention beyond teams that use LiteLLM.

LiteLLM said in March that evidence pointed to the earlier compromise of Trivy, an open-source vulnerability scanner. A release workflow used Trivy to inspect builds. When the workflow fetched a poisoned scanner, that scanner ran inside the release environment and could read the secrets available there. Those secrets reportedly included credentials that allowed unauthorized packages to be published under LiteLLM’s official name on PyPI. (LiteLLM incident report, The Hacker News)

The sequence is painfully ordinary:

1. A trusted build job installed a security tool. The scanner arrived as executable software, not as a passive database of vulnerability names.

2. The scanner inherited the job’s access. The build environment held publishing credentials because the same workflow could inspect code and release packages.

3. The stolen publishing identity created official-looking releases. Users fetched versions 1.82.7 and 1.82.8 from the normal package service under the expected project name.

4. Downstream automation ran the malicious package beside more secrets. The poisoned release reached other build jobs, developer machines, and unpinned installations.

5. Removing the package stopped new collection but did not revoke what had already left. Every captured credential kept its own lifetime and permissions.

Nothing in that chain requires an AI model to turn hostile. LiteLLM happens to sit in the AI stack, where provider keys and broad integration credentials are common, but the mechanism is software supply-chain compromise. A program trusted by one build gained the authority to alter another program, which then inherited authority from thousands of downstream environments.

The label “security scanner” can hide the first mistake. Teams often treat scanners as controls around a build rather than code inside it. In practice, a scanner downloaded during a workflow can execute with the workflow’s filesystem, network, and environment access. If the job can publish packages, sign artifacts, open a cloud account, or read private repositories, the scanner may be standing in the same room as those powers.

The release job combined observation and authority: it inspected the source tree while holding a credential capable of publishing to PyPI. Once those roles shared an environment, compromising executable code in the inspection path could become a route to the publishing identity.

LiteLLM’s response points toward a better shape. The project paused releases, rotated maintainer accounts, audited the chain, published checksums for reviewed releases, and introduced a new CI/CD pipeline with isolated environments, stronger gates, and a separate release path before publishing version 1.83.0. Those changes do not erase the stolen downstream secrets, but they address the upstream condition that turned a scanner compromise into a package release. (LiteLLM incident report)

This is the same design rule The Secure Harness applies to coding agents: a component that reads untrusted material should not quietly share the identity that releases production artifacts. The component may be an agent, a scanner, a formatter, or a build plugin. The name changes. The boundary does not.

Build jobs are identities, not empty rooms

A CI runner looks disposable. It starts, checks out code, runs commands, and disappears. That short life can create a false sense that anything exposed inside it is temporary too.

The machine is temporary. Its authority may not be.

A typical build job receives secrets through environment variables, mounted files, workload identity, metadata services, package-manager configuration, or generated credentials. The values may let the job download private dependencies, push container images, sign releases, update deployment systems, query cloud resources, or publish packages. The runner becomes an identity assembled for the duration of the job.

If that identity uses a six-month cloud key, the job’s disappearance does not end the key. Deleting the runner also does not narrow a personal access token that reaches every repository in an organization. A package publishing token can upload a future release, so its blast radius extends beyond the current build into the trust users place in the next version number.

The LiteLLM data makes this distinction concrete. The credential stealer gathered material while builds were alive, then preserved it outside the runner. Months later, the original job logs may have expired and the compromised machine may no longer exist. The copied credential can still authenticate if nobody revoked it.

The practical model is to treat every build job as a principal, meaning an identity that can cause actions. Write down what it can read, what it can change, which network destinations it can reach, and when its authority ends. That description should be specific enough to answer a bad-day question: if arbitrary code runs in this job for one minute, what remains possible after the job exits?

For many teams, the honest answer is “far too much.” A pull-request test can read deployment credentials. A lint job can reach the public internet and the cloud control plane. A release token appears in every matrix leg. A reusable workflow inherits all repository secrets even though one step needs only read access. Convenience has collapsed several identities into one environment.

The fix is separation. Testing gets no release credential. Scanning gets read-only access to the material it inspects. Publishing happens in a fresh job that consumes a verified artifact rather than rebuilding it. The publishing identity exists only long enough to release that exact artifact, and the package service accepts it only from the expected workflow and repository.

A short-lived job should have short-lived authority. That sentence is easy to approve in a policy meeting and harder to enforce in a working pipeline. The LiteLLM incident shows why the engineering effort is worth paying. Once a permanent key crosses into a poisoned process, cleanup becomes a search across months of services and logs rather than a simple rebuild.

Why checking the current version is not enough

Most dependency incident playbooks start in the right place. Identify the affected versions, search lockfiles and images, stop new builds, remove the malicious package, and rebuild from a known-good source. Those steps contain execution. They do not contain identity theft.

Suppose a team proves that every current image now uses LiteLLM 1.83.0 or later. That tells the team what will execute on the next start. It says nothing about the AWS key present on 24 March, the GitHub token mounted into a release job that morning, or the database password copied from an integration-test environment.

A clean software bill of materials is therefore evidence about code, not evidence about credentials. A green malware scan shows that the current filesystem lacks the known payload. A passing build proves the replacement works. None of those tests asks the services on the other end whether an old key still opens the door.

Rotation can fail quietly too. Teams often create a new credential, update the pipeline, and call that rotation. The old value remains enabled in case rollback is needed. The application has moved on, so no operational alarm fires. An attacker holding the old copy keeps a second working identity that normal tests no longer exercise.

Beaumont’s reported spot check illustrates this exact trap. The affected organization said the keys had been rotated. Almost all the old values he was permitted to test reportedly still worked. Issuing replacements and revoking predecessors are separate operations, and only the second one ends the stolen key’s authority. (Ars Technica)

Logs should settle the question, yet retention often runs out before the breach is understood. The malicious packages appeared on 24 March. The broader dataset analysis arrived in August. A team keeping thirty days of cloud audit logs cannot now search the full period. Absence of visible abuse in the remaining window is useful but incomplete evidence.

That is why response must separate three questions. Did the affected code run? Which credentials and sensitive files could it read? Does each exposed identity still work, and what did it do after exposure? Answering only the first question leaves the most durable part of the incident untouched.

For high-value credentials, do not wait for perfect proof that the package executed. If a build resolved dependencies without a lock, ran during the affected window, and exposed a release or cloud identity, the cost of revocation is usually lower than the cost of certainty. Replace the key, disable the old one at the issuer, and preserve enough audit evidence to investigate its use.

The 40-minute window was multiplied by floating installs

Package managers make software easy to reuse by resolving names and version ranges. Automation makes that resolution frequent. The combination turns a brief malicious release into a wide event when builds ask for “the latest compatible version” instead of a reviewed artifact.

An unpinned command such as pip install litellm answers a changing question. On 23 March it could install one set of bytes. During the poisoned window on 24 March it could install another. A rebuild performed later may fetch clean code and erase the simplest evidence that the bad version ever ran.

Lockfiles reduce this exposure by recording exact versions and often hashes. Hash checking goes further by refusing a file whose bytes do not match the reviewed record. An internal package mirror can create a quarantine point where new upstream releases are scanned and approved before ordinary builds can fetch them. None of these controls makes an approved package harmless, but each stops an unexpected upstream version from silently entering every job.

The better compromise is controlled freshness. A scheduled dependency-update workflow proposes exact changes. Tests and security checks run against those changes in an environment without release credentials. A human or policy gate approves the lockfile. Production and release jobs install only the recorded artifacts. Updates remain frequent, but they become visible changes rather than ambient internet state.

The same principle applies to scanners. Pin the scanner itself, its actions, and any installation scripts. Verify the publisher and digest where the ecosystem supports it. Do not grant a newly downloaded security tool more access merely because its purpose is defensive. A scanner executing inside a privileged release job is privileged code.

This incident also shows why “we did not import LiteLLM” can be a weak answer. Version 1.82.8’s .pth startup hook could run on Python startup, and LiteLLM could arrive through another tool as a transitive dependency. Search installed package records, cached wheels, build manifests, and job logs. Do not limit the hunt to application source containing import litellm.

As of 14 August 2026, LiteLLM says its official proxy Docker image was not affected by the March PyPI packages, and its report names specific safe routes and versions. Preserve that distinction. A useful response targets the installations that could have consumed 1.82.7 or 1.82.8 rather than treating every LiteLLM user as compromised. Precision saves time for the environments that actually need key revocation. (LiteLLM incident report)

What to do if your environment may be in the data

Start with the date and the build route. The malicious PyPI releases were present on 24 March 2026. LiteLLM’s report says the official proxy Docker image was not affected, while direct or transitive Python installs could have been. The immediate goal is to find jobs and machines that could have received 1.82.7 or 1.82.8, then follow every identity those environments could access.

Do not begin by hunting for your company name in a public list and stopping if it is absent. Hudson Rock said many captured files lacked obvious ownership markers, and attribution can point to a subsidiary or service provider rather than the brand visible in an email address. Your own package and credential records are better evidence than an external name match. (Help Net Security)

1. Freeze unsafe resolution paths. Pause builds that install LiteLLM from an unpinned range or fetch executable security tools without a recorded digest. Move affected workflows to a reviewed version and preserve current logs, manifests, caches, and runner metadata before routine cleanup removes them.

2. Find actual execution, not just source references. Search CI job logs, Python package inventories, image layers, lockfile histories, cached wheels, dependency reports, and build attestations for versions 1.82.7 and 1.82.8 on 24 March. Include jobs where LiteLLM arrived transitively and jobs that started Python without importing LiteLLM directly.

3. Inventory everything the environment could read. Record environment variables, mounted secret files, cloud metadata access, repository tokens, package-manager credentials, SSH material, Kubernetes service accounts, database passwords, signing keys, model-provider keys, and internal endpoints. Include secrets fetched during the job, not only values visible in the workflow file.

4. Revoke at the issuing service. Create replacement credentials where continuity requires them, update the legitimate workload, then disable or delete the old values at AWS, GitHub, GitLab, PyPI, Kubernetes, the database, or the relevant provider. Verify that authentication with the predecessor fails. Updating a vault entry alone does not prove revocation.

5. Review use from 24 March onward. Search available cloud, source-control, package, deployment, identity, and network logs for the old credential identifiers and unusual actions. Pay special attention to new access keys, changed workflows, package publishes, force-pushed tags, added deploy keys, altered role policies, and outbound connections from build environments.

6. Treat publishing and signing identities as incident-critical. If the job could release packages, containers, binaries, or infrastructure, inspect every artifact produced after the exposure. Compare hashes and provenance with reviewed source. Rotate signing material according to the ecosystem’s recovery process rather than quietly replacing it.

7. Check persistence beyond the original secret. A stolen cloud key may have created another user. A repository token may have added a deploy key or changed a workflow. A Kubernetes token may have created a long-lived service account. Revocation closes the copied key but does not remove access the key already established.

8. Record proof of closure. For each exposed identity, keep the issuer, old identifier, permissions, exposure window, revocation timestamp, validation result, log coverage, and any follow-up investigation. “Rotated” is too vague for an incident with evidence that old keys survived nominal rotation.

This sequence is deliberately credential-first after containment. Teams are often tempted to spend days proving whether a named machine downloaded the exact wheel before touching a sensitive production key. Where evidence is incomplete and the key has broad authority, revocation is the safer and often faster experiment.

Be careful with evidence. Do not paste captured secrets into tickets, chat channels, or public issue trackers while coordinating the response. Use identifiers and secure incident systems. Keep the minimum copy needed for investigation under restricted access, then follow your legal and retention duties for disposal.

If your logs no longer reach back to March, say that plainly in the incident record. You can still revoke credentials, inspect current persistence, review release history, and improve future evidence. A six-month retention gap does not become proof of clean behavior because the oldest month has disappeared.

Redesign the release boundary before the next package moves

Emergency revocation closes this incident. Pipeline design decides whether the next poisoned dependency can repeat it.

The strongest improvement is to split untrusted execution from release authority. A build job compiles, tests, and scans an artifact without production publishing credentials. It produces a digest and provenance record. A separate release job receives that exact artifact after approval, obtains a narrow short-lived identity, publishes once, and expires.

That shape blocks a common escalation. If a scanner or test dependency is poisoned, it can damage the unprivileged build environment, but it cannot use a package token that is not there. If the release job does not install new dependencies or rerun the build, it has far less executable surface beside the publishing credential.

Modern workload identity can remove stored cloud keys from CI. Instead of keeping a permanent secret in repository settings, the job presents a signed identity from the CI platform and receives a short-lived cloud role for a named repository, branch, workflow, and environment. Conditions at the cloud provider can reject tokens from pull requests, forks, or unexpected workflows.

Package publishing supports a similar direction in some ecosystems through trusted publishing. The package service accepts an identity from the approved workflow rather than a reusable token copied into every build. Where that option exists, it removes a credential that malware can steal and replay months later. Where it does not, use the narrowest token, inject it only into the publishing step, and revoke it on a short schedule.

Network boundaries matter as well. A build job often has unrestricted outbound access because dependencies come from the internet. That gives malicious code a ready route to send secrets away. An internal mirror and an outbound allowlist can let the job fetch approved packages while blocking arbitrary destinations. Egress filtering, meaning control over what the computer can send out, is not a complete defense, but it can turn silent theft into a failed request and a useful alert.

Dependency controls belong at every executable layer. Pin application packages, actions, scanners, setup scripts, container base images, and release tools. Record hashes or immutable digests. Route updates through reviewed changes. A signed package proves who signed a file under a key; it does not prove the publisher’s pipeline was uncompromised. Provenance and identity restrictions help establish how the artifact was produced.

Detection should assume that some control will fail. Alert on package publishing outside the release workflow, changes to trusted publisher settings, new long-lived cloud keys, use of a CI role from an unexpected repository, and outbound traffic from builds to new domains. Keep audit logs long enough to outlast delayed disclosures. The LiteLLM window was in March; the broad stolen-data analysis arrived in August.

Finally, rehearse revocation. Pick a non-production credential, simulate exposure, and measure the time needed to identify its owner, replace it, disable the old value, prove rejection, search its use, and check for persistence. Teams discover during these exercises that a “build token” belongs to no one, powers five pipelines, and cannot be revoked without breaking release day. Finding that on a quiet Tuesday is cheaper.

The goal is not a pipeline in which no third-party code ever runs. Modern software cannot be built that way. The goal is a pipeline where one bad program encounters a small, temporary identity inside a boundary, rather than a ring of permanent keys to the company.

Forty minutes should have meant forty minutes

The poisoned LiteLLM packages were public for about 40 minutes on 24 March 2026. That was long enough for automation to run them across a remarkable number of environments. It should not have been long enough for copied access to remain useful in August.

The gap came from identity lifetime. Disposable runners carried reusable credentials. A clean rebuild removed malicious code, while old keys continued to exist at cloud providers, repositories, package services, and internal systems. “We rotated it” sometimes meant “we issued a replacement,” not “the stolen value can no longer authenticate.”

The August reporting contains uncertainties. The public accounts disagree sharply on the total archive size. External mapping cannot prove that every named organization was compromised, and exposure does not establish later abuse. Those limits belong in the story because inflated certainty produces bad incident work.

The settled facts are enough. Two official-looking package versions were malicious. The payload sought credentials. The packages could run inside automated Python environments. Separate research groups mapped a large body of captured build data, and an independent spot check reportedly found old keys still accepted months later. The right action does not depend on choosing the most dramatic storage figure.

Make the response smaller and harder. Prove what ran. Revoke what it could read. Check what those identities did. Then rebuild the pipeline so tests and scanners cannot borrow the identity that publishes software.

A 40-minute package window should create, at worst, a 40-minute identity window. Short jobs deserve short credentials, narrow permissions, and a release boundary that does not ask every downloaded tool to behave perfectly. That is how a supply-chain incident becomes containable engineering work instead of a half-year search for keys nobody remembers creating.

If you want practical security and AI engineering analysis without a daily flood, the newsletter sends one email per month. The signup is on this site.

Sources

• LiteLLM: Security Update, Suspected Supply Chain Incident, accessed 2026-08-14
• GitHub: LiteLLM package versions 1.82.7 and 1.82.8 compromised, issue 24518, accessed 2026-08-14
• CloudSEK: 2,500+ Companies and 434,000 CI/CD Pipelines Exposed, accessed 2026-08-14
• Ars Technica: Terabytes of credentials leaked in massive supply-chain attack, accessed 2026-08-14
• Help Net Security: 153GB of stolen credentials surface after LiteLLM supply chain attack, accessed 2026-08-14
• The Hacker News: Malicious LiteLLM Releases Tied to Trivy Hack May Have Exposed 2,500 Organizations, accessed 2026-08-14