ChatGPT Health and the Limits of a Training Opt-Out

One good promise, and what it leaves open

OpenAI launched Health in ChatGPT for US adults in July 2026, letting people connect Apple Health and medical records so the assistant can discuss their actual numbers. The accompanying commitment was that connected health information and the conversations around it would not be used for model training or ad targeting.

That commitment is meaningful, and the two things it rules out are the two the public worries about most. It also addresses a smaller share of the risk than most readers will assume, because training and advertising are only two of the ways data moves through a system of this kind.

The gap matters beyond one product. Any team building AI features on top of sensitive data will eventually write a similar sentence in their own privacy policy, and it is worth being clear about what such a sentence does and does not cover before you rely on it.

The questions that decide the outcome

Where does the data physically sit, and under whose law. Health data processed in one jurisdiction and stored in another is subject to both, and cross-border transfer rules for special-category data are stricter than for ordinary personal data. A product available in the US only may be storing in one region today and another after the next infrastructure change.

How long is it retained. Training exclusion says nothing about storage duration. A conversation retained for thirty days for abuse monitoring is a different exposure from one retained for seven years, and the retention period is what determines how much data exists to be lost in a breach.

Who can read it. Most AI products route some fraction of traffic to human reviewers for safety and quality work, and this is the practice users are most surprised by. A training exclusion is compatible with human review, because review is not training. Whether health conversations are exempted from review is a separate question that needs a separate answer.

Which subprocessors touch it. A model provider running on someone else’s cloud, using a third party for logging, another for analytics, and another for content moderation has four organisations in the data path. The commitment you were given binds the company you contracted with, and flows to the others only through their own agreements.

What happens on deletion. Deleting a conversation from a user interface removes it from that view. Whether it is removed from backups, from logs, from the vector index built over it, and from the analytics warehouse are four more questions, and the honest answer for most systems is that deletion propagates unevenly and slowly.

Is it in the context window of other requests. A system that retrieves your prior conversations to personalise a new one is putting health data into a context that may be logged, cached, or sent to a different model than the one you authorised.

What is the breach notification path. For a covered entity under HIPAA there are statutory timelines and obligations. A consumer app is frequently not a covered entity, which means your health data can be held by an organisation with no HIPAA duty at all, and the applicable regime is general consumer protection law.

Is derived data covered by the same promise. Embeddings, summaries, extracted structured facts, and risk scores computed from your health data are all data about you. A commitment phrased around “your health information” may or may not extend to a vector representation of it, and the answer is usually not stated.

Does the promise survive corporate change. Privacy commitments are contractual, and contracts are assets in an acquisition or a bankruptcy. The history of consumer genetics companies is the cautionary reading here.

The nine questions, and what a good answer looks like

Asking those questions is the easy part. Recognising a bad answer takes more work, because the bad answers are fluent and they are usually given in good faith by someone who has never traced the data path personally. The pattern to watch for is an answer phrased as a value rather than as a fact. “We take your privacy seriously” is a value. “Conversations are retained for 30 days, logs for 90, evaluation extracts indefinitely” is a fact, and a fact can be audited.

Question	The answer that means nothing	The answer worth having
Where does the data sit	”In the cloud, securely”	Named regions per data category, plus the contractual mechanism that stops processing from moving
How long is it kept	”Only as long as necessary”	A number of days per store: conversations, logs, traces, evaluation sets, backups
Who can read it	”Only authorised personnel”	The named roles, the approval flow, the sampling rate for human review, and the opt-out
Which subprocessors touch it	”We use industry-standard providers”	A published list, a notification commitment on change, and a right to object
What happens on deletion	”You can delete your data at any time”	A per-system propagation table with a stated maximum lag, covering backups and indexes
Is it reused as context	”We personalise your experience”	Which prior data is retrieved, into which model, and whether retrieval is logged
Breach notification path	”We will notify you as required by law”	The regime the vendor believes applies, the clock in days, and who notifies whom
Is derived data in scope	Silence	An explicit statement that embeddings, summaries, and scores inherit the source classification
Does the promise survive a sale	”Our terms may change”	A commitment that data transfers only with the same restrictions, or a deletion right on change of control

Read the right-hand column again and notice the shared property: every entry is a claim someone can be wrong about. That property is the whole point. A promise that cannot be falsified cannot be relied on either, and the distance between the two columns is mostly the distance between a marketing review and an engineering review of the same sentence.

The ninth question, about corporate change, gets skipped because it feels like a lawyer’s problem. It has produced some of the most concrete harm on the list. Consumer genetics is the worked example everyone cites, and the mechanism generalises: a company holding a large sensitive dataset under a strong privacy promise runs out of money, and the dataset becomes the most valuable thing in the estate.

All nine questions assume that promises are what protect you. The next thing to check is whether any law sits underneath them.

The HIPAA gap most people get wrong

A specific misconception is worth correcting because it drives bad decisions. HIPAA applies to covered entities and their business associates: providers, plans, clearinghouses, and the vendors handling data on their behalf. It does not attach to health information simply by virtue of the information being about health.

When you type your cholesterol result into a consumer chatbot, or connect an app that syncs it from your phone, that data has generally left the HIPAA perimeter. The protections that apply are whatever the company’s own terms promise plus state law, and state law in this area varies enormously. Some states have enacted specific consumer health data statutes with real teeth. Others have nothing beyond general unfair-practices law.

For a user, the practical implication is that the strength of protection depends on the company’s promises rather than a statutory floor. For a team building such a product, the implication is that “we are not a covered entity” is a description of your legal exposure and not a description of your users’ expectations, and the gap between those two is where reputational damage lives.

The state laws that fill the HIPAA gap

Saying that consumer health data falls outside HIPAA describes what is absent. What is present is a patchwork of state statutes written since 2023, drafted precisely because legislators noticed the same gap, and two of them reach almost any product touching health information about a resident of those states.

Washington’s My Health My Data Act is the strict one. It defines consumer health data broadly enough to catch inferences, requires opt-in consent for collection that is separate from any other consent, demands a distinct written authorisation before any sale, bans geofencing around facilities that provide health services, and carries a private right of action through the state’s Consumer Protection Act. There is no revenue or volume threshold, so a three-person startup with one Washington user sits inside the scope.

Nevada’s SB 370 copies most of the substance and changes the enforcement model. Violations are treated as deceptive trade practices, investigated by the Attorney General and the Commissioner of Consumer Affairs, with civil penalties under the state consumer protection statutes. Class action exposure disappears, which changes the risk calculation for a legal team and changes nothing at all about what an engineering team has to build.

	Washington MHMDA	Nevada SB 370	Connecticut, as amended	Comprehensive state privacy laws
Applicability threshold	None	None	Tied to the state privacy law’s thresholds	Revenue or data volume thresholds
What counts as health data	Broad, reaching inferences and some location signals	Broad, with location counted where it is used to infer care seeking	Consumer health data added to the sensitive category	Health data treated as a sensitive category
Consent to collect	Opt-in, separate from other consents	Opt-in, separate from other consents	Opt-in for sensitive data	Usually opt-in for sensitive data
Sale of the data	Separate written authorisation	Separate written authorisation	Consent plus restrictions	Opt-out or opt-in, varying by state
Geofencing near care facilities	Prohibited	Prohibited	Prohibited within a stated radius	Generally unaddressed
Enforcement	Attorney General plus a private right of action	Attorney General, as a deceptive trade practice	Attorney General	Attorney General, cure periods vary

The practical reading for an engineering team is that these are consent and disclosure regimes rather than security regimes. They tell you what you must say, what you must obtain before collecting, and what you may never do with the result. None of them will tell you how long a trace may sit in your observability vendor, which is why the technical controls in this article have to exist independently of any statute.

For anyone shipping in Europe the analysis differs in shape. Health data is a special category under Article 9 of the GDPR, so processing is prohibited by default and permitted only on a narrow set of grounds, with explicit consent the usual one for a consumer product. The right to erasure under Article 17 then applies to everything you derived from it, which is where the engineering gets hard.

One operational consequence of the patchwork deserves attention before you plan around it. Because Washington and Nevada attach no size threshold, the usual startup reasoning that compliance work can wait until a revenue milestone does not apply here, and the consent flow has to exist on the day the feature ships to the first user in either state. Geographic gating is the alternative teams reach for, and it works poorly: IP-based location is approximate, users travel, and a wrong answer means collecting special-category data without valid consent. Building the strict flow once for everyone is usually cheaper than maintaining two flows and a location oracle that decides between them.

What enforcement has actually looked like

Statutory text tells you what is prohibited. Enforcement history tells you what regulators actually pursue, and the pattern in this area has been consistent: the cases are about disclosure into advertising and analytics infrastructure rather than about a classic intrusion.

The Federal Trade Commission’s Health Breach Notification Rule sat unused from 2009 until 2023. Its first application was against GoodRx, which agreed to a $1.5 million civil penalty over sharing user health information with advertising platforms through tracking pixels. Months later the ovulation tracking app Premom, published by Easy Healthcare, settled a similar action with a $100,000 civil penalty. BetterHelp, the therapy platform, paid $7.8 million to consumers over disclosures to advertising services after promising confidentiality; that action ran under the FTC Act’s prohibition on unfair and deceptive practices rather than under the breach rule.

The Commission finalised an expansion of the rule in 2024 that put the interpretation beyond argument. Health apps and connected devices are in scope, and a “breach of security” includes an unauthorised disclosure the company performed itself. A pixel firing on a page, a third-party SDK reading a screen, or an analytics event carrying a diagnosis string all count. No attacker is required.

Three lessons transfer directly to AI features.

The first is that the sanctioned behaviour was a data flow engineers set up for ordinary reasons. Nobody at those companies decided to sell health records. Somebody added a marketing tag, and the tag saw whatever was on the page.

The second is that the privacy promise creates the liability. BetterHelp’s exposure grew out of having said something specific and then doing otherwise. A team that writes “we do not train on your health data” and then copies flagged conversations into an evaluation set has built exactly that structure, if a reasonable user would read the promise as covering the copy.

The third is that penalties here have been modest while the reputational cost has not. Those numbers would barely register at a well-funded company. The coverage, the consent decree, and twenty years of compliance reporting are the real price.

Both of the flows regulators went after were invisible on an architecture diagram. That makes the next step obvious: enumerate where the data actually lands.

Where the data goes that nobody mentions

Three destinations account for most surprise disclosures, and none of them are training.

Observability platforms are the largest. Modern applications ship request and response data to logging and tracing systems as a matter of course, and those systems are optimised for retention and searchability rather than confidentiality. Access is typically broad across engineering, retention is measured in months, and the data frequently leaves your infrastructure entirely for a vendor. A team can honestly say they do not train on your data while a full transcript sits in a log aggregator that forty engineers can query.

Evaluation datasets are the second. Improving a product means measuring it, and measuring it means collecting examples of real usage where the system did badly. Those examples are real user data by construction, they get copied into a dataset that lives outside the production pipeline, and they persist far longer than any conversation retention policy because the dataset is treated as an engineering asset. Whether this counts as “training” is a definitional question that different companies answer differently.

Caching is the third and the least visible. Prompt caches, embedding caches, and response caches all hold content, sometimes in shared infrastructure, and cache eviction policies are written for performance rather than for data protection. A deletion request that propagates correctly through your primary database may leave the same content in a cache for as long as it stays warm.

The common thread is that each of these was built by engineers solving a legitimate problem, without a data classification conversation, and each one predates the sensitive feature being added. That is why the audit question worth asking internally is not “do we train on this” but “list every system that holds a copy”.

Every system a health message touches

Naming those three destinations helps more once you can see them in sequence. Draw the path a single message takes from the moment a user types a symptom into a chat box, then count the systems that end up holding a copy. Teams doing this exercise for the first time usually find between eight and fifteen, and the ones they forget are reliably the same: the load balancer access log, the error tracker, the feature flag evaluation events, and the support tool that renders a conversation for a human agent.

The diagram is deliberately unflattering. Every box is a defensible engineering decision, and together they mean a single sentence about a medication can be read by a support agent, a data analyst, an on-call engineer looking at a trace, and whoever at your APM vendor holds production access, all while nobody trains a model on anything.

Destination	Typical retention	Who can read it	Usually in the deletion path
Primary application database	Product policy	Service accounts, database administrators	Yes
Gateway and access logs	7 to 90 days	Platform and SRE teams	Rarely
Tracing and APM vendor	15 to 90 days	Most of engineering	Rarely
Error tracker	Around 90 days	Most of engineering	Rarely
Prompt and response cache	Minutes to days	Service accounts	Rarely
Vector index	Lifetime of the document	Any service holding index credentials	Sometimes
Analytics warehouse	Years	Analysts, data science, BI users	Rarely
Evaluation dataset extract	Indefinite	ML and quality teams	Almost never
Model provider logs	Provider policy	Provider staff under provider controls	Contractual only
Backups and snapshots	30 to 365 days	Restore operators	On expiry only
Support tooling	Ticket lifetime	Support organisation	Rarely

The rightmost column is the one worth printing out. A deletion feature that reaches only the first row is the normal state of a system built quickly, and it stays defensible right up until somebody asks for a demonstration.

Run the exercise as a workshop rather than as a document review. Put the people who own each system in one room, take a single real request identifier from a staging environment, and have each owner say out loud where that identifier or its content appears in their system and for how long. Ninety minutes produces a more accurate map than a quarter of data-flow paperwork, because the person who added the tracing attribute six months ago is the only one who remembers that it carries the full prompt. Write the result down as a table with an owner per row, and revisit it whenever a new dependency lands.

Embeddings and derived data are personal data

The vector index in that diagram deserves separate treatment, because it is where the promise and the plumbing diverge most sharply. An embedding feels like a derived artefact, a list of floats with no visible relationship to the sentence that produced it, and that intuition is what lets teams drop it from the deletion path without ever writing the exclusion down.

The intuition fails on both the legal and the technical side.

Legally, European regulators have taken the position that data derived from personal data remains personal data while the individual stays identifiable through it, and the European Data Protection Board’s 2024 opinion on AI models declined to treat trained models as automatically anonymous. The assessment turns on whether personal data can be extracted using means reasonably likely to be used. Washington’s health data law reaches the same conclusion by a different route, since its definition of consumer health data explicitly covers inferences.

Technically, embeddings are far less opaque than they look. Inversion research has repeatedly shown that text can be approximately reconstructed from its embedding given access to the same embedding model, and reconstruction improves as the text gets shorter, which is exactly the regime a symptom description falls into. Most production vector stores also keep the source text in the record payload for citation and display, so whether the vector itself qualifies as personal data is often an academic question: the sentence is sitting right beside it.

Then comes the deletion mechanics problem, which surprises engineers rather than lawyers.

• Approximate nearest neighbour indexes such as HNSW do not remove nodes on delete. A delete marks the entry as filtered from results while the vector remains in the graph structure until a compaction or a rebuild reclaims it. The data is gone from queries and present on disk.
• Snapshots and replicas of the index carry the same vectors, on their own schedule.
• Caches of embeddings keyed by content hash outlive the document they were computed for.
• Any model fine-tuned on the content has absorbed it into weights, and no reliable method removes a specific example after the fact. Machine unlearning research offers approximations, with nothing close to the guarantee you get from retraining without the data.

That last point drives the design conclusion more than any of the others. Fine-tuning on sensitive user content converts a deletion obligation you can satisfy into one you cannot, so it deserves an explicit, documented decision rather than becoming the default route to better quality. Retrieval keeps the data in a store you can operate on. Weights give nothing back.

The practical rules are short. Give every derived artefact a foreign key back to its source record, so deletion can cascade. Treat index compaction as part of the deletion job rather than as a background optimisation. Key embedding caches by source identifier as well as content hash, so a purge can find them. And write into the design document that embeddings inherit the classification of their input, before somebody has to guess during an incident.

Questions worth putting to a vendor

If you are buying rather than building, the answers to a short list of questions separate providers who have thought about this from those who have written a marketing sentence.

Ask for the retention period as a number, for each category of data, including logs and evaluation sets. Vague answers here are the reliable signal.

Ask whether human review occurs, under what circumstances, and whether you can opt out. Every provider does some form of review, so a denial means the person answering does not know.

Ask for the subprocessor list and whether you are notified of changes. Enterprise agreements usually include this and consumer terms usually do not.

Ask what deletion means operationally: which systems it reaches, on what timescale, and whether it covers derived data such as embeddings and summaries. A provider who can answer this precisely has built the capability, and one who answers “we delete your data” has probably built a soft delete on one table.

Ask where processing happens, and whether that can be constrained by contract. For regulated data this frequently determines whether the product is usable at all.

A due-diligence questionnaire you can copy

Those five questions belong in a call. The list below belongs in a document, because written answers are attributable, and a vendor who will say something on a call sometimes declines to put it in writing, which is itself the finding.

Send it with space for prose answers rather than as a form with checkboxes. Checkboxes produce columns of yes.

# AI vendor due diligence: sensitive data
# Send before contract. Request written answers, per data category.

residency:
  - In which regions is data processed, per category?
  - In which regions is it stored at rest, per category?
  - Can region be constrained contractually, and what is the remedy if it changes?
  - In which regions do your subprocessors operate?

retention:
  - Retention in days for each of: conversations, prompts, completions,
    request logs, traces, error reports, caches, evaluation extracts, backups.
  - Is zero-retention available, and does it also apply to abuse monitoring?
  - Maximum lag between a deletion request and removal from backups?

human_access:
  - Under what conditions may an employee read customer content?
  - What share of traffic is sampled for safety or quality review?
  - Can review be disabled for our tenant, in the contract?
  - Is that access logged, and can we receive those access logs?

subprocessors:
  - Provide the current list, with each subprocessor's role and location.
  - How much notice before a new subprocessor is added?
  - May we object, and what happens if we do?

deletion:
  - List every system holding customer content, and the deletion lag for each.
  - Does deletion cover embeddings, summaries, extracted attributes, scores?
  - Is index compaction triggered by deletion, or by a schedule?
  - Can you produce evidence of deletion for a named record?

training_and_derived:
  - Define "training" as your terms use the word.
  - Are evaluation sets, red team corpora, or few-shot example libraries
    built from customer content?
  - Does the training exclusion extend to derived artefacts?
  - Does it extend to fine-tunes and adapters as well as base models?

security_and_assurance:
  - Which certifications, and what is the scope statement of each?
  - Date of the most recent penetration test, and a summary of findings?
  - Encryption at rest: whose keys, and can we bring our own?

incident:
  - Which breach notification regime do you believe applies to this data?
  - Notification clock, in hours, from your detection to our notification.
  - Who in your organisation makes the notification decision?

corporate:
  - Do these commitments survive a change of control?
  - On termination, what is the deletion timeline, and can we get proof?

Score the answers rather than filing them. A number earns a point, a link to a policy page earns half, and a paragraph about how seriously the vendor takes security earns zero. Vendors who have built the capability answer the deletion block in specifics within a day, because somebody on their side already owns that runbook. Vendors who have not will route the question to a sales engineer who paraphrases the trust page.

Two answers should stop a purchase for regulated data. The first is a refusal to name subprocessors, since a data path you cannot see is a data path you cannot assess. The second is an inability to state log retention as a number, which usually means the observability stack never entered the privacy review.

Designing for this if you are building it

Teams shipping AI features over sensitive data can adopt a few structural decisions that make the privacy promise easier to keep and easier to verify.

Segregate the sensitive path. Sensitive data should flow through a code path with its own storage, its own retention policy, its own logging configuration, and its own access controls, rather than through the general pipeline with a flag set. Flags get missed, and a single logging statement added by someone unaware of the flag writes protected data to a general-purpose log store.

Log metadata, not content. The instinct to log request and response bodies for debugging is strong and it is how sensitive data ends up in observability platforms with broad internal access and ninety-day retention. Log identifiers, timings, token counts, and error classes. When you genuinely need content for debugging, gate it behind an explicit, audited, time-limited toggle.

Make deletion real and test it. Write a deletion test that creates a record, exercises every downstream system that touches it, issues the deletion, and then asserts absence in each one. Run it in CI. The number of teams that discover their vector index retains embeddings of deleted documents is high, and the discovery usually happens during an audit rather than during development.

Keep derived data in scope explicitly. State in your own design documents that embeddings, summaries, and extracted attributes inherit the classification of their source. Then build the deletion path accordingly.

Prefer not collecting. The strongest control remains not having the data. If a feature can work with a coarse category rather than a precise value, or can compute locally on the device and send only a result, the residual risk drops to something no policy language is needed to manage.

The segregated path in code

Those five decisions are easy to agree with in a meeting and easy to lose in a pull request six months later. They survive by being expressed in types and interfaces rather than on a wiki page, so that writing the wrong thing becomes awkward.

Start with classification as a property of the data, attached where the data is defined.

from dataclasses import dataclass, field, fields
from enum import IntEnum
from typing import Any


class Sensitivity(IntEnum):
    PUBLIC = 0
    INTERNAL = 1
    PERSONAL = 2
    HEALTH = 3          # special category: segregated path only


def classified(level: Sensitivity):
    """Mark a dataclass with the highest sensitivity it may carry."""
    def wrap(cls):
        cls.__sensitivity__ = level
        return cls
    return wrap


def sensitive_field(level: Sensitivity, **kw):
    return field(metadata={"sensitivity": level}, **kw)


@classified(Sensitivity.HEALTH)
@dataclass
class HealthTurn:
    turn_id: str
    user_id: str
    created_at: str
    text: str = sensitive_field(Sensitivity.HEALTH)
    source: str = sensitive_field(Sensitivity.PERSONAL)

    def metadata(self) -> dict[str, Any]:
        """Everything safe to emit. Content is excluded by construction."""
        out: dict[str, Any] = {}
        for f in fields(self):
            level = f.metadata.get("sensitivity", Sensitivity.INTERNAL)
            if level >= Sensitivity.PERSONAL:
                continue
            out[f.name] = getattr(self, f.name)
        out["char_len"] = len(self.text)
        out["sensitivity"] = self.__sensitivity__.name
        return out

The important detail is the metadata() method. Logging code never receives the object itself, so an engineer who wants to log something gets handed the safe view and has to go out of their way to reach the unsafe one. Redaction applied on the way out eventually fails. A shape that never carries the content past the boundary rarely does.

The logger then refuses anything it was not handed deliberately.

import json
import logging

_ALLOWED = {
    "turn_id", "user_id", "created_at", "source",
    "char_len", "sensitivity", "model", "latency_ms",
    "prompt_tokens", "completion_tokens", "outcome", "error_class",
}


class MetadataOnly(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        ctx = getattr(record, "ctx", None)
        if ctx is None:
            return True
        leaked = set(ctx) - _ALLOWED
        if leaked:
            # Fail loudly in CI; drop the keys in production.
            raise ValueError(f"unallowed log keys: {sorted(leaked)}")
        record.msg = json.dumps({"event": record.msg, **ctx}, separators=(",", ":"))
        return True


log = logging.getLogger("health")
log.addFilter(MetadataOnly())

# turn is a HealthTurn instance; only its metadata view ever reaches the logger.
log.info("turn.completed", extra={"ctx": {
    **turn.metadata(),
    "model": "assistant-v4",
    "latency_ms": 812,
    "outcome": "ok",
}})

An allowlist beats a denylist here for the same reason it does in firewall rules. New fields appear constantly, and a denylist only protects you against the ones you already thought of.

Routing is the third piece. A sensitive turn goes to a store with its own credentials, its own retention job, and its own encryption key, while the general repository refuses it outright rather than accepting it with a flag set.

class GeneralRepository:
    def save(self, obj) -> None:
        level = getattr(obj, "__sensitivity__", Sensitivity.INTERNAL)
        if level >= Sensitivity.HEALTH:
            raise PermissionError(
                f"{type(obj).__name__} is HEALTH classified; "
                f"use SegregatedRepository"
            )
        ...

That exception has caught more real mistakes in production codebases than any policy document, because it converts a governance decision into a failing test.

A deletion test that asserts absence

The deletion test mentioned earlier is worth writing out, because its shape determines whether it is worth anything. A test that calls the delete endpoint and asserts a 204 tests the endpoint. A test that asserts absence in every store tests the promise.

The propagation path is the design input.

Two properties of that diagram matter more than the boxes. The fan-out is explicit, so adding a new store forces an edit to the deletion job and a reviewer sees it. And the tombstone records which systems were covered, so an auditor asking you to prove that record 4471 was deleted receives an answer instead of a shrug.

The test then walks the same fan-out.

import pytest

SYSTEMS = ["primary", "vector", "cache", "warehouse", "search", "objects"]


@pytest.mark.integration
def test_deletion_removes_health_turn_everywhere(env):
    turn = env.create_health_turn(
        text="Metformin 500mg twice daily, fasting glucose 6.1"
    )

    # Force every downstream system to actually hold a copy.
    env.run_embedding_pipeline(turn.turn_id)
    env.run_completion(turn.turn_id)          # populates cache and traces
    env.run_warehouse_sync()
    env.run_search_indexer()
    assert env.presence(turn.turn_id) == {s: True for s in SYSTEMS}

    env.delete_user_data(turn.user_id)
    env.drain_deletion_queue(timeout_s=120)

    survivors = [s for s, present in env.presence(turn.turn_id).items() if present]
    assert not survivors, f"content survived deletion in: {survivors}"

    # Absence of the identifier does not prove absence of the content.
    assert env.vector_search(turn.text, k=5) == []
    assert env.warehouse_contains("Metformin 500mg") is False
    assert env.tombstone(turn.turn_id)["systems"] == sorted(SYSTEMS)

The second half is the part teams leave out. Checking that a row with the identifier is gone passes trivially against a soft delete. Searching for the distinctive content string catches the copy written into a denormalised table under a different key, and it catches the vector still reachable because compaction never ran.

Run it against a real environment on a schedule rather than on every commit, since it is slow and it needs real infrastructure to mean anything. Then wire its failure to the same alert channel as a production incident, because a silent regression in the deletion path is a compliance finding that has yet to be discovered.

Deciding whether a feature may take sensitive data

Deletion tests and segregated stores answer the question once you have decided to accept the data. The prior question, asked far too rarely, is whether a particular feature should receive it at all, and a written decision procedure keeps that from being settled by whoever is most enthusiastic in the room.

Three branches on that tree deserve comment.

The coarse category branch removes more risk than any control further down. A medication adherence feature that stores “on a daily prescription” rather than the drug name has stepped out of most of the analysis above. Teams skip the branch because the precise value is more useful, which is true, and because measuring how much more useful takes work nobody scheduled.

The on-device branch has become genuinely practical for a narrower set of tasks than vendors claim and a wider set than most teams assume. Classification, redaction, and entity extraction run acceptably on a phone. Anything needing a frontier model does not, and pretending otherwise produces a feature that works in the demo and quietly falls back to the cloud in production.

The subprocessor branch fails late. A path can be correct inside your own infrastructure and still route through an analytics vendor whose terms cover general personal data while saying nothing about special categories. Put that check in the design review, when moving the boundary costs a conversation rather than a migration.

Common mistakes and anti-patterns

The failures in this area repeat with enough regularity to be listed. Most of them are ordinary engineering habits colliding with a data category that punishes them.

Redacting on the way out instead of never collecting. A redaction layer over log writes is the most common design and the most fragile. It depends on a pattern list, and health content resists patterns: a sentence describing a diagnosis contains nothing a regex can find. Shape the data so the content never reaches the logger.

Treating a flag as the control. A boolean is_sensitive on a general record means every future code path has to remember to check it. The number of code paths grows and the memory does not. Separate storage is the control; the flag is a label.

Assuming the vendor’s default is your policy. Model providers differ on retention for abuse monitoring, and the default for a self-serve API key is frequently not the default under an enterprise agreement. Teams prototype on the self-serve key, ship, and never revisit which account the traffic runs through.

Calling an evaluation set something other than data. A CSV of two thousand real conversations where the model performed badly is a copy of the most sensitive content in the product, held outside the retention policy, on a laptop that syncs to a personal cloud drive. Give evaluation extracts a retention period, a location, and an owner, or accept that they are your breach.

Deleting the row and keeping the derivative. Covered above, and worth repeating because it survives every round of remediation aimed at the primary datastore.

Letting the privacy policy describe an aspiration. Every sentence in a policy is a factual claim about a system. Somebody has to confirm the sentence still matches the implementation after the migration that changed retention.

Building consent as a single checkbox. Washington and Nevada both require consent to stand apart from other terms, and a bundled acceptance covering terms of service, marketing, and health data collection is the exact pattern those statutes were written against.

Testing deletion by identifier only. The identifier is what your delete query uses, so of course it is gone. Search for the content.

Giving the AI feature the same database credentials as the rest of the application. Retrieval code with broad read access will eventually pull something into a context window that then travels to a provider, a trace, and a cache. Scope the credential to the records the feature needs.

Assuming a training exclusion covers a fine-tune. Some teams read “we do not train foundation models on your data” as covering adapters and fine-tunes on customer content, and some vendors read it the other way. Get the definition in writing, which is why that line sits in the questionnaire above.

Running the deletion job without alerting on partial failure. A queue that retries forever and never pages anyone produces a system where deletion works for most records and silently stalls for the rest, usually the ones with unusual attachments.

The thread connecting these is that each one is a reasonable shortcut in a system handling ordinary data. Sensitive categories remove the margin, and the only reliable way to notice is to walk the data path with a list rather than reason about it from the architecture diagram.

Writing the disclosure so it survives a reader

Controls and tests establish what is true. The disclosure decides whether anybody believes it, and drafting it is a separate skill from building the system.

The rule that produces good privacy text is to write the sentence a journalist would write on finding out, then publish that sentence yourself. Vague phrasing reads as concealment the moment a reader learns the specifics from somewhere else.

Common phrasing	What a reader takes from it	Replace with
”We take your privacy seriously”	Nothing at all	Delete the sentence
”We retain data only as long as necessary”	Indefinitely	”Conversations: 30 days. Logs: 14 days. Backups: 35 days."
"Authorised personnel may access your data”	Anyone at the company	”About 1 in 500 conversations is reviewed by trained safety staff. Health conversations are excluded."
"We may share data with service providers”	Sold to advertisers	A named list, each with its role, linked and versioned
”You can delete your data at any time”	One click and it is gone	”Removed from active systems within 24 hours and from backups within 35 days."
"We do not sell your data”	Depends on the definition	Say what you do with it, then say this
”We use your data to improve our services”	Training, probably	Name the uses: quality review, evaluation sets, aggregate metrics

Structure matters as much as wording. A health data disclosure that works has four parts in this order: what is collected, where it goes, how long it stays, and how to make it stop. Give it a page of its own instead of burying it in section 7.3 of a general policy, write it at the reading level of a discharge summary rather than a contract, and show it at the moment of connection rather than at signup.

A worked example, in the voice of the product rather than of the legal team.

When you connect your health records, we store them in a separate system from the rest of your account. We keep the connected records for as long as the connection is active, delete them within 24 hours of you disconnecting, and remove them from our backups within 35 days. Your health conversations are excluded from human review and from model training. Three companies process this data on our behalf: [cloud provider] hosts it, [model provider] generates responses under a contract that forbids retention beyond 30 days, and [analytics provider] receives no health content at all. You can export or delete everything from Settings, and we will show you what was deleted.

Every clause there is checkable, which is the point of writing it that way. The last one, showing the user what was deleted, is unusual enough to be worth building: a deletion receipt listing the record types removed and the systems covered turns an assertion into evidence, and it costs one screen on top of a tombstone table you already needed for your own audits.

One caution about timing. Disclosure drafted before the implementation is finished tends to describe the intended system, and in the intended system deletion always propagates in seconds. Write the text from the output of the deletion test.

What to tell users, and what to ask as one

The disclosure that earns trust is specific and includes the awkward parts. “We retain conversations for 30 days, a small sample may be reviewed by trained staff for safety, we do not train on your health data, deletion removes content from active systems within 24 hours and from backups within 35 days” tells a reader what they are actually agreeing to. A single sentence about training does not.

As a user of these products, the questions worth asking before connecting a medical record are shorter. How long is it kept, can a human read it, and what happens when I delete it. The answers are usually findable in a privacy policy, and the ones that are not findable are themselves an answer.

Connecting health records to a capable assistant is a genuinely useful feature, and treating it as automatically reckless is not the right conclusion. The right conclusion is that the value is real, the promise made is narrow, and the questions that determine your actual exposure are the eight that were not addressed in the announcement.