How AI Systems Create New PHI Exposure Vectors: STT Pipelines, LLM Logging, and Vector Embeddings
The Memorial Hermann Health System paid $2,400,000 to OCR in May 2017 for impermissible PHI disclosures — a case that predates modern AI deployments. Today, AI-driven care coordination systems introduce three entirely new PHI exposure vectors that HIPAA's drafters never anticipated: speech-to-text transcription pipelines, large language model inference logging, and vector embedding databases that encode patient context. Each vector operates outside the perimeter that traditional HIPAA compliance programs were designed to protect.
️ HHS OCR Resolution Agreement — Memorial Hermann Health System
| Announced: | May 10, 2017 |
| Settlement: | $2,400,000 plus corrective action plan |
| Covered Entity: | Memorial Hermann Health System, Houston TX |
| Violation: | Impermissible PHI disclosure — patient name disclosed in press release |
| PHI Disclosed: | Patient name linked to protected health condition in public statement |
| Root Cause: | Absence of policies governing PHI disclosure in communications workflows |
Memorial Hermann's violation — disclosing a patient's name alongside protected health information in a public communication — illustrates the core principle of 45 CFR §164.502(a): PHI may only be used or disclosed as HIPAA explicitly permits. What's changed since 2017 is not the regulation; it's the number of automated systems that handle PHI in ways no compliance officer explicitly reviewed and approved. AI systems are the largest new source of these unreviewed PHI flows.
What Counts as PHI in an AI Context
Before examining AI-specific exposure vectors, it's essential to understand what constitutes PHI under 45 CFR §160.103. PHI is any individually identifiable health information that relates to: (1) the past, present, or future physical or mental health condition of an individual; (2) the provision of health care to an individual; or (3) the past, present, or future payment for the provision of health care. Critically, PHI includes information that identifies or could be used to identify the individual — including name, address, dates, phone numbers, account numbers, and 12 other categories enumerated in §164.514(b)(2).
In AI systems, PHI appears in forms that traditional security tools were not built to detect:
- Free-text conversation content — A patient saying "I'm calling about my metformin prescription" creates PHI the moment it's associated with their caller ID or account record
- Audio recordings — Voice files processed by speech-to-text systems constitute PHI when they contain identifiable health information
- Vector embeddings — Numerical representations of patient conversations that encode clinical context without appearing as readable text
- LLM inference logs — Prompt-completion pairs logged by AI infrastructure providers that capture patient context sent to the model
- Derived metadata — System-generated annotations like "caller expressed medication concern" that link patient identity to health status
The Core Regulatory Framework
General PHI Use and Disclosure Rules
A covered entity may not use or disclose PHI except as permitted or required by the Privacy Rule. Every call to an external AI API that transmits patient context is a potential disclosure requiring authorization or an applicable exception.
De-identification Standard
PHI is only de-identified if Safe Harbor (removing 18 specific identifiers) or Expert Determination (statistical analysis confirming re-identification risk is very small) standards are met. Vector embeddings derived from patient conversations are not de-identified merely because they are numerical.
Risk Analysis Obligation
Organizations must conduct an accurate and thorough risk analysis of all systems that create, receive, maintain, or transmit ePHI. This includes AI inference APIs, STT processing pipelines, and vector databases — even when operated by a BAA-covered vendor.
Exposure Vector #1: Speech-to-Text Transcription Pipelines
When a patient calls a healthcare organization and speaks to an AI assistant, their voice is captured, transmitted to a speech-to-text (STT) engine, converted to text, and then passed to a language model. Each step in this pipeline is a distinct ePHI handling event requiring HIPAA-compliant controls.
The typical cloud STT pipeline for healthcare AI looks like this:
- Audio capture — Patient voice recorded at the telephony layer (VoIP, PSTN bridge)
- Audio transmission — Raw audio sent over HTTPS to STT API endpoint (Google Speech-to-Text, AWS Transcribe Medical, Azure Cognitive Services)
- Transcription processing — Audio processed on cloud infrastructure; transcript generated
- Transcript return — Text transcript returned to AI system for LLM processing
- Vendor log storage — STT vendor may retain audio and transcript for quality/training purposes unless explicitly opted out
The training data retention trap: Major cloud STT providers retain audio and transcripts by default for service improvement. AWS Transcribe Medical, for example, stores audio files in customer-controlled S3 by default — but the API call metadata and model training data policies vary by service tier. Healthcare organizations that do not explicitly configure data deletion and disable training data use may be transmitting patient voice PHI to infrastructure that retains it beyond the session. This retention requires explicit BAA coverage and deletion procedures.
The PHI risk in STT pipelines is concentrated at two points: transmission (audio containing PHI sent to third-party infrastructure) and storage (transcripts retained in vendor logging systems). 45 CFR §164.502(a) requires that any disclosure to the STT vendor be covered by either a valid BAA (if the vendor is a business associate) or an applicable HIPAA exception. The BAA must specifically address audio data, not just text transcripts.
Exposure Vector #2: LLM Inference Logging
Every API call to a large language model — OpenAI, Anthropic, Google Gemini, or any hosted model — involves sending a prompt that may contain patient PHI. AI healthcare systems routinely construct prompts that include patient context: appointment history, medication lists, recent diagnoses, insurance information. This context-enriched prompt is transmitted to the LLM provider's infrastructure for inference.
The PHI exposure risk in LLM inference occurs at three levels:
Level 1: Prompt Transmission
The prompt itself — containing patient name, date of birth, diagnosis codes, medication names — is transmitted to the LLM provider's API endpoint. This transmission is a PHI disclosure. The LLM provider must be a business associate with a signed BAA. As of 2026, major providers (Azure OpenAI Service, AWS Bedrock, Google Cloud Vertex AI) offer BAA coverage under enterprise agreements. Standard consumer-tier APIs (api.openai.com directly) do not include BAA coverage and must not receive PHI.
Level 2: Provider-Side Logging
LLM providers log API requests for abuse detection, debugging, and rate limiting. These logs contain the full prompt — including PHI. Under enterprise BAA agreements, providers commit to: not using customer data for model training, deleting logs within defined retention periods, and applying appropriate access controls. Organizations must verify these commitments are reflected in the executed BAA, not just marketing materials.
Level 3: Application-Side Logging
Most AI application frameworks (LangChain, LlamaIndex, custom orchestration layers) implement verbose logging by default. These application logs capture: the full prompt sent to the LLM, the completion received, intermediate reasoning steps, tool call arguments, and retrieved context. When any of this content includes patient PHI, the application log becomes a PHI-containing record requiring HIPAA-compliant storage, access controls, and retention management.
The LangChain callback logging problem: LangChain's default callback system logs every LLM call, including the full prompt and response, to stdout and optionally to LangSmith (a third-party observability platform). A healthcare organization deploying a LangChain-based AI assistant that passes patient context to GPT-4 may inadvertently be: (1) logging patient PHI to stdout captured by a centralized logging system without HIPAA controls, and (2) transmitting PHI to LangSmith without a BAA. Neither is intentional — both are defaults that require explicit override.
Exposure Vector #3: Vector Embedding Databases
Retrieval-augmented generation (RAG) architectures store patient context as vector embeddings in a vector database (Pinecone, Weaviate, pgvector, Chroma). These embeddings are numerical representations derived from text containing PHI. Many engineers treat vector stores as "just numbers" that don't contain sensitive data — a misconception with significant HIPAA implications.
Research published in the proceedings of the 2023 IEEE Symposium on Security and Privacy demonstrated that text can be reconstructed from embedding vectors with high accuracy using inversion attacks. A vector embedding derived from "Patient John Smith, DOB 1965-03-15, diagnosis: Type 2 Diabetes, HbA1c 8.2%, prescribed metformin 1000mg" encodes the clinical content of that sentence in recoverable form. The embedding is PHI.
The HIPAA implications are significant:
- Vector databases must be included in the HIPAA risk analysis — They create, receive, maintain, and transmit ePHI under 45 CFR §164.304's definition
- Vector database vendors must be business associates — If the vendor's infrastructure hosts embeddings derived from patient PHI, a BAA is required
- De-identification does not apply — Embeddings fail both Safe Harbor (they can encode all 18 PHI identifiers in recoverable form) and Expert Determination (re-identification via inversion is demonstrated in academic literature)
- Patient right to deletion — If a patient invokes their HIPAA right to request restriction of PHI use (§164.522), organizations must be able to delete the patient's embeddings from the vector store — a technical requirement most vector DB deployments do not implement
The de-identification misconception: Embeddings are not de-identified by virtue of being numerical. 45 CFR §164.514(b)(1) (Expert Determination) requires a qualified expert to determine that re-identification risk is very small. No such determination has been published for modern transformer-based embeddings given the demonstrated success of embedding inversion attacks. Organizations relying on "it's just vectors" as a compliance argument are operating without regulatory foundation.
Applying §164.514(b): The De-identification Standard to AI Outputs
45 CFR §164.514(b) establishes that health information is de-identified — and therefore not PHI — only when one of two standards is met. The Safe Harbor method requires removing 18 specific categories of identifiers including names, geographic data smaller than state, dates other than year, phone numbers, email addresses, SSNs, medical record numbers, and nine others. The Expert Determination method requires a qualified statistician to certify that the risk of re-identification is very small using generally accepted statistical and scientific principles.
AI systems fail the de-identification test in three common scenarios:
Scenario 1: Partial Identifier Removal
An organization removes patient name from LLM prompts but retains date of birth, diagnosis code, zip code, and appointment date. Under Safe Harbor, each retained element is a prohibited identifier. The information remains PHI regardless of how many identifiers were removed — all 18 categories must be absent.
Scenario 2: Statistical Singling-Out
A patient population in a rural county (population 8,000) with a rare diagnosis (prevalence 1:10,000) cannot be de-identified by removing their name — the combination of geographic area and diagnosis uniquely identifies them. Expert Determination would confirm re-identification risk is not small; Safe Harbor would require removing geographic data below the state level.
Scenario 3: Embedding Reconstruction
Embeddings derived from PHI-containing text retain the information content of the source text in a recoverable form. The numerical representation is not de-identified under either standard. Expert Determination has not been applied to this attack surface, and Safe Harbor cannot apply because embeddings implicitly encode the identifiers removed from the surface text.
AI PHI Exposure Audit Checklist: 12 Technical Controls
Map every system that receives patient audio or text during AI interactions. Include the telephony layer, STT API endpoint, LLM API endpoint, application logging system, vector database, and any analytics platforms receiving interaction data.
Confirm a signed BAA exists for each system identified in the map. BAA must specifically address the data type (audio, text, embeddings) that system receives. A generic "HIPAA compliance" certification is not a BAA.
Verify STT vendor data retention and training data policies in writing. Confirm audio is deleted post-transcription. Confirm transcripts are not used for model training without explicit opt-in. Document the policy and the version effective at the time of deployment.
Audit LLM API tier for BAA coverage. Consumer API tiers (api.openai.com, free-tier Google AI) do not include BAA coverage. Only enterprise tiers with executed BAAs (Azure OpenAI, AWS Bedrock, Google Vertex AI under BAA) are HIPAA-compliant for PHI-containing prompts.
Disable or redirect application-level LLM logging. LangChain, LlamaIndex, and custom orchestration layers log prompts by default. Override logging configuration to exclude PHI from log output, or route logs to HIPAA-compliant SIEM with appropriate access controls.
Classify vector embedding stores as ePHI systems requiring Security Rule controls. Apply encryption at rest (AES-256), encryption in transit (TLS 1.3), access logging, and role-based access controls to vector databases containing patient-derived embeddings.
Implement patient data deletion capability for vector stores. When patients request restriction of PHI use under §164.522, the organization must be technically capable of deleting associated embeddings. Test this capability before deployment, not after a patient request arrives.
Validate de-identification claims with a qualified statistician. If any AI system outputs are characterized as "de-identified" to avoid PHI classification, the Expert Determination must be performed by a qualified expert whose analysis is documented and retained. Do not rely on the AI vendor's de-identification claim without independent validation.
Include AI systems in the annual HIPAA risk analysis. STT APIs, LLM providers, vector databases, and AI orchestration systems are all ePHI systems under 45 CFR §164.308(a)(1). The risk analysis must be updated whenever these systems are added, upgraded, or reconfigured.
Implement prompt PHI minimization controls. AI prompts should include the minimum PHI necessary to accomplish the purpose (§164.502(b) minimum necessary standard). Use session tokens and EHR lookup tools rather than embedding full patient records in prompts.
Establish AI-specific incident response procedures. Traditional breach detection (unauthorized database access, stolen credentials) does not detect AI-specific PHI disclosures (PHI in LLM prompts sent to non-BAA endpoints, embedding data exfiltration). AI incidents require purpose-built detection rules.
Review sub-processor chains for AI vendors. Your LLM vendor may use sub-processors for inference compute (NVIDIA Triton clusters), logging (Datadog, Splunk), and model serving. Each sub-processor that receives PHI-containing prompts must be covered by a BAA chain extending from your organization.
How Claire Eliminates AI PHI Exposure at the Architecture Level
1. PHI Never Enters the LLM Prompt — EHR-Side Resolution via MCP
Claire's Model Context Protocol architecture inverts the typical AI data flow. Rather than embedding patient PHI in LLM prompts, Claire passes an opaque session token to the LLM and equips it with EHR FHIR API tools. The LLM requests only the specific data fields needed for each task. PHI is fetched ephemerally from your EHR, used within the session, and never retained in Claire's infrastructure. LLM inference logs contain tool call arguments (FHIR resource types and IDs), not patient health data.
2. No STT Data Retained — Audio Processed and Discarded
Claire's telephony integration processes voice input using HIPAA-BAA-covered STT infrastructure. Audio is transcribed in real-time and deleted immediately after transcription — not stored in S3, not retained for model improvement, not passed to training pipelines. The transcript is processed ephemerally within the MCP session and deleted when the session closes. There is no audio archive and no transcript store in Claire's infrastructure.
3. No Vector Embedding Store for Patient Data
Claire does not maintain a vector database of patient embeddings. Long-term context for returning patients is retrieved from your EHR system at session start via FHIR API — not from a proprietary embedding store. This means there is no embedding database to secure, audit, or delete from when a patient requests PHI restriction. The entire re-identification risk surface of embedding stores is eliminated by architecture, not policy.
4. Application Logging Produces Zero PHI
Claire's application logs contain session IDs, FHIR resource types accessed, response times, and error codes — not patient names, diagnoses, or conversation content. Logs are structured specifically to support HIPAA audit requirements (who accessed what, when) without creating PHI-containing log records. Your security team can monitor Claire's operational logs without handling patient PHI.
The PHI Disclosure Risk in AI Is Not Theoretical
Memorial Hermann paid $2.4M for a single press release. AI systems can generate thousands of equivalent PHI disclosures per day — through LLM prompts sent to non-BAA endpoints, through audio retained in STT vendor infrastructure, through embedding stores that reconstruct patient health information. The difference is that Memorial Hermann's disclosure was deliberate and visible. AI PHI disclosures happen automatically, invisibly, and accumulate over every patient interaction the system processes.
The regulatory framework has not changed since 2017: §164.502(a) prohibits impermissible PHI use and disclosure regardless of the technology involved. What has changed is the number of systems that handle PHI without explicit review, and the number of engineers deploying those systems without recognizing that their architecture creates HIPAA obligations. The audit checklist above is the starting point — but the more durable solution is choosing AI architectures that eliminate PHI exposure by design rather than attempting to retrofit controls onto systems that were built to maximize model context.
For technical teams evaluating AI vendors for healthcare deployment, the right question is not "Are you HIPAA-ready architecture?" but "Show me exactly where patient PHI exists in your system's data flow, and demonstrate the controls protecting each location." A vendor who cannot answer that question with architectural specificity has not solved the PHI exposure problem — they have documented it differently.
