AI must be confidential

In 2024, Dell Technologies and Enterprise Strategy Group modeled enterprise LLM usage at 50 queries per user per day, describing that as a moderate level of demand.

Multiply that across a 5,000-person organization, across a year, and you get something on the order of 50 million prompts flowing into a handful of AI providers. Most of these prompts are mundane: rewriting emails, summarizing meetings, debugging code, drafting slides.

The risk that matters isn't in any single prompt. It's in their aggregation.

When security teams evaluate AI risk, they typically evaluate it one prompt at a time. Is this request sensitive? Does it contain personal data? Is this document confidential? If the answer is no, the interaction feels harmless. The prompt goes through and the work gets done.

This logic is dangerously incomplete. The real risk of enterprise AI is not only the sensitive prompt. It is the aggregation of a hundred million non-sensitive ones.

The Instagram lesson

Consider what Meta or Amazon know about you. Not from any single tap, scroll, or three-second video pause. Each of those is meaningless on its own. The intelligence comes from aggregated ambient data: billions of trivial signals, combined over time, resolving into a model so precise it can predict your purchases before you make them and your emotions before you feel them.

The same mechanism that makes social platforms unnervingly accurate at advertising is now being pointed at something far more revealing than scroll behavior. It is being pointed at prompts.

Behavioral data on social or shopping platforms is implicit, inferred from what you click and how long you linger. Prompt data is explicit. Users are typing their reasoning, their concerns, their workarounds, org charts, project names, and unfinished thoughts directly into a text box owned by a third party. There has never been a richer dataset about how a user or an organization actually thinks and operates. And right now, most of it is being processed in plaintext on infrastructure the security team does not control.

What aggregation makes visible

Take a representative slice of an employee's prompts over a month. Individually, almost none would fail a DLP scan:

• "Rewrite this in a more diplomatic tone."
• "Summarize the action items from this transcript."
• "Draft a follow-up to the vendor about the missed deadline."
• "Help me debug this Terraform error."
• "What's the best way to phrase a counter-offer?"

Now aggregate a thousand of these from one person. You get communication style, decision patterns, stress signals, who they negotiate with, what they negotiate over, what infrastructure they run, what they're insecure about, and what they're confident about. You get a behavioral model.

Aggregate across an organization, and you get an operational blueprint: org structure, supplier relationships, internal terminology, technology stack, security posture, project codenames, compliance gaps, strategic priorities, and the specific frictions that surface in day-to-day work. None of this requires a single "sensitive" prompt. It only requires enough of them.

This is the same pattern that produced targeted advertising, electoral micro-segmentation, and credit scoring. Aggregation is how trivial data becomes strategic intelligence. The question is not whether AI providers actually construct these profiles. It is whether the architecture they're using makes it possible for them, or attackers, to do so.

Prompts are uniquely revealing

There is a category difference between prompt data and other forms of telemetry. Search queries reveal intent. Location pings reveal routines. Transaction logs reveal behavior. But prompts reveal cognition. They capture how people reason, what they struggle with, what they prioritize, and how they frame problems before they have a finished answer.

A search query is the result of thinking. A prompt is the thinking.

This is what makes AI interaction data the most valuable behavioral signal ever collected at scale, and the most dangerous one to aggregate outside an organization's control. A sufficiently large corpus of an enterprise's prompts is functionally a mind-reading device pointed at that enterprise.

Contracts are not architecture

The standard enterprise response is to negotiate data handling terms. Zero retention. No training on our data. A DPA.

These commitments matter, but they are promises, not guarantees. Contracts can be breached, intentionally or accidentally. Providers get acquired and policies change. Subprocessors get added. Internal access controls get misconfigured. Logs are retained for abuse detection. Breaches happen, and when they do, the data was always there to be breached.

More importantly, the next generation of frontier models is being trained on staggering amounts of interaction data. Even with rigorous filtering, patterns at scale leak through. A model trained on years of enterprise interactions across an industry will absorb how that industry makes decisions, even if no individual prompt is memorized verbatim.

The only durable answer to an architectural problem is an architectural fix.

Confidential AI as the default

Confidential AI inverts the trust model. Instead of asking the provider to promise not to look, it makes looking impossible.

The technology exists today. Prompts and responses can be processed inside hardware-enforced Trusted Execution Environments (TEEs) running on NVIDIA GPUs in confidential computing mode. The data remains isolated from the host operating system, from the cloud provider, and from the inference operator itself.

Remote attestation provides cryptographic proof that the code running inside the environment is exactly what was claimed. Not a modified version. Not a logged version. Not a version with a debug hook.

The practical consequence: there can be no aggregated dataset of an enterprise's prompts sitting on someone else's infrastructure, because there is no plaintext for anyone to retain or aggregate.

Edgeless Systems is building confidential AI. The technology I am describing is a design requirement that should be standard for every inference endpoint. NVIDIA confidential computing — Privatemode

This is already becoming a standard for the most sensitive workloads: healthcare, defense, financial services. The argument I want to make is that the sensitive/non-sensitive distinction breaks down at scale. The prompts that would fail a DLP check are not the only dangerous ones.

What to do now

For organizations running AI at any meaningful scale, three things are worth doing now:

Estimate prompt volume by provider. Most security teams have no idea how many prompts their organization sends to external LLMs daily.

Audit the trust model, not just the contract. Ask the AI providers what prevents them from reading prompts, not what they promise about them. The answers are revealing.

Pilot confidential inference for one workload. It does not need to be the most sensitive one. It needs to be the one where the organization can learn what production confidential AI feels like, before having to deploy it everywhere.

The architectural decisions enterprises make this year about AI will outlast the current generation of models.

Whatever we expose now will eventually be analyzed by something far more capable than what exists today.

Originally published on X