Private AI Has a Compute Problem. A New Paper With Fair Math and NVIDIA Researchers Shows Where the Bottleneck Is Moving
From 3,523-bit arithmetic to sub-8 ms bootstrapping, a new research paper examines the infrastructure challenge behind private AI — and why GPUs, memory bandwidth and numerical architecture may matter as much as the models themselves.
AI has a data problem that bigger models cannot solve.
Many of the applications with the highest potential economic value also depend on some of the world's most sensitive information: medical records, financial data, proprietary enterprise knowledge, customer information and personal data.
The problem is not necessarily whether AI can understand this data. It is whether organizations can safely give AI systems access to it in the first place.
A new paper, Efficient Large-Integer Arithmetic for FHE, co-authored by researchers including teams from Fair Math and NVIDIA, looks at this problem several layers beneath the model itself. The paper examines how the computational architecture required to process encrypted information has evolved, why today's dominant approaches became successful, and where their limitations are beginning to appear.
The underlying technology is Fully Homomorphic Encryption, or FHE: a form of cryptography that allows computation to be performed while data remains encrypted.
But the more interesting story for the AI industry is not the cryptography itself. In fact, we believe it is what has to happen computationally before private AI becomes practical infrastructure.
Privacy is becoming a constraint on where AI can actually be deployed
Inside Deep Tech asked Gurgen Arakelov, founder of Fair Math, why privacy remains one of AI's major open challenges.
"Privacy is becoming a fundamental constraint on where and how AI can be deployed. Many of the most valuable AI use cases involve highly sensitive data and in many of these areas, sending raw data to a third party for inference is simply not acceptable.
This means privacy is not just a security feature. It directly determines which data AI can access and, ultimately, which problems AI can solve. If we want AI to become part of critical infrastructure and operate on the world's most sensitive data, we need a way to compute without exposing that data.”
That distinction matters.
Privacy is often discussed as a feature added after an AI product has been built: permissions, access controls, data retention policies or anonymization. Encrypted computation proposes something structurally different.
Instead of trusting the infrastructure provider not to expose the data, the goal is to make it possible for the infrastructure to perform the computation without gaining access to the underlying plaintext in the first place.
The paper opens with exactly this pressure: AI is increasing demand for privacy-preserving computation across healthcare, finance, cloud computing and edge AI, where models may need to operate on information that cannot simply be disclosed to service providers. The mathematics makes that possible. The remaining question is economics.
Private AI asks today's hardware to perform very unusual arithmetic
One of the clearest numbers in the paper is 64 bits.
That is roughly the width of the fixed-size integers contemporary processors handle efficiently. The arithmetic required for the encrypted computation analyzed in the paper can instead involve coefficient moduli spanning hundreds or thousands of bits.
At the highest parameter range shown in the paper's table, a ring dimension of 131,072 corresponds to a maximum ciphertext modulus of approximately:
3,523 bits
under the specific 128-bit security parameters presented by the authors.
Among the representative applications listed for these larger configurations is private LLM inference.
That does not mean an LLM simply “uses 3,523-bit numbers.” The architecture is more complicated than that. What the number illustrates is the scale mismatch engineers have to overcome. The problem becomes even more visible when looking at a ciphertext.
In one example discussed by the authors, a ciphertext contains polynomials with 16,384 coefficients, with each coefficient represented modulo an integer hundreds of bits wide. A single homomorphic multiplication can consequently require millions of modular operations involving integers far larger than native machine words.
For an AI infrastructure company, these mathematical details eventually become very familiar business metrics: latency, memory, GPU utilization, throughput and cost per inference.
The breakthrough was turning one huge problem into many smaller ones
A major part of the paper tracks the industry's transition toward the Residue Number System, or RNS.
Conceptually, RNS allows a very large number to be represented through a collection of smaller residues. Instead of repeatedly manipulating the enormous integer directly, much of the computation can happen independently across these smaller channels.
That is important for AI infrastructure because independent computation maps exceptionally well onto parallel hardware.
The paper describes how operations across polynomial coefficients and RNS components can be executed independently, while transformations used in encrypted computation can be broken into large numbers of parallel operations.
In other words, a cryptographic representation decision made years ago also happens to create a workload that GPUs can attack very effectively.
And that is where the NVIDIA side of the story becomes especially relevant.
GPUs are beginning to compress the private-compute gap
The rise of modern AI was not driven by model architecture alone.
It was the combination of algorithms, massive parallelism, specialized software and rapidly improving GPU hardware that changed what became economically possible.
Encrypted AI appears to be benefiting from a similar dynamic.
The Fair Math/NVIDIA paper surveys an earlier GPU implementation that achieved homomorphic multiplication in approximately:
51 milliseconds
at multiplicative depth 98 and 128-bit security.
The paper reports this as two orders of magnitude faster than the CPU results used for comparison.
That particular benchmark is not an AI inference benchmark, and it should not be interpreted as one. Its significance is architectural.
If one of the expensive primitive operations underneath encrypted computation can improve on the scale of 100× through hardware acceleration and implementation design, workloads previously dismissed as impractical deserve to be periodically re-evaluated. And newer GPU results cited in the paper make that argument considerably more interesting.
Below 8 milliseconds across eight B200 GPUs
The next number is even more aggressive. The survey references Cerium, which evaluates a CKKS bootstrap in: under 8 ms across eight NVIDIA B200 GPUs.
The obvious objection is that eight B200s represent an enormous amount of compute for a single operation. And that is precisely why this benchmark is useful. It shows both sides of private AI's current state.
The technical barrier is falling rapidly. The economic barrier has not disappeared. What happens next depends on how quickly those curves converge.
Better arithmetic algorithms can reduce the amount of work required. GPU generations can increase throughput and memory bandwidth. Software can improve utilization. Batching can amortize costs. Workloads can become more carefully optimized around encrypted computation.
Those improvements can compound. That is the same pattern that repeatedly changed the economics of conventional AI infrastructure.
134 seconds for an encrypted Llama3-8B inference may be the most important number in the paper
The fastest benchmark is not necessarily the most revealing.
The survey also cites a Cerium evaluation of encrypted Llama3-8B inference in approximately 134 seconds. At first glance, more than two minutes for an LLM inference sounds commercially unattractive.
For a consumer chatbot, it almost certainly is. But that may be the wrong comparison. Suppose a financial institution, healthcare provider or enterprise cannot send a particular dataset to an external AI service at all because the information is too sensitive. Its choice is not necessarily:
2 seconds of normal inference vs 134 seconds of private inference.
It can instead be:
134 seconds vs not being able to use an external AI model on that data at all.
That produces a very different economic calculation. And it points toward how private AI could reach the market.
Private AI does not need to become cheap everywhere at once
Encrypted AI probably does not need to reach plaintext-compute economics before it creates meaningful businesses. It needs to cross the economic threshold for specific categories of data first.
The simplest way to think about that threshold is:
Value created by accessing sensitive data > additional cost of keeping the computation private.
For a recommendation engine operating on public product data, paying a large private-compute premium makes little sense. For a high-value medical analysis, confidential financial computation or proprietary enterprise workload, the equation can look completely different. That suggests the first significant private-AI markets may emerge from the top of the privacy-value curve, rather than from mass-market consumer inference.
It also explains why the business opportunity is larger than simply selling “more secure AI.” If privacy determines which datasets can legally, contractually or commercially be brought into an AI system, then improving privacy infrastructure can effectively increase the amount of usable data available to AI. That is a market-expansion argument, not merely a cybersecurity argument.
The next AI bottleneck may be memory movement
There is another important AI infrastructure signal in the paper. The authors note that encrypted workloads can be constrained not only by arithmetic throughput, but by memory bandwidth and data movement.
GPUs bring large register files, high-throughput GDDR/HBM memory and substantial on-chip caches, which can help reduce some of these bottlenecks compared with conventional CPU architectures. This should sound familiar to anyone following AI infrastructure.
As accelerators have become extraordinarily fast, feeding them data efficiently has increasingly become part of the performance problem. Private AI intensifies that challenge because ciphertext representations can be much larger and operations can involve extensive movement across arithmetic components.
In other words, the competition may increasingly move from:
Who has the fastest cryptography? toward: Who has the best encrypted-compute system architecture?
That includes arithmetic representations, GPU kernels, memory layouts, compilers, batching strategies and ultimately hardware/software co-design.
And Fair Math's paper asks whether today's dominant architecture is the final one
The paper becomes especially interesting toward the end because it does not assume the architecture that enabled today's performance improvements will necessarily remain optimal. Modern implementations largely moved toward RNS because it converted large arithmetic into parallel machine-word operations extremely well suited to current hardware.
But Full RNS introduces its own structural costs. Representation of large numbers, the transforms required for polynomial computation, precision management and key switching become interconnected.
The authors highlight renewed research into positional arithmetic, where some of these concerns are decoupled again. Recent approaches discussed in the survey can make the number of transforms involved in certain key-switching operations grow linearly rather than quadratically with the number of limbs, while recovering more flexible, bit-granular rescaling.
For approximate arithmetic such as CKKS, highly relevant to machine-learning workloads, this remains an exploratory direction rather than an established replacement for today's RNS implementations.
But commercially, the open question is significant. The private-AI stack may still be early enough that important pieces of its eventual computational architecture have not yet settled.
What this means for AI businesses
Taken together, the numbers in the paper describe a technology crossing from a primarily cryptographic challenge into a broader AI systems-engineering problem. The survey gives us several useful markers. We encourage you to read the original paper for more information.
Long story short, none of these benchmarks means private AI has reached cost or performance parity with conventional AI. That is not yet the interesting question.
The question is whether performance is improving quickly enough that specific valuable datasets cross the threshold from “AI cannot touch this” to “AI can process this privately at an acceptable premium.”
If that happens, privacy-preserving computation becomes much more than a security layer. It becomes infrastructure that expands the addressable market for AI itself. And that is why the work examined by Fair Math and NVIDIA researchers matters beyond the FHE community. The next breakthrough in private AI may not come from another model.
It may come from making the mathematics underneath the model cheap enough that, eventually, the user no longer needs to know it is there.
About Fair Math
Fair Math is building cryptographic infrastructure for privacy-preserving AI and encrypted computation. Its current stack includes infrastructure designed to allow AI models and agents to operate directly on encrypted data, alongside an FHE computing platform and open-source development tooling.
Backed by: Polymorphic Capital, Inception Capital, gumi Cryptos Capital, Stake Capital and Nazaré Ventures.
