Hackers can read private AI-assistant chats even though they’re encrypted

A Deep Dive into Token-Length Side-Channel Attacks on AI Assistants

LLMs have rapidly become integral to daily operations at work, and within a short span, they are already being trusted with highly sensitive business information. Employees often turn to these AI-powered tools for assistance with drafting confidential documents, reviewing emails that contain proprietary trade secrets, seeking advice on delicate HR issues, or analyzing internal reports. Without realizing it, people are sharing critical company secrets and sensitive personal information with LLMs, from discussions about product strategies to financial data.

However, new research has uncovered a novel attack that can decode AI assistant responses with alarming precision. This method takes advantage of a side-channel vulnerability present in nearly all major AI assistants, with the notable exception of Google Gemini. By utilizing large language models specifically trained for this task, the attack refines the raw data intercepted between the user and the assistant. As a result, a passive adversary monitoring the traffic between the user and the AI assistant can determine the topic of 55% of captured responses, often with high accuracy. Even more concerning, 29% of responses can be deduced with perfect word-for-word accuracy.

In a recent paper titled "What Was Your Prompt? A Remote Keylogging Attack on AI Assistants", researchers from Ben-Gurion University of the Negev unveil a novel side-channel attack that leverages token-length patterns from encrypted traffic to reconstruct user queries and AI-generated responses. This groundbreaking discovery has massive implications for IT professionals tasked with safeguarding data privacy and security in an era where AI plays a central role in business and personal communication.

The Token-Length Side-Channel Attack: What It Is and How It Works

Encryption is supposed to be the frontline defense against eavesdropping on sensitive communications. But the token-length side-channel attack discovered by the researchers bypasses this defense in a highly sophisticated way, making it possible for attackers to infer sensitive information without decrypting any data.

Here’s how the attack works:

1. Tokenization in AI Models: AI language models break down user input and generate responses in small units called tokens (often corresponding to words or subwords). Each time a user sends a prompt or query, the model processes it and responds by generating a sequence of tokens.
2. Packet Size as a Leak: When communicating with an AI assistant, the input prompt and the output tokens are sent as encrypted packets. Although the content of these packets is encrypted, the packet size remains visible. Since the number of tokens correlates directly to the packet size, an attacker observing network traffic can infer the length of each token sequence.
3. Reconstructing the Conversation: By analyzing the size of the encrypted packets, the attacker can build a sequence of token lengths. Using this information in conjunction with their own large language model, the attacker can reconstruct significant portions of the user’s input and the AI's responses—without ever decrypting the traffic.

In simpler terms, even though the data transmitted between the user and the AI assistant is encrypted, an attacker can still piece together meaningful information simply by observing the size and sequence of the packets exchanged.

Key Findings from the Study

The researchers tested their attack on two popular AI systems—OpenAI’s GPT-4 (via ChatGPT) and Microsoft Copilot—across various platforms, including web apps and integrated development environments (IDEs). The results are eye-opening:

• 29% of Full Responses Reconstructed: In nearly a third of cases, the attacker was able to reconstruct entire user-AI conversations by observing token-length sequences.
• 55% Success in Inferring Conversation Topics: In over half of the attempts, the attack succeeded in deducing the subject matter of conversations. While this might not reveal exact details, knowing the general topic can be enough to compromise sensitive information, such as revealing a company’s project discussions or financial consultations.
• Applicable to Both Web and Desktop Applications: The attack demonstrated that it could work regardless of whether the AI assistant was accessed through a browser or a desktop application, increasing its potential scope for exploitation.
• Cross-Platform Vulnerability: The attack was successful across various operating systems, meaning it could affect organizations using Windows, macOS, or even mobile platforms, making it a versatile threat that isn’t bound to a single environment.

As AI assistants become increasingly integrated into critical workflows—from handling customer support to generating code for development teams—their use inevitably involves processing sensitive data. Whether it’s personal identifiable information (PII), proprietary business information, or other confidential material, the data passing through these systems must remain secure.

Here are some of the specific reasons why this attack is significant:

1. Threat to Confidentiality: Even if encryption is in place, the ability to infer prompts and responses through packet size undermines the confidentiality of the communication. This is particularly troubling for industries dealing with sensitive customer data, such as healthcare, finance, and legal sectors, where even partial exposure of conversation content could lead to serious breaches.
2. Enterprise AI Usage: Many businesses are integrating AI assistants into their day-to-day operations. Whether for automating routine tasks, developing software, or assisting employees with complex queries, AI assistants process vast amounts of organizational data. If attackers can infer sensitive information through side-channel attacks, they may gain unauthorized insights into business strategies, financial data, or intellectual property.
3. Compliance and Legal Risks: Many industries are governed by stringent regulations like GDPR, HIPAA, and CCPA, which mandate strict data protection measures. A side-channel attack that exposes personal or sensitive information—even indirectly—could result in hefty fines, loss of reputation, and legal action against the organization.

Mitigation Strategies: What Can Be Done?

While this attack is sophisticated and novel, there are some actionable steps that IT professionals can take to mitigate the risk:

1. Random Padding of Packets: By adding random padding to packets before they are sent, organizations can obscure the relationship between token length and packet size. While this increases the bandwidth needed, it effectively breaks the attacker’s ability to use packet size as an indicator of token length.
2. Token Grouping: Another solution involves batching tokens together and sending them in larger, fixed-size groups. This would reduce the granularity of the information that the attacker can infer from packet sizes, although it could slightly degrade the real-time responsiveness of the AI system.
3. Use of Differential Privacy: Applying differential privacy techniques can prevent LLMs from memorizing or leaking specific details about users' queries in response to prompts. This can limit the damage if partial information is exposed through side-channel analysis.
4. Network-Level Defense: Organizations can implement additional network-level security measures to monitor and restrict suspicious traffic patterns. For example, by detecting unusual packet size fluctuations or analyzing outgoing traffic for patterns that match known side-channel attack vectors, companies can proactively block such attacks.
5. Red Teaming and Security Audits: Regular security assessments, including red-teaming exercises, should be conducted to test for vulnerabilities in AI systems and communication channels. This will help identify and patch any weaknesses before they can be exploited.

Looking Ahead: AI Security Needs to Evolve

As AI continues to be integrated into more sensitive and mission-critical operations, it’s essential for IT professionals to not only keep up with the technological advancements but also with the evolving security landscape. The battle to protect sensitive data is ongoing, and this research underscores the need for constant vigilance.

‍