LLM Steganography Demo

This demo lets you hide a secret message inside ordinary-looking AI-generated text. The output reads like a normal sentence a language model might produce, but it secretly encodes your message bit by bit.

The practical point is covert communication: two people who share a key can exchange messages that, to anyone watching, look like innocuous GPT-2 text. There's no obvious ciphertext, no encrypted file, and no metadata screaming "this is encrypted." The secret is hidden in plain sight.

It's also a neat demonstration of how much information is packed into every token a language model emits — each token can carry a full secret bit while still looking natural.

This demo uses a half-splitting entropy coding scheme built on top of GPT-2's next-token probability distribution.

1. The secret message is converted to a bit string (UTF-8 → bits).
2. For each secret bit, GPT-2 produces a probability distribution over the entire vocabulary for the next token.
3. Tokens are sorted by probability (descending) and split into two halves at the 50% cumulative probability mark.
4. Bit 0 → the next token is sampled from the first (higher-probability) half; bit 1 → from the second half.
5. A shared-key PRNG (mulberry32) selects the exact token within the chosen half, so the decoder can reproduce the same random draws.
6. The decoder re-runs GPT-2 on the same context, observes which half each cover token fell into, and recovers the bits → original message.

Because both sides share the same model, context, and PRNG seed, the decoder can perfectly reconstruct the hidden bits. The resulting cover text reads like normal GPT-2 output, hiding the secret in plain sight.

To make this tangible, let's use a toy model with a small vocabulary of 11 words so you can see every number. The real scheme works identically, just with vocabularies of ~50,000 tokens and floating-point probabilities.

Setup

The model: A tiny "LLM" with a fixed vocabulary of 11 words: {apple, date, banana, cherry, pie, juice, tart, sauce, for, with, and}. At each step, the model assigns a probability to all 11 words based on the context. The probabilities change as the context grows, and they always sum to 1.0.

Shared context/prompt: "I like to eat" — both Alice and Bob have this. It is not secret, just shared.

Shared secret key: Used to seed a PRNG, producing the random stream: 0.15, 0.62, 0.40, ...

Secret message: The bits 0 1 1 (3 bits).

Encoding (Alice's Side)

Step 1: Alice runs the model

Alice feeds "I like to eat" into the model. After "I like to eat", fruits are the most natural continuation, so they get the highest probabilities:

The probabilities sum to 1.0, and the intervals partition [0, 1).

Step 2: Alice encodes her first secret bit

Alice's first secret bit is 0 → "look in the first half [0.00, 0.50)." The words in that half are apple, date, banana, and cherry. Alice uses her next random number, 0.15, to pick within [0.00, 0.50). The value 0.15 falls in [0.00, 0.20), which is apple.

Alice outputs: "apple" → text so far: "I like to eat apple"

Step 3: Alice runs the model again

Context is now "I like to eat apple". After "apple", food preparations like pie and tart become more likely:

Alice's next bit is 1 → "look in the second half [0.50, 1.00)." Random number 0.62 → scaled: 0.50 + 0.62 × 0.50 = 0.81, which falls in [0.70, 0.86) → tart.

Alice outputs: "tart" → text so far: "I like to eat apple tart"

Step 4: Alice runs the model again

Context: "I like to eat apple tart". After "apple tart", connectors like "with" and "and" become most likely:

Alice's next bit is 1 → second half [0.50, 1.00). Random number 0.40 → scaled: 0.50 + 0.40 × 0.50 = 0.70, falls in [0.60, 0.78) → with.

Alice outputs: "with" → text so far: "I like to eat apple tart with"

Alice is done

Alice sends Bob the text: "I like to eat apple tart with" (plus whatever padding she wants to make it look like a complete sentence). To any observer, this looks like someone generated a sentence about food. Nothing suspicious.

Decoding (Bob's Side)

Bob has the received text, the same model, the same shared context "I like to eat", and the same shared key (so the same random stream: 0.15, 0.62, 0.40, ...).

Step D1: Bob runs the model

Bob feeds "I like to eat" into the model and gets the same distribution Alice got. He sees Alice chose apple, interval [0.00, 0.20). "Which half?" First half [0.00, 0.50) → bit 0 ✓

Step D2: Bob runs the model again

Context: "I like to eat apple". Same distribution as Alice's Step 3. Bob sees Alice chose tart, interval [0.70, 0.86). "Which half?" Second half [0.50, 1.00) → bit 1 ✓

Step D3: Bob runs the model again

Context: "I like to eat apple tart". Same distribution as Alice's Step 4. Bob sees Alice chose with, interval [0.60, 0.78). "Which half?" Second half [0.50, 1.00) → bit 1 ✓

Result

Bob has recovered: 0 1 1 — exactly the message Alice sent.

Key Points

1. The LLM never saw the secret message. The secret bits controlled which word was picked from the distribution. The LLM just produced distributions.
2. The output looks natural. "I like to eat apple tart with" is a perfectly normal sentence. A censor seeing this has no reason to be suspicious.
3. Both sides run the same model with the same context. This is why they get the same distributions. If Bob had a different model or different context, decoding would fail.
4. The shared key provides the random stream. Without it, a censor could run the same model and try to decode. With the key, the censor can't reproduce the random choices.
5. Each token encodes roughly 1 bit in this toy example (2-way split). With arithmetic coding over a 50,000-word vocabulary, you can encode ~2–4 bits per token.
6. The "half" splitting is the simplified version. The real scheme (from the Meteor paper) uses full arithmetic coding, which is more efficient. But the principle is the same: secret bits steer selection within the model's probability distribution.

This project is inspired by research into linguistic steganography — the art of hiding information within natural language. Traditional steganography hides data in images, audio, or file metadata, but text-based steganography is harder because text is compact and every character is visible.

The breakthrough idea is to use a language model's own probability distribution as the carrier channel. Instead of tweaking pixels or LSBs, you steer which token the model emits next, encoding one secret bit per token. Key inspirations include:

• Entropy coding approaches (e.g. Cachin's information-theoretic steganography) that treat the cover as a probability distribution and map secret bits onto it.
• "Dissociated Press" / Markov-chain steganography — early experiments that used statistical text generation to hide messages, limited by the poor quality of pre-LLM language models.
• Modern LLM steganography research — papers that use GPT-style models to achieve near-perfect cover text quality while encoding multiple bits per token via arithmetic or half-splitting codes.

The half-splitting scheme used here is one of the simplest robust approaches: it guarantees exactly one bit per token, is symmetric (encoder and decoder run the same logic), and produces text that's indistinguishable from normal model output to a casual reader.