It's different from what I was thinking. Here is a Chinese LLM generated example for irony.
## A Concrete, Step-by-Step Example
To make this tangible, let's use a **toy model with a small vocabulary of 11 words** so you can see every number. The real scheme works identically, just with vocabularies of ~50,000 tokens and floating-point probabilities.
### Setup
**The model:** A tiny "LLM" with a fixed vocabulary of 11 words:
`{apple, date, banana, cherry, pie, juice, tart, sauce, for, with, and}`
At each step, the model assigns a probability to **all 11 words** based on the context. The probabilities change as the context grows, and they always sum to 1.0. (A real LLM does the same thing, just with ~50,000 tokens instead of 11.)
**The shared context/prompt:** `"I like to eat"` — both Alice and Bob have this. It is not secret, just shared.
**The shared secret key:** Used to seed a pseudorandom number generator (PRNG). The key produces the random stream: `0.15, 0.62, 0.40, ...` (uniform random numbers between 0 and 1).
**The secret message Alice wants to send:** The bits `0 1 1` (3 bits — in reality this would be a longer message, but let's keep it tiny).
---
### Encoding (Alice's Side)
#### Step 1: Alice runs the model
Alice feeds the shared context `"I like to eat"` into the model. The model outputs a probability distribution over all 11 words. After "I like to eat", fruits are the most natural continuation, so they get the highest probabilities:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.20 | [0.00, 0.20) |
| date | 0.15 | [0.20, 0.35) |
| banana | 0.10 | [0.35, 0.45) |
| cherry | 0.05 | [0.45, 0.50) |
| pie | 0.20 | [0.50, 0.70) |
| juice | 0.12 | [0.70, 0.82) |
| tart | 0.08 | [0.82, 0.90) |
| sauce | 0.05 | [0.90, 0.95) |
| for | 0.03 | [0.95, 0.98) |
| with | 0.015 | [0.98, 0.995) |
| and | 0.005 | [0.995, 1.00) |
This is just normal LLM behavior — the model is predicting what word comes next. Notice the probabilities sum to 1.0, and the intervals partition [0, 1).
#### Step 2: Alice encodes her first secret bit
Alice's first secret bit is `0`. She interprets this as: "look in the **first half** of the range [0, 1), i.e., [0.00, 0.50)."
The words whose intervals fall within [0.00, 0.50) are:
- apple: [0.00, 0.20)
- date: [0.20, 0.35)
- banana: [0.35, 0.45)
- cherry: [0.45, 0.50)
Alice uses her next random number from the shared key, `0.15`, to pick within [0.00, 0.50). The value `0.15` falls in [0.00, 0.20), which is **apple**.
**Alice outputs: "apple"**
Text so far: `"I like to eat apple"`
#### Step 3: Alice runs the model again
Now the context is `"I like to eat apple"`. The model outputs a new distribution over all 11 words. After "apple", food preparations like pie and tart become more likely, while the other fruits become less likely:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.15 | [0.00, 0.15) |
| date | 0.10 | [0.15, 0.25) |
| banana | 0.08 | [0.25, 0.33) |
| cherry | 0.07 | [0.33, 0.40) |
| for | 0.06 | [0.40, 0.46) |
| with | 0.04 | [0.46, 0.50) |
| pie | 0.20 | [0.50, 0.70) |
| tart | 0.16 | [0.70, 0.86) |
| juice | 0.08 | [0.86, 0.94) |
| sauce | 0.04 | [0.94, 0.98) |
| and | 0.02 | [0.98, 1.00) |
Alice's next secret bit is `1`. This means "look in the **second half** [0.50, 1.00)."
The words whose intervals fall within [0.50, 1.00) are:
- pie: [0.50, 0.70)
- tart: [0.70, 0.86)
- juice: [0.86, 0.94)
- sauce: [0.94, 0.98)
- and: [0.98, 1.00)
Alice uses her next random number, `0.62`, to pick within [0.50, 1.00). She scales it into that range: `0.50 + 0.62 × 0.50 = 0.81`. The value `0.81` falls in [0.70, 0.86), which is **tart**.
**Alice outputs: "tart"**
Text so far: `"I like to eat apple tart"`
#### Step 4: Alice runs the model again
Context: `"I like to eat apple tart"`. Model output — after "apple tart", connectors and prepositions like "with" and "and" become the most likely continuations, while the fruits and preparations drop in probability:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.10 | [0.00, 0.10) |
| date | 0.08 | [0.10, 0.18) |
| banana | 0.07 | [0.18, 0.25) |
| cherry | 0.06 | [0.25, 0.31) |
| pie | 0.08 | [0.31, 0.39) |
| juice | 0.06 | [0.39, 0.45) |
| for | 0.05 | [0.45, 0.50) |
| tart | 0.06 | [0.50, 0.56) |
| sauce | 0.04 | [0.56, 0.60) |
| with | 0.18 | [0.60, 0.78) |
| and | 0.22 | [0.78, 1.00) |
Alice's next secret bit is `1`. Look in the second half [0.50, 1.00):
The words whose intervals fall within [0.50, 1.00) are:
- tart: [0.50, 0.56)
- sauce: [0.56, 0.60)
- with: [0.60, 0.78)
- and: [0.78, 1.00)
Random number `0.40` → scaled: `0.50 + 0.40 × 0.50 = 0.70`. Falls in [0.60, 0.78), which is **with**.
**Alice outputs: "with"**
Text so far: `"I like to eat apple tart with"`
#### Alice is done
Alice sends Bob the text: **"I like to eat apple tart with"**
(Plus whatever padding/continuation she wants to make it look like a complete sentence.)
To any observer, this looks like someone generated a sentence about food. Nothing suspicious.
---
### Decoding (Bob's Side)
Bob has:
- The received text: `"I like to eat apple tart with"`
- The same model
- The same shared context: `"I like to eat"`
- The same shared key (so the same random stream: `0.15, 0.62, 0.40, ...`)
#### Step D1: Bob runs the model
Bob feeds `"I like to eat"` into the model. He gets the **same** distribution Alice got:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.20 | [0.00, 0.20) |
| date | 0.15 | [0.20, 0.35) |
| banana | 0.10 | [0.35, 0.45) |
| cherry | 0.05 | [0.45, 0.50) |
| pie | 0.20 | [0.50, 0.70) |
| juice | 0.12 | [0.70, 0.82) |
| tart | 0.08 | [0.82, 0.90) |
| sauce | 0.05 | [0.90, 0.95) |
| for | 0.03 | [0.95, 0.98) |
| with | 0.015 | [0.98, 0.995) |
| and | 0.005 | [0.995, 1.00) |
Bob sees that Alice chose **apple**, which is in the interval [0.00, 0.20).
Now Bob asks: "Which half was apple in?" Apple is in [0.00, 0.20), which is in the **first half** [0.00, 0.50).
**First half → bit `0`**
**Bob recovers bit: `0`** ✓
#### Step D2: Bob runs the model again
Bob appends "apple" to context: `"I like to eat apple"`. Model output:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.15 | [0.00, 0.15) |
| date | 0.10 | [0.15, 0.25) |
| banana | 0.08 | [0.25, 0.33) |
| cherry | 0.07 | [0.33, 0.40) |
| for | 0.06 | [0.40, 0.46) |
| with | 0.04 | [0.46, 0.50) |
| pie | 0.20 | [0.50, 0.70) |
| tart | 0.16 | [0.70, 0.86) |
| juice | 0.08 | [0.86, 0.94) |
| sauce | 0.04 | [0.94, 0.98) |
| and | 0.02 | [0.98, 1.00) |
Bob sees Alice chose **tart**, interval [0.70, 0.86).
"Which half was tart in?" [0.70, 0.86) is in the **second half** [0.50, 1.00).
**Second half → bit `1`**
**Bob recovers bit: `1`** ✓
#### Step D3: Bob runs the model again
Context: `"I like to eat apple tart"`. Model output:
| Word | Probability | Interval |
|--------|-------------|---------------|
| apple | 0.10 | [0.00, 0.10) |
| date | 0.08 | [0.10, 0.18) |
| banana | 0.07 | [0.18, 0.25) |
| cherry | 0.06 | [0.25, 0.31) |
| pie | 0.08 | [0.31, 0.39) |
| juice | 0.06 | [0.39, 0.45) |
| for | 0.05 | [0.45, 0.50) |
| tart | 0.06 | [0.50, 0.56) |
| sauce | 0.04 | [0.56, 0.60) |
| with | 0.18 | [0.60, 0.78) |
| and | 0.22 | [0.78, 1.00) |
Bob sees Alice chose **with**, interval [0.60, 0.78).
"Which half?" [0.60, 0.78) is in the **second half** [0.50, 1.00).
**Second half → bit `1`**
**Bob recovers bit: `1`** ✓
---
### Result
Bob has recovered: `0 1 1` — exactly the message Alice sent.
---
## Diagram
```
┌──────────────────────────────────────────────────────────────────┐
│ ALICE (ENCODER) │
│ │
│ Secret bits: 0, 1, 1 │
│ Shared context: "I like to eat" │
│ Shared key → random stream: 0.15, 0.62, 0.40 │
│ │
│ For each bit: │
│ 1. Run model with current context → distribution │
│ 2. Use bit to pick which HALF of [0,1) to look in │
│ 3. Use random# to pick a word within that half │
│ 4. Append word to context, repeat │
│ │
│ Output: "I like to eat apple tart with" │
└──────────────────────────────────────────────────────────────────┘
│
│ (sent over channel — looks innocent)
▼
┌──────────────────────────────────────────────────────────────────┐
│ BOB (DECODER) │
│ │
│ Received text: "I like to eat apple tart with" │
│ Shared context: "I like to eat" (SAME as Alice) │
│ Shared key → random stream: 0.15, 0.62, 0.40 (SAME as Alice) │
│ │
│ For each word: │
│ 1. Run model with current context → SAME distribution │
│ 2. Find which interval the word falls in │
│ 3. Determine which HALF that interval is in │
│ 4. That half = the recovered bit │
│ 5. Append word to context, repeat │
│ │
│ Recovered bits: 0, 1, 1 ✓ │
└──────────────────────────────────────────────────────────────────┘
```
---
## Key Points
1. **The LLM never saw the secret message.** The secret bits controlled *which word was picked* from the distribution. The LLM just produced distributions.
2. **The output looks natural.** "I like to eat apple tart with" is a perfectly normal sentence. A censor seeing this has no reason to be suspicious.
3. **Both sides run the same model with the same context.** This is why they get the same distributions. If Bob had a different model or different context, the distributions would differ and decoding would fail.
4. **The shared key provides the random stream.** This is needed for the encoding to be secure — without it, the censor could run the same model and try to decode. With the key, the censor can't reproduce the random choices.
5. **Each token encodes roughly 1 bit** in this toy example (because we split into 2 halves). In reality, with arithmetic coding over a 50,000-word vocabulary, you can encode **multiple bits per token** — roughly 2–4 bits per token for a real LLM, depending on how uncertain the model is at each step.
6. **The "half" splitting is the simplified version.** The real scheme (from the Meteor paper, eprint.iacr.org/2021/686) uses full arithmetic coding, which is more efficient — it doesn't waste the probability mass the way a 2-way split does. But the principle is the same: the secret bits steer the selection within the model's probability distribution.
