ReplayMark: Probe-and-Replay Model-Response
Watermarking for Diffusion Language Models

NeurIPS 2026 · FLMSec Workshop★ Distinguished Paper Award

Chia-Ming Lee^1,3Wen-Hsin Tsai²Tung Yuk Ying²Chih-Chung Hsu¹

¹ National Yang Ming Chiao Tung University² National Cheng Kung University³ University at Albany, State University of New York

Paper OpenReview Code

TL;DR: ReplayMark adds a hidden watermark by favoring model responses that match a secret key. To verify text later, replay the original model and measure that agreement.

Why is a word green? Follow the color assignment, then compare the two watermarking paradigms.
Conceptual examples. View the full-size animation

The idea

Where the green and red labels come from

Green means the group favored by the secret key. Red is the other group. These labels describe the watermark preference.

In a word-list method, a hash uses the context and key to assign candidate words to the two lists. In ReplayMark, the model predicts with two different parts of the earlier context hidden. The key chooses which direction of change counts as green.

Take the word “carpet”

Word-list method

The hash assigns carpet to the red list in this example.

Its color comes from keyed list membership.

ReplayMark

Probe A predicts 15%; Probe B predicts 35%. The key favors B > A, so carpet is green.

Its color comes from the model’s response direction.

Illustrative values. A probe is a prediction made with part of the earlier context hidden. Both methods’ groups can change with context.

What changes between the two approaches?
Question	Token watermark	ReplayMark
What is green?	Words assigned to a keyed green list.	Words whose model responses match the keyed direction.
How is it verified?	Rebuild the lists and count green words.	Replay the original model and count green responses.
Is the original model needed to verify?	No model forward pass for the token-list baseline.	Yes—the response is the evidence.

ReplayMark selects usable positions before the key chooses a response direction.

Technical detail: compare the generation procedures

For a closer look, compare the ordinary decoder with the extra steps ReplayMark adds.

Generic dLLM Decoding

Require: prompt x, model θ, decoder F

1  y ← masked continuation
2  while masked positions remain:
3    p ← model predictions for y
4    J ← positions chosen by F
5    for i in J:
6      v ← reference token draw from pᵢ
7      yᵢ ← v
8  return y

ReplayMark: Probe and Write

Require: same x, θ, F; key K, R, κ

1  y ← masked continuation
2  while masked positions remain:
3    p ← model predictions for y
4    J ← same reference schedule F
5    for i in J:
6      rebuild probes; select carrier
7      if carrier: consult keyed direction
8        try ≤ R plausible matching draws
9        otherwise: unchecked fallback draw
10     else: reference token draw
11     yᵢ ← selected token
12 return y

Conceptual pseudocode. Carriers are selected independently of the keyed direction; accepted candidates must have probability at least κ times the live maximum. Verification re-masks the finished text, rebuilds the probes and carriers, and applies the exact binomial-tail test. See the paper for the complete construction.

View the original paper figure (Figure 1)

Token-level watermarking counts keyed tokens from text alone. ReplayMark queries the same checkpoint on re-masked text and counts keyed model-response matches. — Figure 1 from the paper. The signal moves from token identities to the checkpoint’s response to re-masked context.

How it works

How ReplayMark writes and verifies a watermark

The provider uses the key while writing. Later, the verifier repeats the prediction experiment with the original model.

1
Find usable responses
Hide different parts of the earlier context and compare the predictions. Keep positions where plausible words support both directions of change.
2
Favor the keyed direction
After selecting a position, let the key choose the green direction. Favor a plausible word with that response. The model keeps its ordinary order for filling missing words.
3
Replay and count
Use the finished text to reconstruct the hidden-context predictions. Count the responses that agree with the key and measure how surprising that agreement would be by chance.

Watch generation and verification step by step

Generate → keep the text → check later. The verifier repeats the prediction experiment, without a generation log.
Illustration only; the example matches are not measured results.

View the original paper figure (Figure 2)

During generation, two hidden-prefix probes split candidate tokens into response directions; a key selects one direction. During verification, the same checkpoint replays those probes on the finished text. — **Write with the model. Verify by replay.** The decoding schedule stays intact; the verifier reconstructs the probes from the finished text.

Explore which positions can carry a watermark

The key needs room to choose. Use positions that support both response directions; skip ones that are almost decided.
Illustrative probabilities. The measured example from the paper is available below.

View the measured carrier map from the paper (Figure 3)

Carrier availability in a saved 512-token LLaDA output, arranged in 16 blocks of 32 positions. Blue positions pass the response-balance threshold; outlined positions are skipped. — **Where can the signal live?** In this saved LLaDA example, 490 of 512 positions pass the carrier threshold. Positions close to revealed text are often too predictable to support both directions. Figure 3 from the paper.

From responses to evidence

How much agreement is convincing?

One green response provides little evidence. A larger number of matches can be harder to explain by chance. Try changing the responses below to see how the evidence changes.

Why the statistical test works

For text produced without the document key, each selected response has a 50–50 chance of matching. The exact binomial test measures the probability of getting at least the observed number of matches by chance.

p = Pr[Binomial(n, ½) ≥ T]

The guarantee assumes independent hash domains and text produced without the document key. Carrier selection must not use the keyed direction.

Experimental results

What the experiments show

The main setting detects 80% of watermarked LLaDA outputs and 90% of Dream outputs at a 1% false-positive threshold. The settings transfer between the two models without retuning.

80%LLaDA outputs detected

90%Dream outputs detected

144Model calls to verify 512 tokens

View a saved model-output example

Can you tell which text is watermarked? The text looks ordinary. Replaying the model reveals the difference.
Real saved outputs and full-document scores; animation timing is illustrative. Source

Explore the detection, quality, and compute results

Detection, writing quality, and cost

Select a model below. Detection rate measures how often a watermark is found; the quality ratio and model calls show what it costs.

Detection and cost · LLaDA-8B-Instruct
Method / setting	TPR @ 1% ↑	TPR @ 0.1% ↑	PPL ratio ↓	Generation passes ↓	Verification passes ↓
KGW δ = 1	0.90	0.83	1.22×	512	0
dgMARK k = 1	0.83	0.80	1.45×	512	0
dgMARK +3-beam	0.93	0.93	1.21×	~2,048	0
ReplayMark R = 1	0.43	0.20	0.97×	656	144
ReplayMark R = 8, κ = 0.1	0.70	0.60	0.87×	656	144
ReplayMark R = 16, κ = 0.05	0.80	0.60	0.74×	656	144
Dream-7B-Instruct
KGW δ = 1	0.57	0.37	1.84×	512	0
dgMARK k = 1	0.37	0.27	1.75×	512	0
dgMARK +3-beam	0.30	0.30	0.71×	~2,048	0
ReplayMark R = 1	0.40	0.27	1.34×	656	144
ReplayMark R = 8, κ = 0.1	0.70	0.67	0.79×	656	144
ReplayMark R = 16, κ = 0.05	0.90	0.90	1.09×	656	144

Table 1 of the paper. C4 realnewslike prompts; 512-token continuations; 32-token blocks. Detection cohorts: n = 30, or n = 20 for R = 16. Dream PPL uses 8–13 scored pairs. PPL ratios are relative to each method’s own unwatermarked decoder.

The takeaway: The main setting detects 80% of watermarked LLaDA outputs and 90% of Dream outputs at a 1% false-positive threshold. The tradeoff is verification: it needs the original model and 144 model calls. Some baselines detect LLaDA output more often and require no model calls to verify.

Downstream task performance MMLU · GSM8K · HumanEval

Table 2 · Task accuracy under greedy and multinomial decoding
Model / method	Greedy			Multinomial
Model / method	MMLU	GSM8K	HumanEval	MMLU	GSM8K	HumanEval
LLaDA-8B-Instruct
Unwatermarked	0.605	0.740	0.409	0.595	0.640	0.335
KGW	0.615	0.680	0.384	0.535	0.740	0.293
dgMARK	0.595	0.660	0.396	0.530	0.660	0.341
dgMARK +3-beam	0.595	0.620	0.329	0.530	0.660	0.280
ReplayMark	0.590	0.680	0.335	0.575	0.700	0.384
Dream-7B-Instruct
Unwatermarked	0.740	0.840	0.549	0.695	0.660	0.427
KGW	0.680	0.760	0.524	0.625	0.700	0.421
dgMARK	0.740	0.720	0.530	0.570	0.560	0.421
dgMARK +3-beam	0.745	0.580	0.470	0.645	0.560	0.329
ReplayMark	0.745	0.700	0.470	0.710	0.680	0.439

ReplayMark: R = 16, κ = 0.05. MMLU and GSM8K report accuracy; HumanEval reports pass@1 (higher is better). Evaluated on 200, 50, and 164 problems, respectively. Harnesses differ across methods; the paper’s accuracy-change analysis uses each method’s paired unwatermarked runs.

Scope & limitations

Requirements and limits

Verification needs the original model and the registered key. Providers or auditors with that access can reconstruct the response groups.

Original model access

Use the same checkpoint, numerical precision, key, and probe settings. The reported 512-token setup needs 144 model calls to verify—about 27 seconds on a TITAN RTX.

Rewriting can erase the signal

In the reported editing study, detection holds up better when edits stay together. Scattered edits hurt more, and full paraphrasing removes detection. Short fragments also provide less evidence.

A positive result is evidence of agreement with the registered watermark. Requester identity and passage-level attribution are outside its scope. The study covers two models and small test cohorts.

Citation

BibTeX

@inproceedings{lee2026replaymark,
  title     = {ReplayMark: Probe-and-Replay Model-Response
               Watermarking for Diffusion Language Models},
  author    = {Chia-Ming Lee and Wen-Hsin Tsai and
               Tung Yuk Ying and Chih-Chung Hsu},
  booktitle = {NeurIPS 2026 Workshop on Foundations of
               Language Model Security},
  year      = {2026},
  url       = {https://openreview.net/forum?id=fmXX4kVZ30}
}