What Does Claude's Text Watermark Actually Tell Us?

Given the secret pseudorandom state, token selection contains deliberate structure. Across that randomness, the expected distribution can remain the same as ordinary model sampling.

A small artificial example makes the distinction easier to see. Suppose a model has two candidate tokens:

A watermarking mechanism does not need to make both tokens equally likely, nor does it need to preserve every individual random draw. Its construction can produce key-dependent decisions while preserving the $0.7/0.3$ marginal distribution when averaged over the relevant watermark randomness.

That is why “the watermark changes token selection” and “the sampler is non-distortionary” can coexist.

It also clarifies what non-distortion does not imply about an individual response. If I submit the same prompt once to an unwatermarked sampler and once to a watermarked sampler, there is no guarantee that

Sampling was stochastic already. Non-distortion concerns distributions over possible outputs.

DeepMind goes beyond the token-level property. By combining Tournament sampling with repeated context masking, the authors construct stronger sequence-level forms of non-distortion. Their main experimental configuration is described as single-sequence non-distortionary.

The stronger guarantee still comes with system-level tradeoffs. DeepMind reports some reduction in inter-response diversity in its non-distortionary configuration. Stronger levels of non-distortion can also reduce detectability or increase computational complexity. A separate distortionary SynthID configuration can trade some quality for a stronger watermark when detectability matters more.

So “non-distortionary” has a precise and useful meaning. It does not mean that every observable property of a deployment remains mathematically identical.

Empirical support for preserved quality

Anthropic goes further than invoking the mathematical property. It says its internal testing has found no impact on content, creativity, or readability, and describes watermarked output as indistinguishable to readers. Those Claude-specific internal results are asserted in the announcement but are not reported there in enough detail for independent evaluation.

The underlying SynthID paper provides much more public evidence for the method.

DeepMind ran a production experiment in Gemini in which a random fraction of queries received watermarked responses and a matched fraction received unwatermarked responses. Across approximately 20 million responses, the reported thumbs-up rates differed by 0.01 percentage points and thumbs-down rates by 0.02 percentage points, with the watermarked model slightly favored in both cases. Neither difference was statistically significant.

The authors also conducted a controlled side-by-side evaluation using 3,000 ELI5 questions and Gemma 7B-IT responses. Human raters assessed grammaticality/coherence, relevance, correctness, helpfulness, and overall quality. They found no significant preference between watermarked and unwatermarked outputs across those dimensions. Automated evaluations similarly found no significant differences in perplexity or benchmark performance.

That is considerably stronger evidence than a few cherry-picked response examples.

I would nevertheless phrase the empirical conclusion carefully. The studies support the claim that the tested non-distortionary SynthID configuration produced no detectable practical quality degradation under a fairly broad set of evaluations. They cannot establish literal zero degradation for every model, task, decoding setting, language, or conception of quality.

There is also an implementation boundary between the evidence and Anthropic’s deployment. DeepMind studied specific SynthID configurations, generally using 30 tournament layers, a four-token seed context, and a Bayesian scoring function in its experiments. Anthropic has disclosed that Claude will use a version of SynthID-Text, without confirming that those settings carry over.

My interpretation of Anthropic’s quality claim is therefore fairly favorable. Changing token selection does not, by itself, demonstrate a quality cost. SynthID’s distributional construction explains why such a cost can be very small, and DeepMind’s empirical validation is unusually large for a watermarking study. “No practical impact” is still an empirical claim tied to what has been measured, rather than a theorem covering every future Claude output.

Detection turns the watermark into a statistical decision

Generation explains how the signal gets into the text. Detection asks whether enough of that signal remains to distinguish the passage from ordinary text.

Given a text and the watermark key, a detector can reconstruct the context-dependent scores associated with the observed token sequence. A simplified score discussed in the SynthID paper has the form

where $T$ is the number of evaluated token positions and $m$ indexes watermarking functions or tournament layers. The actual paper evaluates multiple scoring methods and generally uses a Bayesian scoring function in its main experimental configuration.

Watermarked text should accumulate unusually high keyed correlation. The detector can compare $S(x)$ with a threshold $c$:

That immediately creates a statistical classification problem.

A useful null hypothesis is something like

with an alternative

I prefer this formulation to “human versus Claude.” Human authorship is already a much stronger claim than the statistic directly tests.

DeepMind evaluates detectability primarily through the true-positive rate at a fixed false-positive rate. For example, at FPR $=1\%$, the threshold is chosen so that approximately one percent of the unwatermarked evaluation texts exceed it; the true-positive rate then measures how many watermarked texts exceed that same threshold. The paper also explores selective prediction, where the detector abstains on uncertain samples to meet desired error-rate targets among the remaining classifications.

That abstention idea may become especially important when watermark detectors reach real users.

A false negative depends on what we hoped to detect

The usual definitions are straightforward:

and

Their interpretation depends on how $H_1$ is defined.

Consider four documents.

For the second document, suppose extensive rewriting eliminates the detectable statistical pattern. If the target question is

Does this final token sequence still carry Claude’s watermark?

then a negative result may be perfectly correct.

Suppose the institution actually cares about

Did Claude materially contribute to the intellectual production of this document?

Now the same detector result may function as a practical false negative.

Anthropic is unusually explicit about this boundary. It says the watermark applies only to words Claude chooses; light proofreading may leave too few watermarked choices to detect. A sufficiently complete rewrite can remove the signal. The company also says detection cannot distinguish “Claude wrote this” from “Claude heavily edited this.”

The ambiguity comes from moving between two different latent variables:

and

$W$ can provide evidence about $C$. They are not the same variable.

That gap grows whenever the content passes through transformations after generation.

A denominator is necessary for “Probability Claude wrote this”

Anthropic says that checking whether a sequence is consistent with its keyed choices can allow one to assign a probability that the text was generated by Claude, and elsewhere phrases the target as the likelihood that Claude was “partly” involved. Its detector API has not yet been released, and Anthropic says it is still working out the details of the implementation.

The eventual API semantics will matter.

A watermark score, a $p$-value, a false-positive rate, a likelihood, and a posterior probability such as

represent different statistical quantities.

A posterior probability also depends on the population in which the detector is being used.

Suppose, purely for illustration, that a detector has sensitivity

and false-positive rate

If half the documents being tested really involve Claude, then Bayes’ rule gives

The same detector applied to a population where only one percent of documents involve Claude gives

These numbers are hypothetical and should not be read as estimates of Anthropic’s detector. They illustrate a general point: the evidentiary meaning of a positive detection depends partly on where we looked.

A 19th-century book in a digitized archive and a document submitted through a workflow where Claude use is already common come with very different prior probabilities.

This is one reason I hope Anthropic’s detector exposes the statistical object it actually computes rather than collapsing everything into a percentage labeled “AI-generated.” The latter would be easier to consume and easier to misuse.

For short or low-entropy passages, “insufficient evidence” may be the most informative result available.

Claude involvement is narrower than many real-world questions

At this point, the distinction between detection and provenance becomes operational.

Anthropic describes the watermark as a way to determine the likelihood that Claude was involved in writing text. It also gives several important limits. A watermark carries no information about the individual user, organization, or chat that produced the text. Absence of the Claude watermark cannot confirm human authorship. A Claude detector cannot identify text generated by another provider merely because that provider also uses watermarking.

Those constraints are easier to appreciate if we map them onto the questions people may eventually ask.

These questions touch three concepts that are useful to keep separate:

Provenance concerns where content came from and how it was transformed.

Authorship assigns responsibility or creative contribution under some social, institutional, or legal rule.

Integrity concerns whether an artifact and its claimed history can be trusted.

A statistical trace from Claude can contribute to provenance. Provenance may then become one piece of evidence in an authorship or integrity decision. The intervening inference matters.

Consider a university investigating prohibited AI assistance. The policy may care whether a student delegated reasoning or composition to an LLM. A watermark detector observes patterns in token choices made by Claude. Heavy rewriting, use of another model, or small amounts of Claude assistance can separate those two questions considerably.

A positive detector result could therefore justify further review under an appropriate policy. A negative result provides much weaker evidence of compliance.

Robustness check: an attacker and a transformation

“Is the watermark robust?” sounds like a single engineering question. In practice, several threat models sit inside it.

The easiest setting is benign persistence. A user copies text, fixes punctuation, or changes a few sentences. Anthropic says light editing probably will not remove the watermark completely, while a full rewrite will. The SynthID paper likewise reports that edits and paraphrasing weaken generative watermarks.

A more demanding setting involves scrubbing. An adversary deliberately transforms the text while preserving its useful meaning until the watermark detector stops recognizing it.

The opposite attack is spoofing: make text that did not come from the target model appear to carry its watermark.

There is also watermark stealing, where repeated access to a watermarked generator reveals enough about the effective watermarking behavior to improve subsequent spoofing or scrubbing.

These are established research concerns rather than hypothetical terminology. The SynthID paper explicitly identifies stealing, spoofing, scrubbing, and LLM paraphrasing among the limitations of generative watermarks.

Jovanovic, Staab, and Vechev studied watermark stealing against several then-state-of-the-art watermarking schemes. Their attack used black-box queries to approximately reverse-engineer watermark behavior and then applied that information to spoofing and scrubbing. Across the schemes they studied, they report average attack success above 80% for attacks costing less than $50. Their experiments were not a demonstrated attack against Anthropic’s currently undisclosed Claude implementation, so the appropriate lesson concerns the attack surface of generative watermarking rather than a claim that Claude’s watermark has already been broken.

A second ICML 2024 paper, Watermarks in the Sand, studies an even stronger security goal: whether a computationally bounded attacker can be prevented from erasing a watermark without substantially degrading output quality. Under explicit assumptions that the attacker has access to a useful quality oracle and a perturbation process capable of moving among high-quality outputs, the authors prove that this form of strong watermarking is impossible. They also demonstrate the attack against three existing watermark schemes.

The assumptions are important. The theorem should not be summarized as “all watermarks are useless.” It establishes a boundary on a particularly strong adversarial guarantee under a defined attacker model.

Natural language creates a structural difficulty for that guarantee. Many prompts admit a large set of semantically adequate responses:

If an attacker can move efficiently through $\mathcal{Q}(p)$ while repeatedly checking quality, the attacker can search for another high-quality expression whose statistical relationship with the original watermark has weakened.

As rewriting models improve, exploring such equivalent formulations becomes cheaper.

That does not prevent watermarking from being useful. It changes what kind of security claim we should attach to it.

Operational key security matters too

The published discussion usually focuses on the statistical construction, but real deployment adds a familiar security problem: secret management.

A detector’s attribution value presumes that the relevant watermark behavior remains controlled by the intended provider. Operationally, that suggests requirements such as

If an attacker obtained a key, or learned enough effective watermark behavior to synthesize convincing positives, the meaning of future detections under that key would change.

I have not seen Anthropic publicly document its key-management architecture for this system, and absence of public documentation gives no basis for assuming it lacks one. The point is about the type of infrastructure required once detector outputs begin supporting real decisions.

A watermark detector is eventually part of a security system, rather than an isolated sampling algorithm. That system includes keys, detector versions, calibration datasets, access controls, monitoring, incident response, and policies for interpreting historical detections after a compromise.

Statistical watermarks and cryptographic provenance provide different evidence

Anthropic’s announcement gives us a useful comparison inside Claude itself.

For text, Anthropic plans to use statistical watermarking. For supported generated files such as PNG, JPG, and SVG, Claude will attach C2PA Content Credentials: cryptographically signed provenance information stored with the file.

C2PA organizes assertions about an asset into a claim, digitally signs that claim, and binds the resulting information into a C2PA Manifest. The specification describes these manifests as provenance data and bases trust decisions on the signer associated with the cryptographic signing key.

The two mechanisms answer related but structurally different questions.

A statistical text watermark asks approximately:

A signed provenance record asks approximately:

That difference leads to different failure modes.

C2PA itself is careful about this distinction. Its specification describes Content Credentials as cryptographically verifiable information whose trustworthiness must still be assessed under a defined trust model.

NIST likewise treats watermarking, provenance tracking, content authentication, and synthetic-content detection as related components of digital-content transparency rather than one universal solution.

That layered view seems more realistic to me. Statistical watermarking has useful persistence properties that ordinary metadata lacks. Cryptographic provenance can carry explicit claims and a chain of transformations that a statistical watermark cannot reconstruct. Human disclosure can cover forms of AI assistance that leave almost no machine-detectable trace. Platform-side records may provide additional evidence when stronger attribution is needed.

The engineering objective then becomes broader than detecting synthetic text. We want relevant provenance claims to be representable, testable, and interpreted according to the strength of evidence that supports them.

Regulation has reached the point where these distinctions matter

Anthropic says the immediate reason for introducing Claude’s text watermark is compliance with the EU AI Act and the associated Code of Practice on Transparency of AI-Generated Content.

Article 50(2) has applied since August 2, 2026. It requires providers of systems generating synthetic audio, images, video, or text to mark their outputs in a machine-readable format and make them detectable as artificially generated or manipulated. The regulation says technical solutions should be “effective, interoperable, robust and reliable” as far as technically feasible while accounting for content-specific limitations, implementation costs, and the state of the art. It also contains an exception where an AI system performs standard editing or does not substantially alter the input or its semantics.

The European Commission’s final transparency Code of Practice is intended to help providers operationalize these obligations. By the end of July 2026, around 190 companies and organizations had signed it.

The technical analysis above makes the regulatory adjectives surprisingly demanding.

“Reliable” requires a definition of false positives, false negatives, calibration, and abstention.

“Robust” requires specifying whether we mean copy/paste, ordinary editing, paraphrasing, deliberate scrubbing, spoofing, or key compromise.

“Interoperable” requires deciding whether providers should share technical mechanisms or simply report evidence in a common semantic form.

“Effective” depends on the underlying policy objective. Vendor attribution, universal AI detection, academic authorship enforcement, and misinformation provenance are different objectives.

A regulation can require a mark. The harder institutional work begins when somebody decides what a detected mark permits them to conclude.

Frontier labs should standardize the meaning of detection results

I do not think every frontier lab needs the same watermark algorithm. Diversity of implementations may even be desirable from a security perspective.

The more urgent interoperability problem concerns detector semantics.

Imagine that four providers eventually expose these outputs:

“97% AI-generated”

“High confidence”

“Watermark score: 8.31”

“Claude involvement likely”

Without definitions, those strings cannot be meaningfully compared.

A useful detection interface could expose enough information to answer questions such as:

• Which watermark and detector version produced this result?
• What statistical hypothesis is being tested?
• What false-positive calibration applies?
• Is there enough text and token-choice entropy for meaningful detection?
• Under which transformations was the detector validated?
• Did the system abstain because the evidence is insufficient?

The eventual interface does not have to expose every internal score to ordinary users. Its underlying semantics should still be documented precisely.

Performance reporting should also move away from a single accuracy number. A realistic benchmark describes detection as a function:

where $L$ represents text length, $D$ the domain or generation setting, and $T$ the transformation applied after generation.

Useful $T$ values include light editing, heavy editing, paraphrasing, translation, rewriting by another model, and adversarial scrubbing.

The SynthID paper already demonstrates why this matters: detectability changes with text length and entropy, and its authors explicitly evaluate edits and paraphrasing. Anthropic similarly warns that factual passages, code, and proofreading contain less watermark capacity.

A detector should be allowed to say that the evidence is too weak.

Policymakers should regulate interpretation as carefully as marking

Once watermark detection reaches schools, publishers, platforms, employers, and courts, I would worry at least as much about overinterpretation as under-detection.

One inference is particularly unsafe:

Several paths lead to a negative result:

another model generated the text; an older or unwatermarked model generated it; Claude generated it before extensive rewriting; Claude only proofread it; the sample is short; the passage contains too few discretionary token choices; an adversary deliberately scrubbed the signal.

Anthropic itself acknowledges most of these technical boundaries.

Positive detections also deserve proportional treatment. Even low false-positive rates can become consequential when detectors are applied at enormous scale. The appropriate evidentiary standard will depend on the cost of a mistaken decision.

For high-stakes disciplinary or legal uses, a watermark result makes more sense as one source of evidence that can trigger or support investigation than as an automatic adjudicator.

Policymakers and standards bodies can help by requiring evaluation under realistic conditions. A provider claiming robustness could report detection performance by text length, language, domain, transformation, and adversarial setting at explicitly defined false-positive rates.

Interoperability can likewise focus on common semantics instead of a common secret. Providers could agree on how to represent:

• detector version;
• hypothesis tested;
• calibration;
• evidence sufficiency;
• abstention;
• known threat-model boundaries.

That would allow institutions to reason consistently about outputs from different providers without requiring every provider to embed the same statistical pattern.

Most importantly, rules about transparency should preserve the distinction between evidence of model involvement and judgments about authorship. Those judgments often depend on context that no token-level watermark can encode.

The watermark is most useful when its claim stays narrow

Return to the hypothetical detector result:

Strong evidence of Claude’s watermark is present in this passage.

After reading the mechanism, I find that statement more technically impressive than I initially expected.

SynthID-Text uses the stochastic freedom already present in language-model sampling to create keyed statistical correlations. Tournament sampling can satisfy carefully defined non-distortion properties, and DeepMind’s empirical evidence suggests that its production configuration preserved perceived response quality at impressive scale. The mechanism also adds very little computational overhead.

The same result becomes less informative when translated into stronger claims.

“Claude authored the ideas in this document” requires a theory of authorship.

“No AI was used” requires coverage far beyond one provider’s watermark.

“This document has not been manipulated” requires integrity evidence.

“This person violated a policy” requires information about the policy and the person’s behavior.

“This watermark cannot be forged or removed by an adversary” requires a security guarantee that current research gives us good reason to examine carefully.

The useful object in the middle is statistical provenance evidence: evidence that a particular keyed generator probably participated in the production of a token sequence, with confidence depending on the amount of available signal and assumptions about how the text reached us.

That is already valuable.

The next step for frontier labs is to make the uncertainty, calibration, applicability, and security assumptions surrounding that evidence legible. The corresponding task for policymakers is to preserve those boundaries when detector outputs become inputs to institutional decisions.

A watermark does not need to answer every question about AI-generated text to be worth deploying. Its long-term value will depend on whether we continue asking it the question it was actually designed to answer.

References

• Anthropic, How Claude’s text watermark works, August 14, 2026.
• Dathathri et al., Scalable watermarking for identifying large language model outputs, Nature 634, 818-823 (2024).
• Jovanovic, Staab, and Vechev, Watermark Stealing in Large Language Models, ICML 2024.
• Zhang et al., Watermarks in the Sand: Impossibility of Strong Watermarking for Language Models, ICML 2024.
• Regulation (EU) 2024/1689, Article 50.
• European Commission, Code of Practice on Transparency of AI-generated Content.
• NIST AI 100-4, Reducing Risks Posed by Synthetic Content: An Overview of Technical Approaches to Digital Content Transparency.
• C2PA, Content Credentials Technical Specification.