Casual test shows readers cannot distinguish watermarked AI text from unwatermarked

A static-site quiz tested whether readers could distinguish watermarked AI text from unwatermarked responses to identical prompts. The test used SynthID-Text to watermark one of three generated answers per question, with participants scoring no better than random chance. This suggests watermarking does not introduce detectable artifacts that degrade text quality or readability for casual readers.

The initial test had a flaw: watermarked responses were disproportionately placed as the first option, skewing results toward a 6/10 score for users who selected the first answer by default. After reshuffling, scores clustered around the expected 3.33/10 mean for random guessing, reinforcing that participants could not perceive watermarks. The test was small-scale and informal, but the consistency of results across two rounds adds weight to the finding.

Watermarking schemes like SynthID-Text work by subtly biasing token selection within the model’s existing probability distribution, rather than introducing overt distortions. This approach avoids degrading output quality but also means watermarks are imperceptible to humans. If the goal is to flag AI-generated text for users, this method may fail unless paired with automated detection tools, which could introduce new challenges for false positives or circumvention.

The test’s setup was minimal: a rented H200 GPU generated responses for ~$2, and analytics tracked scores via page visits. While not rigorous, the results align with claims in watermarking research that the technique preserves output quality. However, the lack of human detectability raises questions about its effectiveness for transparency, especially if watermarks are intended to signal AI origin to end users rather than machines.