AI Detection Tools Compared: What Makes a Good Detector?

High false positive rate on formal writing

Academic papers, legal documents, technical documentation, and formal business writing all have low perplexity because they follow strict conventions. Single-signal detectors systematically flag these as AI-generated. Studies found false positive rates exceeding 50% for ESL academic writing.

Defeated by paraphrasing tools

Perplexity-based detection can be substantially reduced by running AI output through a paraphrasing tool. Some commercial humanizer tools can drop a text from 90% AI score to below 30% on single-signal detectors, while the content is still entirely AI-generated.

Model-specific training gap

A detector trained primarily on GPT-3 output performs differently on Claude, Gemini, or Llama output. The perplexity signature varies by model family. A new model version can partially evade a detector trained before its release.

Length sensitivity

Statistical detection accuracy drops sharply below 200-300 words because there is insufficient text to establish reliable average perplexity. Single-signal detectors are particularly unreliable on short content.

No single signal is reliable across all content types and all AI models. An ensemble that combines statistical patterns, semantic deep learning, phrase pattern matching, and frequency analysis requires an evasion technique to defeat all signals simultaneously. Evading one often makes another more detectable.

Fine-tuned transformer models (like DeBERTa v3) learn detection features that are not easily described as surface patterns. They resist evasion techniques that work on purely statistical methods. A detector without a semantic component has a significant gap in its coverage.

A single number hides whether the score is driven by one weak signal or by consensus across multiple independent signals. Per-signal transparency lets users assess confidence and identify likely false positives. A 75% score with only the statistical detector elevated is less reliable than a 75% score with four detectors elevated.

Detection accuracy against a specific model degrades if the detector was not trained on samples from that model version. Good detectors regularly incorporate synthetic training data from newly released generators.

A detector that reports 94% confidence on text it cannot reliably assess is worse than one that returns 55% with a note that the signal is weak. Overconfident tools lead to overconfident decisions. Good tools report uncertainty honestly.

Text and image detection are distinct disciplines using different signals. A tool that covers both with dedicated models for each is more useful than separate tools for each medium.