A vendor publishes a benchmark showing its own tool beating three competitors by a wide margin. The number is real, the methodology is published, and the company that built the winning tool also designed the test, scored the results, and wrote up the findings. None of that makes the benchmark worthless. It does mean reading it well takes a bit more than glancing at the headline percentage.
This pattern shows up constantly in the AI writing tools space, and learning to read one example carefully makes every future benchmark claim easier to evaluate on its own merits.
What Phrasly’s Ultra Benchmark Actually Measured
In September 2026, Phrasly published a benchmark comparing its new Ultra model against three other AI humanizers, StealthGPT, WriteHuman, and Undetectable AI, using 72 long, AI-generated English texts sampled from a public dataset on Hugging Face. Every tool rewrote the same texts, and every output was scored using Pangram’s detector, with results also reported for GPTZero, ZeroGPT, Turnitin, and Originality.ai.
Phrasly Ultra averaged a 99.8 percent human score on Pangram across the 72 outputs, with 70 of 72 rated entirely human, outperforming the three other tools, which averaged 50 percent or less on the same measure. Phrasly also published a hallucination count for each tool, finding Ultra introduced fewer factual changes to the original meaning than its competitors, averaging 0.85 per rewrite against notably higher rates elsewhere.
The hallucination metric is worth noting specifically because it measures something a pass rate alone cannot: whether a rewrite actually preserved what the original text said, rather than just changing its statistical texture enough to score differently on a detector.
Why the Source of a Benchmark Changes How to Read It
Phrasly ran this test, scored it, and published the results, including the model it builds and sells. That is not evidence the numbers are false, but it is a reason to read them the same way any specific claim from an interested party deserves to be read, as a starting point for verification rather than a settled conclusion on its own.
To its credit, Phrasly published the underlying data openly: the specific 72 texts used, per-output Pangram scores, hallucination counts, and a cryptographic fingerprint of each output, specifically so outside parties could attempt to reproduce the results rather than simply trusting the summary. That level of transparency is meaningfully more verifiable than a benchmark that reports only a final number with no supporting data released alongside it.
The fingerprint detail matters more than it might first appear. Publishing a cryptographic hash of each output lets an outside party confirm the exact text that was actually scored, without Phrasly needing to hand over the raw files directly, a middle ground between full transparency and no verification at all.
Questions Worth Asking Before Trusting Any Benchmark Like This
A benchmark this specific invites a specific kind of scrutiny, and the answers matter more than the final percentage.
Before treating a vendor-run benchmark as a reliable guide, it helps to check:
- Whether the underlying data and methodology were published in enough detail for an outside party to actually reproduce the test
- Whether the test used a single detector or multiple independent ones, since a result that holds across several tools is a stronger signal than one tied to a single scoring system
- Whether competing tools were given a fair, comparable setup, the same texts, the same scoring method, the same evaluation window
- Whether the benchmark measures what actually matters for a given use case, rather than only the metric most favorable to the tool being sold
What a High Pass Rate Does Not Establish
A 99.8 percent average human score on one detector, however well documented, describes how a specific set of outputs scored against a specific scoring system at a specific point in time. Detectors update their own models regularly, often specifically in response to new rewriting techniques, which means a benchmark result is a snapshot, not a permanent guarantee that will hold indefinitely as detection technology keeps changing on the other side.
It also says nothing about how the same rewriting approach performs on types of text the benchmark did not include, shorter passages, a different language, a more technical register, since 72 long English texts from one dataset is a specific, bounded sample rather than a comprehensive test of every writing context a result might get applied to.
This is worth keeping in mind when evaluating a tool like Phrasly Ultra or any competitor making a similar claim: a published, reproducible benchmark is meaningfully more trustworthy than an unverified marketing claim, but it still describes a moment in an ongoing, fast-moving contest, not a fixed and final result.
Transparency Is a Reason for Cautious Trust, Not Blind Trust
The most useful response to a vendor-published benchmark like this one is neither dismissing it outright because of who ran it, nor accepting the headline number uncritically because the methodology looks rigorous. It is checking whether the data was actually published in a form that allows independent verification, and treating that openness, rather than the final percentage itself, as the real measure of how much the benchmark deserves to be trusted.
That standard applies equally to every vendor making a similar claim, not only the one being discussed here. A competitor publishing an equally transparent, reproducible benchmark deserves the same cautious trust, and one publishing only a headline number with no underlying data deserves considerably less.
For more on how AI detection benchmarks are conducted and what their limitations tend to be, further reading on the Phrasly blog covers the underlying research for anyone evaluating a vendor’s published claims.
