Dataset
165 models · 4 languages[1]
Reported by the paper for English, Chinese, Russian, and Arabic testing. This is not a ChinaAPI dataset.
Paper · data record
Research preview
Private beta proposed
One token is enough to audit a model route.
A published black-box method shows that repeated one-token outputs can form a measurable behavioral fingerprint. ChinaAPI is prototyping a reproducible consistency audit for API routes.
Audit cost
~100 one-token queries[1]
The paper reports this approximate request count for an 8-cell protocol. Actual API cost and evidence strength depend on the endpoint and configuration.
PaperVerification result
<11% EER with 8 probe cells[1]
A result reported under the paper's experimental protocol. It is not a guaranteed error rate for arbitrary routes, prompts, or providers.
PaperWhy one response proves nothing
A model ID is metadata. A single answer is a noisy observation.
Names are assertions
A model identifier tells you what a route claims to serve. It is not independent evidence of the route's behavior.
Sampling changes outputs
Random sampling, service updates, system prompts, and routing policy can all change one completion.
Latency is incomplete
Response time can expose infrastructure conditions, but it cannot by itself identify a model.
Distributions carry the signal
The usable evidence comes from the shape of repeated normalized answers, not from any single answer.
What the paper actually did
Build a controlled behavioral fingerprint, then compare distributions.
The researchers used constrained prompts designed to yield one-word answers, repeated the sampling, normalized the returned answer categories, and compared the resulting probability-like fingerprints.
Constrained prompts
Ask questions with a small, controlled answer space so repeated outputs can be counted consistently.
Repeated sampling
Collect many independent one-token outputs under fixed endpoint and sampling settings.
Answer normalization
Map equivalent surface forms into stable answer categories before comparison.
Distribution comparison
Measure whether two answer distributions are closer than expected for different models.
Behavioral fingerprint
Repeated answers create a comparable distribution shape.
Illustrative schematic, not ChinaAPI production data
Three model profiles
Model A
Model B
Model C
Same model, split samples
Independent splits keep a similar shape.
Different models
Peaks and category weights diverge.
Text alternative: the left panel shows three different illustrative bar distributions. The middle overlays two independently sampled distributions from the same illustrative model and shows similar bar heights. The right overlays two different illustrative models and shows distinct peaks and category weights. No real model measurements are shown.
Key research results
Same-model fingerprints remained substantially closer.
The paper found that same-model fingerprints remain substantially closer than fingerprints from different models.
When samples from the same model were split in half, their fingerprints were about an order of magnitude closer than samples from different models. This comparison is reported by the paper under its probe construction and distance metrics.[1]
The paper also reports an ecosystem anomaly in which one endpoint carrying a proprietary flagship label was distributionally indistinguishable from open-source Qwen models. This is the paper authors' reported observation, not a ChinaAPI test or attribution claim.[1]
| Paper result | Reported value |
|---|---|
Model-family classification[1]Leave-one-out accuracy reported by the paper. Paper | 59.5% |
Chance baseline[1]Random baseline reported alongside the classification result. Paper | 18.4% |
Full 40-cell verification EER[1]Equal error rate under the paper's complete 40-cell protocol. Paper | 7.3% |
8-cell verification EER[1]Reported with roughly 100 one-token requests; not a universal route guarantee. Paper | <11% |
- Model-family leave-one-out classification: 59.5 percent.
- Chance baseline: 18.4 percent.
- Full 40-cell verification equal error rate: 7.3 percent.
- 8-cell verification equal error rate: below 11 percent.
All values are reported by One Token Is Enough; the associated research record is available on Zenodo.
Proposed private beta
From a research protocol to route observability.
The private beta will expose the evidence needed to reproduce and challenge a comparison. The report is designed to retain method context, not collapse the result into an unexplained score.
Protocol identity
Request parameters and the exact probe-set version.
Sampling window
Test timestamps, route, configuration, and sample count.
Answer record
Raw responses alongside their normalized categories.
Returned identity
The requested model name and any model identifier returned by the API.
Distance measure
Jensen-Shannon divergence from a versioned reference fingerprint.
Uncertainty
Confidence interval, anomaly type, and change from prior windows.
How to interpret a report
Three evidence states, none of them an accusation.
Consistent with reference
Observed differences remain within the expected sampling range for this configuration.
Insufficient evidence
The current sample or task panel cannot support a reliable comparison.
Mismatch signal detected
Multiple independent probe cells differ from the reference beyond the stated threshold. Further investigation is warranted.
Mismatch signal is not attribution. A behavioral difference can identify a need to investigate. It does not, by itself, prove what changed or who caused it.
Method boundaries
A route audit describes a time, configuration, and behavior—not an immutable model identity.
- Models may be updated without advance notice, which can move a legitimate reference fingerprint.
- The same model can produce different distributions under different system prompts or sampling parameters.
- A public probe set may be recognized or handled specially by a server.
- An audit can evaluate only the route, time window, and configuration that were actually tested.
- Audit output should not be used to publicly accuse a third party without independent evidence and investigation.
Beta data boundary
No third-party API key in this form.
The first beta is intended to evaluate requests made under a participant's own ChinaAPI account. A later Dashboard workflow may offer short-lived, restricted test credentials. This page will not ask for a long-lived key from another provider.
Interest-form details are used to evaluate and contact beta candidates. If no beta relationship begins, the inquiry is scheduled for deletion within 90 days; participants may request earlier deletion at [email protected]. If a participant joins, the beta agreement shown before testing will state the audit-record retention period and deletion workflow. See the Privacy Policy.
Private beta interest
Help shape a reproducible route-consistency standard.
Tell us which routes and workloads matter. Do not include an API key, prompt contents, customer data, or other secrets.
Research preview
Model names are metadata. Behavior is evidence.
Join the ChinaAPI private beta to help shape a reproducible standard for model-route consistency testing.