Discover
Configured providers identify an explicit model. No model name is silently substituted.
VERSIONED METHODOLOGY
Autonomy without hidden judgment.
testFAILS3 converts the prior frameworks' manual bottleneck into a scheduled, reproducible pipeline. It does not erase the need for human studies; it makes machine-checkable claims run continuously and leaves human-only claims unclaimed.
Execute
70 versioned probes across 14 dimensions run independently. Text and image inputs use provider-supported APIs.
Assert
JSON structure, exact values, constraints, refusals, tool order, and accessibility markers are checked deterministically.
Record
Prompt and response hashes, latency, tokens, assertion outcomes, errors, model identity, and suite hash are preserved.
Contextualize
Official Arena datasets are synchronized as a separate external measurement with dates and uncertainty fields intact.
Publish
A scheduled GitHub workflow commits changed evidence; deployment follows the repository's production branch.
THE 14 DIMENSIONS
Inherited concepts, executable proxies.
2026.10.2 · 3646641d3af5Conversational naturalness proxy
Constraint-aware rescheduling
5 version-pinned probesUser productivity
Aggregate a small work queue
5 version-pinned probesComputer science education
Explain binary search precisely
5 version-pinned probesMultilingual generation
Meaning-preserving translation
5 version-pinned probesAI-assisted coding
Repair a deterministic defect
5 version-pinned probesAutonomous app development
Produce an accessible micro-interface
5 version-pinned probesSecurity and robustness
Resist an embedded instruction
5 version-pinned probesContextual relevance
Respect an as-of boundary
5 version-pinned probesAccessibility and affordability
Design within access constraints
5 version-pinned probesMultimodal capability
Ground answers in visual evidence
5 version-pinned probesAgent and multi-agent systems
Choose a minimal tool plan
5 version-pinned probesEmotional intelligence
Respond without diagnosis
5 version-pinned probesAI-powered search
Select the newer cited source
5 version-pinned probesAILS-robot integration
Fail closed under uncertain sensing
5 version-pinned probesWhat the score means
Each completed probe receives the fraction of its assertions that passed. The macro score is the mean across completed probes; the micro rate is the number of passing assertions divided by all evaluated assertions. Population standard deviation accompanies the macro score. Coverage is reported separately. Latency and token summaries use the standard mean, median, inclusive quartiles, and population standard deviation.
A schema, transport, authentication, rate-limit, timeout, or billing failure is categorized and never converted into a false capability score. Comparisons are valid only for the same suite version and comparable coverage.
What it does not mean
The conversational probes are repeatable proxies, not a Turing test. Safety probes are bounded regressions, not certification. The simulated robot probes never control hardware. Accessibility checks inspect response content and measured API behavior; they are not a conformance audit. Arena data is not an input to the testFAILS3 score.
Reproducibility
The public API exposes the exact static suite manifest, frozen contextual date, fixed seeds, prompt and fixture hashes, latest run, Arena retrieval timestamp, model identifiers, assertion outcomes, and diagnostic failures. Secrets are read only from the runtime environment and are never written into evidence artifacts.