§01The surrogate model credibility gap
Physics-ML vendors claim CFD-level accuracy. Engineering buyers need proof before design decisions or procurement. Comparotor provides sealed, replayable, contamination-resistant benchmark reports that make surrogate model performance comparable across vendors — on the same dataset, using the same SU2 RANS reference.
Unverifiable claims
Every vendor claims state-of-the-art accuracy. Without a neutral benchmark, buyers can't compare across vendors or validate independently.
Contaminated benchmarks
Models overfit to published test sets. Comparotor rotates 50 of 200 test airfoils quarterly — results can't be gamed quarter-over-quarter.
No procurement evidence
Procurement requires defensible evidence. Benchmark reports are signed, reproducible, and audit-ready.
§02The Comparotor benchmark
RotorBench-Aero v0.1 — a standardised benchmark for 2D aerodynamic coefficient prediction
Public leaderboard
Compare surrogate models on Cl, Cd, Cm prediction accuracy, L/D rank correlation, OOD generalisation, and inference latency.
Private evaluation API
Submit via ONNX or Docker. Receive signed JSON results and PDF reports via webhook — without exposing your model weights publicly.
Contamination resistance
200-airfoil test set. 150 stable for quarter-to-quarter comparability. 50 rotated quarterly to prevent benchmark overfitting.
Signed PDF reports
Procurement-grade reports with dataset version, scoring breakdown, OOD results, latency distribution, and artefact checksums.
§03Hermetic evaluation pipeline
- 01
Submit
Upload ONNX + wrapper or Docker image via API or web.
- 02
Archive
Artefact sealed with SHA-256 checksum; test set decrypted in memory only.
- 03
Evaluate
Hermetic runner: no outbound network, ephemeral container, SU2 RANS oracle comparison.
- 04
Score
Composite score computed across Cl, Cd, Cm, L/D rank correlation, OOD subset, and latency.
- 05
Report
Signed PDF + webhook delivery. Replay-ready audit trail stored 24 months.
§04Built for three audiences
Physics-ML vendors
Prove your surrogate model outperforms competitors on a neutral benchmark. Get a third-party PDF report you can use in sales, marketing, and procurement responses.
OEM & Tier-1 procurement
Compare vendor surrogate models privately before procurement. Receive signed reports with standardised scoring that hold up to engineering governance review.
University labs & researchers
Submit public baselines. Publish performance against a stable, versioned rotating-machinery benchmark with a citable DOI-ready format.
§05Public leaderboard
full board| # | Model | Composite | MAE Cl | MAE Cd | ρ L/D |
|---|---|---|---|---|---|
| 1 | FourierNet-v0.1 baseline | 0.0389 | 0.0147 | 0.00038 | 0.9954 |
| 2 | MLP Neural Network baseline | 0.0512 | 0.0198 | 0.00051 | 0.9923 |
| 3 | Gaussian Process baseline | 0.0589 | 0.0219 | 0.00058 | 0.9901 |
Showing reference baselines. Real submissions appear here after evaluation.
View full leaderboard§06Built for procurement
UIUC training set
Models train on the UIUC airfoil database. Test airfoils are sealed and held out.
SU2 RANS labels
High-fidelity SU2 solver provides ground truth for Cl, Cd, Cm across the operating envelope.
Quarterly rotation
Test set rotates quarterly to prevent contamination and benchmark overfitting.
Artefact sealing
Every submission archived with SHA-256 checksum. Replayed on demand for dispute resolution.
24-month archive
Artefacts stored 24 months. EU R2 storage available on request.
Signed reports
PDF reports signed with HMAC-SHA256. Checksum in report footer.
§07Design partner programme
We are onboarding five founding design partners for RotorBench-Aero v0.1. Design partners shape the benchmark specification, scoring weights, and dataset scope — and receive founding-tier access before public launch.
Target profiles:
- Turbomachinery or aerodynamics research lab (academic)
- Physics-ML vendor seeking third-party validation
- A&D or energy OEM evaluating surrogate models for design workflows
- eVTOL or wind-energy simulation team
- Benchmark or certification organisation
§08About Comparotor
Comparotor is building the independent validation layer for engineering AI. We are a technical team focused on the specific problem of making surrogate model performance claims verifiable, reproducible, and procurement-ready.
The benchmark is designed with input from the rotating-machinery and physics-ML communities. Our scoring formula, dataset choices, and anti-contamination controls are documented in the open methodology specification.
§09Simple pricing
all plans1 public run / week
50 private runs · API access · PDF report
Unlimited · custom suites · SLA