What we have measured, and what we have not
ThermalVariations has not yet completed the blind, multi-site benchmark described at the bottom of this page. This page states what has been measured, on what, and what has not. It will be updated, not replaced, as results land.
Why we publish this
No aerial-thermography vendor we could find publishes quantified precision and recall. A research pass in August 2026 found the major platforms publish IEC-adherence statements and superlatives only. We would rather show a small, honest number than a large, vague claim - and we would rather you knew our limits before you paid than after.
What we have measured
One site, public CC0 data: 120 DJI Zenmuse XT2 radiometric frames over a single 6 MWp plant in Italy (Kaggle “Photovoltaic System Thermography”, dataset 1), of which 23 frames contain labelled faults and 97 are clean, with 26 ground-truth boxes. Labels are binary (defective or healthy) and class-agnostic. Production configuration (local-peak trigger 3.0 °C).
| Metric | Value | 95% CI (Wilson) | Count |
|---|---|---|---|
| Precision (frame level) | 1.00 | 0.65 to 1.00 | 7 / 7 flagged frames correct |
| Recall (frame level) | 0.30 | 0.16 to 0.51 | 7 / 23 fault frames found |
| False positives on clean radiometric frames | 0 | upper bound 0.04 | 0 / 97 clean radiometric frames |
| Box precision, IoU 0.30 | 1.00 | 0.65 to 1.00 | 7 / 7 boxes |
| Box recall, IoU 0.30 | 0.27 | 0.14 to 0.46 | 7 / 26 labelled boxes |
Source: training/eval/results/sweep-d1-lp3-ci.json in the ThermalVariations repository, produced by the eval harness on 2026-07-03. Box-level precision and recall at IoU 0.50 are identical to IoU 0.30 on this set. Pre-tuning baseline recall on the same set was 0.17 (4 / 23).
What that means, plainly
When we flag a module we have not yet been wrong on this set: zero false alarms across 97 clean radiometric frames. But we currently find about 3 in 10 of the labelled anomalies here. The misses are subtle, low-ΔT defects. For context, peer-reviewed work on 4.3 GW and 107,842 modules (Bommes et al. 2021, Progress in Photovoltaics 29:1236-1251) reports 84 to 96% recall on offline modules and diode faults but 13 to 31% on single-cell hotspots, with cross-plant performance swinging widely under domain shift. Sub-headline hotspot recall is the literature norm, not a quirk of our model - which is exactly why we publish the number instead of hiding it.
The practical consequence: treat a ThermalVariations report as a high-precision short list, not an exhaustive census. A module we flag deserves a site visit. A module we do not flag is not confirmed healthy.
What we have not measured
- Fault-type classification accuracy. This dataset's labels are binary (defective or healthy), so we cannot say how often the class we assign is the right one.
- Multi-site generalisation. One site, one campaign, one camera. Cross-plant performance under different modules, mounting and climate is unmeasured.
- ΔT error against a handheld reference thermometer.
- Location accuracy in metres, and duplicate rate on real overlapping flights.
- Turnaround time at utility scale.
- Performance on non-radiometric imagery, on black-hot palettes, or on DJI R-JPEGs (see supported cameras below).
The bars we publish against
From our validation protocol. A class is commercially ready only when it clears its bar on a blind, multi-site set.
| Metric | Bar |
|---|---|
| Recall, offline modules / strings | at least 0.95 |
| Recall, diode / substring faults | at least 0.90 |
| Recall, multi-cell hotspots | at least 0.80 |
| Recall, single-cell hotspots | report as measured (literature norm 0.13 to 0.31) |
| Precision, High / Critical findings | at least 0.90 |
| Unexplained High / Critical misses | 0 (each miss gets a written cause) |
| Duplicate rate (after de-duplication) | at most 0.05 |
| ΔT mean absolute error vs reference | at most 2 K |
What a class that misses its bar does
It ships in your report marked “suspected, under validation” - never silently. Findings without a measured ΔT are separately labelled “suspected (pattern-based)”. We do not present a guess as a measurement.
Current blocker
The blind benchmark needs at least three unseen sites, roughly 10,000 modules and 300 expert-labelled anomalies, labelled without sight of our output. We are seeking partner sites. If you operate a site with confirmed faults and are willing to share flight imagery and the confirmed fault list under NDA, request a trial inspection: the analysis is at the published rate, human-reviewed, and not payable if nothing verifies on site.
Supported cameras
| Camera / format | Today | Notes |
|---|---|---|
| FLIR Vue Pro R / Zenmuse XT2 (FLIR radiometric JPEG) | Radiometric today | Measured °C and ΔT per finding. The validation figures above were produced on XT2 frames. |
| DJI Mavic 3T, Matrice 4T, Matrice 30T, Zenmuse H20T / H20N (DJI R-JPEG) | Screening-level today | Radiometric decode requires the DJI Thermal SDK, which is not yet licensed and deployed on our worker; once it is, these cameras move to measured °C and ΔT. Until then, DJI files are analysed from the rendered palette: findings are pattern-based and labelled 'suspected', without measured ΔT. The intake check tells you this before you pay. |
| 16-bit radiometric TIFF (any vendor) | Best effort | Decoded where the file carries a linear temperature scale; otherwise treated as non-radiometric. |
| Black-hot palette exports | Unsupported | Inverted palettes defeat the thermal-outlier gate. Re-export as ironbow or white-hot, or upload the radiometric original. |
The full capture specification, including altitude limits per camera, is on the capture-spec page. Thermal analysis is indicative, not diagnostic: verify flagged modules on site before repair work.