STT B / BENCHMARK

CVPR 2025 modelIndependent evaluation

STT B
Reliability
Benchmark

A reproduction and reliability study of STT-B—testing whether fast, point-prompted segmentation holds up beyond a headline score.

Explore the findings ↓ Read the report ↗
Qualitative STT-B segmentation results across several Oxford-IIIT Pet images
Point promptMask prediction
499images
evaluated

One point.

One mask.

Seven dimensions of evidence.

The question

Fast segmentation is useful.
Reliable segmentation is better.

We evaluated the official STT-B checkpoint without retraining, then went further: prompt placement, image degradation, boundary quality, confidence calibration, failure cases, and a paired comparison against SAM ViT-B.

DatasetOxford-IIIT Pet
PromptDeepest interior point
HardwareNVIDIA Tesla T4
01

Primary results

Strong regions.
Imperfect edges.

Mean IoU 0.607
95% CI 0.584–0.628
Dice 0.715
95% CI 0.692–0.736
Precision 0.855
Higher than recall
Boundary F1 0.095
Contours remain difficult
↳

Precision exceeds recall, so STT-B more often misses part of the object than includes unrelated foreground. Region overlap alone hides the much weaker contour quality.

02

Paired comparison

STT-B vs.
SAM ViT-B

A clear quality–speed frontier.

SAM produced moderately better masks on the same 100 images. STT-B answered much faster—making the choice less about a winner and more about the workload.

5.7× faster end-to-end
than SAM ViT-B
MetricSTT-BSAM
Mean IoU0.6310.672
Recall0.7590.869
Latency77 ms440 ms
Throughput13.542.28 FPS
Bar chart comparing mean IoU of STT-B and SAM ViT-B
Overlap quality ↑ better
Bar chart comparing end-to-end runtime of STT-B and SAM ViT-B
End-to-end latency ↓ better
03

Stress tests

Small changes.
Material effects.

Mean IoU for four point-prompt strategies
Where you click matters. The centroid prompt led this 100-image ablation.
Mean IoU under original, blur, bright, dark, and low-resolution conditions
Brightness is not neutral. Brightening caused the largest drop in mean IoU.
04

Reliability

Confidence is
not correctness.

Scatter plot of STT predicted IoU against actual IoU
r = 0.518predicted vs. actual IoU

The model score is directionally useful, but the broad scatter makes it unsafe as a per-image guarantee.

66catastrophic failures
22high-confidence failures
Catastrophic failure: actual IoU < 0.25. High confidence: predicted IoU ≥ 0.70.
05

Method

A fixed, reproducible
evaluation protocol.

  1. 01
    Sample

    Shuffle the Oxford-IIIT Pet test split with seed 42 and request 500 valid masks.

  2. 02
    Prompt

    Place one positive point at the deepest interior foreground pixel.

  3. 03
    Infer

    Upscale the short side to 1,280 px, run STT-B, and map the prediction back.

  4. 04
    Measure

    Score regions, boundaries, latency, GPU memory, geometry, confidence, and failures.

Reproduce the work

Everything behind the numbers.

Notebook .ipynb ↗ Full report .pdf ↗ Presentation .pptx ↓ Source code GitHub ↗