vision

Overview

What this is

This site is a structured analysis of the computer-vision research record, assembled from a set of working notes covering two things that are usually kept apart:

• A systematic treatment of the field itself — the task taxonomy, the shared processing pipeline, the objective template that unifies classical and learned methods, the loss and regularizer vocabulary, and the practical mechanisms that make any of it work on real data.
• A chronological account of the key innovations from 2016 to 2026, era by era, with the original papers’ own figures and the mechanism behind each result.

The organising claim is that these two are the same material viewed from different angles. Every innovation in the timeline is a new answer to a question the taxonomy already poses: where does supervision come from, how is correspondence represented, what prior resolves the remaining ambiguity, what does the output interface look like.

The distinction that saves the most confusion A loss measures disagreement during training. A residual is the per-sample quantity the loss consumes. A regularizer encodes a prior. A constraint must hold exactly. A metric is the reported quantity, and it is rarely the loss that was optimised. Detection trains boxes with Smooth L1 and is evaluated with IoU; segmentation trains with pixelwise cross-entropy and is evaluated with mean IoU or Dice.

The five questions the decade answered

• 1 · How deep can a network be? — Answered 2016, then it stopped being interesting. Residual connections made depth free, and architecture ceased to be the binding constraint.

• 2 · Where does supervision come from? — 2019–2023. Contrastive → masked modelling → self-distillation → synthetic teachers → model-in-the-loop data engines.

• 3 · What is the output interface? — 2020–2025. Dense grid + NMS → set prediction → promptable, open-vocabulary output.

• 4 · How is geometry represented? — 2020–2025. Implicit neural fields → explicit primitives → feed-forward any-view prediction.

• 5 · How is the image distribution modelled? — 2018–2024. Adversarial → denoising diffusion → latent diffusion → flow matching and few-step sampling.

• Still open — Calibrated uncertainty, compositional and geometric reasoning, evaluation that survives distribution shift, and video as a model of dynamics, not of appearance alone.

The ten-year arc

Seven overlapping eras. The names describe what became possible in each window, not when the first paper appeared — the eras overlap heavily in time.

• 2016–2018 — Depth & detection
• 2018–2020 — Scale & self-supervision
• 2020–2022 — Transformer takeover
• 2022–2023 — Foundation & prompting
• 2023–2024 — 3D & multimodal
• 2024–2025 — Video, depth, concepts
• 2025–2026 — Any-view geometry

Six mechanisms that recur

Forty papers, six recurring mechanisms. Each is treated in full on the synthesis page.

• 1 · Make the easy path identity — A block that can do nothing at zero cost makes depth — and later, adaptation — safe.
  ResNet → ControlNet zero-convs → adaLN-Zero → LoRA

• 2 · Predict a set, not a grid — Move duplicate suppression out of inference and into the loss.
  DETR → Mask2Former → YOLOv10 → YOLO26 → RF-DETR

• 3 · Manufacture the supervision — Signal constructed by contrast, masking, distillation or synthesis.
  MoCo → MAE → DINOv2/v3 → Depth Anything

• 4 · Work in a latent space — Compress perceptually irrelevant detail away, then do the hard work cheaply.
  Latent Diffusion → DiT → Sora

• 5 · Heavy encoder, cheap head — Encode once, answer many queries.
  SAM → SAM 2/3 → frozen-backbone probing

• 6 · Return to explicit primitives — Where implicit representations are slow, go back to primitives.
  NeRF → 3D Gaussian Splatting

The output interface, in three steps

The clearest single progression in the decade. Each step removes a hand-designed component: the proposal generator, then the duplicate filter, then the fixed class list.

• Dense enumeration 2014–2019 — Anchors or grid cells, a score threshold, and non-maximum suppression. Three hand-tuned components, each with dataset-specific hyperparameters.

• Set prediction 2020–2024 — A fixed number of queries and a bipartite matching loss. Duplicate suppression is trained into the loss, so the NMS stage is removed.

• Promptable output 2023– — A point, box, mask or noun phrase selects what to segment or detect. The vocabulary is open and chosen at query time.

Where to go next

• Task taxonomy — Twelve task families with inputs and outputs, how they depend on each other, and why the field is better organised as inverse problems than as architectures.

• Pipeline & objectives — The ten-step pipeline shared by classical and learned systems, the common objective template, and the full data-term and regularizer vocabulary.

• Photometric consistency — The warp-and-compare principle in depth: the equations, the six assumptions it rests on, and the ecosystem of tricks that make it survive real data.

• The atlas — 29 modules — Every topic with its own page: calibration, restoration, primitives, correspondence, detection, segmentation, keypoints, flow, stereo, SLAM, tracking, video, point clouds, rendering, generation and vision–language.

• Losses & regularization — The complete objective vocabulary, organised by what each loss assumes — plus multi-task weighting, uncertainty, and the surrogate–metric gap.

• Cross-cutting practice — Robust estimation, multiscale processing, the seven consistency constraints, augmentation, imbalance, post-processing — and a twelve-row failure-mode checklist.

• Synthesis — The six mechanisms in full, the supervision ladder, the architectural outcomes, the limits of the benchmarks, and the open problems entering 2026.

• Paper index — Every paper on this site in one searchable, sortable table — filterable by era and by the mechanism it exemplifies, with arXiv links.

• Curriculum — A 29-module teaching sequence derived from the taxonomy, with the recommended production order and a consistent per-module template.

The central conclusion

The unifying idea is that computer-vision methods are combinations of representation + correspondence + constraint + robust optimisation. Deep learning changes the representation and often learns parts of matching or inference, but the classical constraints remain plainly visible in modern systems:

• Photometric warping supervises depth and neural rendering.
• Reprojection error drives calibration and bundle adjustment.
• Smoothness regularizes flow and geometry.
• Pyramids handle scale — as image pyramids, FPNs, U-Net skips, or hierarchical transformers.
• RANSAC and robust losses handle outliers.
• Cost volumes represent correspondence.
• Consistency checks handle ambiguity.

Treating these elements explicitly produces a more durable understanding than studying network names in isolation — which is the reason this site is organised the way it is.

Foundations

The master task taxonomy

This page organises computer vision as a family of inverse problems: infer scene properties from measurements produced by cameras or related sensors.

Purpose and scope

The properties being inferred are appearance, geometry, motion, identity, semantics, or physical state. The measurements come from one or more cameras or related sensors. That framing — measurement in, scene property out — is what makes otherwise unrelated tasks comparable.

A complete treatment of any task therefore needs five layers. Skipping any one of them produces the familiar failure of a method that works on a benchmark and nowhere else:

• 1 · Input and desired output — What is measured, and what must be estimated. Fixes the observability question before any method is chosen.

• 2 · Image-formation assumptions — What physical model links the scene to the measurement — projection, reflectance, exposure, sampling, noise.

• 3 · Processing pipeline — The sequence of representation, hypothesis generation, scoring and inference.

• 4 · Objective or loss — What is being minimised, on which residuals, under which priors — and how that relates to the metric actually reported.

• 5 · Practical mechanisms — How ambiguity, noise, occlusion and computational limits are handled. In practice this layer decides whether a method survives contact with real data.

Loss, residual, regularizer, constraint, metric — not interchangeable These five words are used loosely in the literature and the confusion is expensive. Precisely:

• Residual — the per-observation discrepancy r<sub>i</sub>(θ) between prediction and measurement.
• Loss — the aggregate scalar minimised during training, built by passing residuals through a penalty and summing.
• Regularizer — a term that encodes a prior and resolves what the data term leaves underdetermined.
• Constraint — a condition that must hold exactly, not be traded off against other terms.
• Metric — what is reported at evaluation, frequently non-differentiable and frequently not the training loss.

Classical vision minimises an explicit energy over scene variables; deep vision minimises a training objective over network parameters; post-processing may solve a separate discrete optimisation. Object detection trains box coordinates with Smooth L1 while being evaluated with IoU. Semantic segmentation trains with pixelwise cross-entropy while being evaluated with mean IoU or Dice. The gap between surrogate and metric is a design decision, and usually an unexamined one.

Twelve task families

The authoritative curriculum spans image formation; filtering and multiscale processing; feature detection and matching; segmentation; alignment; structure from motion; dense motion; stitching; computational photography; stereo; 3D reconstruction; recognition; video understanding; and vision–language systems. In 3D vision the corresponding core tasks are classification, segmentation, detection, tracking, reconstruction, registration, completion and 6-DoF pose estimation. Organised by what is estimated, they collapse into twelve families.

How the families depend on each other

These tasks are not independent, and the dependency structure explains why progress propagates the way it does — an improvement in a foundational family shows up months later in several downstream ones.

Upstream enablers

• Calibration supports stereo, pose, SfM and any metric measurement.
• Feature matching underlies registration, stitching, SfM, localization and loop closure.
• Detection feeds tracking, which feeds action and event analysis.
• Optical flow supports tracking and temporal aggregation.

Downstream consumers

• Depth and pose feed view synthesis and reconstruction.
• Segmentation supports measurement, reconstruction and editing.
• Reconstruction feeds simulation, rendering and embodied reasoning.
• Recognition feeds retrieval, grounding and scene reasoning.

Why this matters for reading the timeline ResNet’s effect was not confined to classification. Because recognition backbones sit upstream of detection, segmentation, depth and tracking, a better backbone improved all of them at once without any of those subfields changing their own methods. The same structural argument explains CLIP (a text-aligned encoder unlocked open-vocabulary detection and segmentation), DINOv2/v3 (frozen dense features unlocked depth and tracking probes), and SAM (a promptable mask model became a component in other people’s pipelines). Innovations in upstream families produce field-wide step changes; innovations in downstream families do not.

Reading the taxonomy against the timeline

Each era in the chronological account can be read as pressure applied at a particular point in this table:

• 2016–2018 worked mostly on localization and pixel understanding, with the backbone improvements in recognition spilling into both.
• 2018–2020 attacked the supervision requirement common to all families, and rebuilt the localization output interface with set prediction.
• 2020–2022 replaced the representation layer across every family at once, and moved rendering and generation from adversarial to diffusion training.
• 2022–2023 removed the closed class vocabulary from recognition, localization and pixel understanding simultaneously.
• 2023–2026 rebuilt geometry and 3D perception — first the representation (splatting), then the inference procedure (feed-forward any-view prediction).

Notice which family has been least disturbed: measurement and calibration. The physical model of image formation has not been learned away, and every geometric method on this site still depends on it being right.

Image formation and cameras

Every inverse problem in vision is inverting _this forward model. Getting it wrong produces bias, not noise — and bias does not average away with more data._

The pinhole camera model

A point \(\mathbf{X}_w = (X, Y, Z)^\top\) in the world is mapped to a pixel \(\mathbf{u} = (u, v)^\top\) by a rigid transform into the camera frame followed by a perspective division and an affine map into pixels.

Full projection, homogeneous form

• \(\mathbf{K}\) — intrinsics (3×3, upper triangular)
• \(\mathbf{R},\mathbf{t}\) — extrinsics: world→camera rotation and translation
• \(s\) — the homogeneous scale, equal to the camera-frame depth \(Z_c\)
• \(\mathbf{P}\) — the 3×4 camera matrix, 11 degrees of freedom

Intrinsic matrix

\(f_x, f_y\) are focal length in pixels (they differ only if pixels are non-square); \((c_x, c_y)\) is the principal point; \(\gamma\) is skew and is essentially always \(0\) on real sensors. Note \(f_x = f \cdot s_x\) where \(f\) is metres and \(s_x\) is pixels per metre — focal length in pixels is not a physical property of the lens alone, which is why it changes when the image is resized.

Consequences of resizing and cropping Scaling an image by \(\alpha\) scales \(f_x, f_y, c_x, c_y\) by \(\alpha\). Cropping shifts \(c_x, c_y\) by the crop offset, and leaves \(f\) unchanged. Neither is optional bookkeeping: an augmentation pipeline that resizes images without updating \(\mathbf{K}\) silently trains the network on an inconsistent camera. This is the most common geometry-aware augmentation bug.

Perspective projection and its consequences

Dropping to inhomogeneous coordinates, with \((X_c, Y_c, Z_c)\) the point in camera coordinates:

The division by \(Z_c\) is what makes vision hard. Three consequences follow directly:

• Depth is unobservable from one view. Any point on the ray \(\lambda\,\mathbf{K}^{-1}\tilde{\mathbf{u}}\), \(\lambda > 0\), produces the same pixel. Monocular depth is therefore a prior, not a measurement.
• Scale ambiguity. Scaling the whole scene and the translation by the same factor, \((\mathbf{X} \to k\mathbf{X},\ \mathbf{t} \to k\mathbf{t})\), leaves every image unchanged. Monocular SfM and SLAM recover geometry only up to one global scale.
• Depth resolution degrades quadratically. For a stereo baseline \(B\), \(Z = f B / d\) with disparity \(d\), so \(\;\partial Z/\partial d = -fB/d^2 = -Z^2/(fB)\). A fixed disparity error produces a depth error growing as \(Z^2\) — the reason everything is parameterised in inverse depth.

Weak perspective and when it is safe If the scene’s depth range \(\Delta Z\) is small relative to its distance \(\bar Z\), then \(1/Z \approx 1/\bar Z\) is constant and projection becomes affine: \(u \approx (f_x/\bar Z)X_c + c_x\). This is why distant planar scenes can be aligned with an affine or homography warp rather than a full 3D model — and why choosing the simplest valid transform matters.

Coordinate frames and rigid motion

The rigid transform lives in \(SE(3)\), with rotation in \(SO(3)\). The composition \(\mathbf{T}_{wc} = \mathbf{T}_{wb}\mathbf{T}_{bc}\) chains body-to-camera and world-to-body. Four parameterisations of rotation are in common use, and the choice matters for optimisation:

Exponential map (Rodrigues)

Optimisers update rotations _on the manifold: \(\mathbf{R} \leftarrow \mathbf{R}\exp([\delta\boldsymbol{\omega}]_\times)\) with a 3-vector increment, rather than perturbing 9 matrix entries and re-orthonormalising. This is what “Lie-algebra increments” means in a bundle-adjustment implementation, and it is why the Jacobians are 3-column rather than 9-column._

Lens distortion

Real lenses violate the pinhole model. The standard Brown–Conrady model applies a polynomial correction in normalised coordinates \((x, y) = (X_c/Z_c,\ Y_c/Z_c)\), before multiplication by \(\mathbf{K}\):

Radial and tangential distortion

with \(r^2 = x^2 + y^2\). Radial terms model barrel (\(k_1<0\)) and pincushion (\(k_1>0\)); tangential terms model a lens not parallel to the sensor. Because the correction scales with \(r\), distortion is only observable away from the image centre — the reason calibration targets must cover the periphery.

For wide-angle and fisheye lenses the polynomial model breaks down, because the field of view approaches or exceeds \(180°\) and \(\tan\) diverges. Fisheye models parameterise the angle instead:

Equidistant fisheye

Other standard fisheye projections: stereographic \(r = 2f\tan(\theta/2)\), orthographic \(r = f\sin\theta\), equisolid \(r = 2f\sin(\theta/2)\). Choosing the wrong family produces a residual that no amount of polynomial refinement removes.

Radiometry, reflectance and exposure

Geometry says where a point lands; radiometry says how bright it is. The recorded value is the end of a chain, each stage of which some method depends on:

Irradiance to pixel value

• \(L\) — scene radiance toward the camera
• \(t\) — exposure time
• \(\cos^4\alpha\) — natural vignetting — off-axis rays are attenuated
• \(n\) — noise (see below)
• \(f(\cdot)\) — the camera response function, typically non-linear (gamma-like)

The Lambertian assumption

A Lambertian surface has radiance independent of viewing direction:

This single assumption underwrites most of geometric vision Brightness constancy in optical flow, the matching cost in stereo, the photometric residual in self-supervised depth and direct SLAM, and the RGB loss in neural rendering all assume a point looks the same from two viewpoints. Specularities, transparency, subsurface scattering and retro-reflection all violate it — and they do so silently, producing confident wrong geometry rather than a flagged failure.

Sensor noise

Noise is not additive Gaussian with constant variance. It is dominated by photon shot noise, which is Poisson and therefore signal-dependent:

Heteroscedastic noise model

\(a\) captures shot noise (proportional to the number of photons, hence to intensity), \(b\) captures read noise and dark current (constant). Denoisers and uncertainty-weighted losses trained on constant-\(\sigma\) Gaussian noise systematically under-perform on real raw data, which is the argument for training on realistic degradations.

Rolling shutter and motion blur

Rolling shutter CMOS sensors expose rows sequentially. Row \(r\) is captured at \(t_r = t_0 + r\,\tau\), so each row sees a different camera pose: \(\mathbf{u}_r = \pi\big(\mathbf{K}\,\mathbf{T}(t_0 + r\tau)\,\mathbf{X}\big)\) A single image no longer corresponds to one projection matrix. Ignoring this under fast motion produces skew, wobble and a systematic bias in estimated pose — which is why rolling-shutter models appear in any serious VO system on a phone or drone.

Motion blur A finite exposure integrates the scene over camera motion: \(I*{\text{blur}}(\mathbf{u}) = \frac{1}{t}\int_0^{t} I\big(\mathbf{W}(\mathbf{u};\boldsymbol{\theta}(s))\big)\,ds\) This is a _spatially varying* convolution whenever the motion is not a pure translation, which is what makes blind deblurring hard. See deblurring.

Failure modes

Takeaways

1. The projection equation is the whole of geometric vision’s forward model. Everything downstream is an attempt to invert it under ambiguity.
2. Perspective division creates the three fundamental ambiguities — depth from one view, global scale, and quadratically degrading depth resolution.
3. Optimise rotations on the manifold, with \(\mathfrak{so}(3)\) increments, not by perturbing matrix entries.
4. Distortion is observable only at the periphery, which dictates how calibration data must be collected.
5. The Lambertian assumption is load-bearing and routinely false. Most “mysterious” failures in flow, stereo, depth and rendering are this assumption breaking quietly.
6. Resizing an image changes its intrinsics. An augmentation that does not update \(\mathbf{K}\) leaves the geometry wrong before training starts.

Calibration and sensor alignment

The least glamorous family in vision, the one the decade left almost untouched, and the one every metric claim still rests on.

Problem definition and observability

Input: images of a known target, or natural correspondences across views. Output: intrinsics \(\mathbf{K}\), distortion \(\mathbf{d}\), per-view extrinsics \((\mathbf{R}_i, \mathbf{t}_i)\), and — for multi-sensor rigs — the rigid transform between sensors.

What is and is not observable

• From a single view of a plane: a homography, 8 DoF. Not enough to separate \(\mathbf{K}\) from pose — this is why Zhang’s method needs several orientations.
• Focal length vs. distance: nearly unidentifiable if all views are fronto-parallel at similar depth. The Jacobian columns become almost collinear and the normal equations are ill-conditioned.
• Distortion: only observable where \(r\) is large, i.e. at the image periphery. A centre-only dataset yields confident nonsense at the edges.
• Principal point: weakly observable, and strongly correlated with translation. Many pipelines fix \((c_x,c_y)\) at the image centre rather than estimate it badly.

The objective

Calibration is an instance of the general template with \(\theta = \{\mathbf{K}, \mathbf{d}, \mathbf{R}_i, \mathbf{t}_i\}\) and a reprojection residual.

Reprojection objective

_\(\mathbf{u}_{ij}\) is the detected corner, \(\mathbf{X}_j\) its known 3D position on the target, and \(\pi\) the full projection including distortion. Minimised by Levenberg–Marquardt. The residual is measured in pixels, which is the only space where the noise model is approximately isotropic — this is why one never minimises error in world units here._

Why a single RMS number is not enough

The reported RMS reprojection error is an average over all corners in all views. It hides exactly the failures that matter:

• Plot residuals spatially. A radial pattern means the distortion model is wrong or under-parameterised. Systematic structure is bias; scatter is noise.
• Plot residuals per view. One blurry or mis-detected view can dominate, or be silently absorbed into a distorted \(\mathbf{K}\).
• Check per-camera and cross-camera residuals separately in a multi-camera rig. A good average can hide one badly modelled camera.

The Direct Linear Transform

The standard linear initialiser. From \(\mathbf{u} \times \mathbf{P}\tilde{\mathbf{X}} = \mathbf{0}\) — the cross product vanishes when the projection is parallel to the measurement — each correspondence yields two independent linear equations in the 12 entries of \(\mathbf{P}\):

Stack \(\geq 6\) correspondences and take \(\mathbf{p}\) as the right singular vector of \(\mathbf{A}\) with the smallest singular value. \(\mathbf{K}\) and \(\mathbf{R}\) are then recovered from the left 3×3 block of \(\mathbf{P}\) by RQ decomposition.

Normalise the points first — this is not optional Raw pixel coordinates have magnitudes around \(10^3\) while homogeneous entries are \(1\), so the columns of \(\mathbf{A}\) differ by three orders of magnitude and the SVD is dominated by numerical noise. Hartley normalisation applies a similarity \(\mathbf{T}\) that centres the points at the origin and scales their mean distance to \(\sqrt{2}\), solves in normalised coordinates, then un-normalises: \(\mathbf{P} = \mathbf{T}'^{-1}\hat{\mathbf{P}}\mathbf{T}\). The same argument applies to the eight-point algorithm for the fundamental matrix.

Zhang’s method

The practical workhorse: several views of a planar grid. Setting \(Z=0\) on the target plane collapses the projection to a homography.

Orthonormality of \(\mathbf{r}_1, \mathbf{r}_2\) gives two constraints per view on \(\mathbf{B} = \mathbf{K}^{-\top}\mathbf{K}^{-1}\), the image of the absolute conic:

\(\mathbf{B}\) is symmetric with 6 unknowns (5 up to scale), so three views in general position suffice. \(\mathbf{K}\) follows from the Cholesky factorisation of \(\mathbf{B}\). Distortion is then introduced and everything is refined jointly by nonlinear least squares.

The procedure in full: detect corners to subpixel accuracy → estimate per-view homographies → solve linearly for \(\mathbf{B}\), hence \(\mathbf{K}\) → recover per-view \((\mathbf{R}_i,\mathbf{t}_i)\) → initialise distortion at zero → refine all parameters by LM on the reprojection objective.

Stereo calibration and rectification

For two cameras, calibration additionally yields the relative pose \((\mathbf{R}, \mathbf{t})\), from which the epipolar geometry follows:

Essential and fundamental matrices

\(\mathbf{E}\) has 5 DoF (3 rotation + 2 translation direction; scale is unrecoverable), \(\mathbf{F}\) has 7. The epipolar constraint says a point in one image must lie on a line in the other — it reduces correspondence search from 2D to 1D.

Rectification applies homographies \(\mathbf{H}, \mathbf{H}'\) that map the epipoles to infinity along the horizontal axis, so corresponding points share a row and disparity is a pure horizontal shift. This is the step that makes classical stereo tractable, and it is why an uncalibrated or mis-rectified pair produces stereo that fails in a way no matching cost can repair.

Depth from disparity, after rectification

Multi-sensor alignment

Hand–eye calibration Find the unknown rigid transform \(\mathbf{X}\) between a robot’s end effector and a camera, from pairs of motions: \(\mathbf{A}\mathbf{X} = \mathbf{X}\mathbf{B}\) where \(\mathbf{A}\) is the measured end-effector motion and \(\mathbf{B}\) the observed camera motion. The rotation part, \(\mathbf{R}_A\mathbf{R}_X = \mathbf{R}_X\mathbf{R}_B\), is solvable in closed form (Tsai–Lenz, Park–Martin); the translation follows linearly. Requires at least two motions with non-parallel rotation axes, or the rotation is under-determined.

Camera–LiDAR and camera–IMU > Camera–LiDAR: minimise reprojection of 3D points onto image edges or a shared target; the objective is the same reprojection form, with the LiDAR points playing the role of \(\mathbf{X}_j\).

Camera–IMU: estimate the rigid transform and the time offset \(t_d\), since unsynchronised clocks masquerade as a spatial offset under motion. Typically solved inside a visual–inertial bundle adjustment with IMU preintegration.

Temporal synchronisation is a first-class parameter, not a detail. A 10 ms offset on a camera moving at 1 m/s is a 1 cm systematic error that calibration will happily absorb into the extrinsics, producing a rig that is self-consistent at one speed and wrong at every other.

Engineering tricks

• Cover the entire image with the target, especially the corners — distortion lives there.
• Vary orientation and distance; include strongly tilted views to decorrelate focal length from depth.
• Subpixel corner refinement (saddle-point fitting); checkerboards give higher accuracy than printed circles, ChArUco beats both for partial visibility.
• Initialise distortion conservatively — start with \(k_1\) only and add terms while the residual improves; more coefficients always fit the training corners better and often extrapolate worse.
• Reject blurry views and outlier corners; use a robust \(\rho\) (Huber) in the refinement.
• Print flatness matters. A target taped to a slightly curved surface introduces a systematic bias that looks exactly like distortion.
• Re-calibrate after any mechanical shock, temperature swing, or lens re-focus.

Metrics and failure modes

Takeaways

1. Calibration is a nonlinear least-squares problem with a linear initialiser. DLT/Zhang give the starting point; Levenberg–Marquardt performs the refinement.
2. Normalise before any DLT-style solve. Conditioning, not cosmetics.
3. Observability dictates data collection. Peripheral coverage for distortion, orientation variety for focal length.
4. Never trust one RMS number. Inspect residuals spatially and per view.
5. Time offset is an extrinsic parameter. Unsynchronised sensors produce bias that calibration hides rather than reveals.
6. This family was not learned away. Every metric claim from a modern geometry model still rests on a camera model somebody estimated classically.

Vision objectives and optimization

One template covers most of the field. Learning which of its five slots a method fills is the fastest way to understand an unfamiliar paper.

The common objective template

The template

• \(\theta\) — unknowns: image, motion, geometry, pose, labels, or network parameters
• \(r_i\) — an observation residual — see the nine families
• \(\rho\) — a robust penalty shaping the influence of large residuals
• \(w_i\) — confidence or visibility weight
• \(\mathcal{V}\) — the valid set: visible, unoccluded, in-bounds observations
• \(R_k, \lambda_k\) — priors and their weights

Read any method by filling the five slots What is \(\theta\)? What is \(r_i\), and against what — a label, another measurement, or a rendering? What are \(\mathcal{V}\) and \(w_i\)? What is \(\rho\)? What are \(R_k\)? Most papers state the first and fourth explicitly; the interesting engineering is usually hidden in \(\mathcal{V}\) and \(w_i\) — occlusion handling, visibility masks, and outlier rejection.

The MAP interpretation

The template is not arbitrary. Maximising the posterior \(p(\theta\mid \mathbf{z}) \propto p(\mathbf{z}\mid\theta)\,p(\theta)\) and taking negative logs gives

The data term _is the negative log-likelihood; the regularizer is the negative log-prior. This fixes which \(\rho\) corresponds to which noise model._

Huber

Redescending penalties have a basin of attraction Cauchy and Tukey drive the influence of large residuals toward zero, which is the required behaviour once the estimate is approximately correct, and the wrong behaviour from a poor initialisation, where true inliers resemble outliers and are discarded. This is why the ordering is always RANSAC first, robust refinement second, and why coarse registration precedes ICP.

The nine data terms

The self-supervision insight hiding in this table Rows 1–4 and 8–9 require no human labels — they compare measurements against other measurements under a geometric or physical model. Rows 5–7 require labels. Nearly the whole self-supervised story of 2019–2025 is work migrating out of the labelled rows into the unlabelled ones. See the supervision ladder.

Regularizers and priors

Common spatial priors

Total variation preserves discontinuities (L1 on gradients tolerates a few large jumps); Tikhonov smoothness penalises them quadratically and blurs edges. The choice determines whether the depth map has crisp object boundaries or soft ones.

Edge-aware smoothness

Smooth the estimate where the image is flat, allow jumps where the image has an edge. Standard in self-supervised depth and flow. Note it imports the assumption that _depth discontinuities coincide with intensity edges, which fails for low-contrast object boundaries._

Other priors in routine use: sparsity (\(\ell_1\)), low rank (nuclear norm), non-local self-similarity, piecewise planarity, surface-normal consistency, rigidity, temporal constancy, entropy control, and topology constraints. For the relational family — cycle, left–right, forward–backward — see consistency constraints.

Regularization is not decoration It resolves the underdetermination that remains after the data term. Horn–Schunck is the standard case: brightness constancy gives one scalar equation per pixel for two unknowns, so the system is underdetermined by construction — the aperture problem. Smoothness is what makes it well-posed at all.

Optimization machinery

Gauss–Newton and Levenberg–Marquardt

For a nonlinear least-squares objective \(\tfrac12\lVert \mathbf{r}(\theta)\rVert^2\) with Jacobian \(\mathbf{J} = \partial\mathbf{r}/\partial\theta\):

GN approximates the Hessian as \(\mathbf{J}^\top\mathbf{J}\), dropping second-derivative terms — valid when residuals are small or nearly linear. LM adds a damping term: large \(\mu\) gives a short, safe gradient-descent step; small \(\mu\) recovers GN’s fast quadratic convergence. \(\mu\) is adapted each iteration based on whether the step reduced the cost.

Robust penalties enter through iteratively reweighted least squares: minimising \(\sum\rho(r_i)\) is equivalent to repeatedly solving a weighted least-squares problem with \(w_i = \psi(r_i)/r_i\), recomputed each iteration.

Exploiting structure: the Schur complement

In bundle adjustment \(\theta\) splits into camera parameters \(\mathbf{c}\) and points \(\mathbf{p}\), and a point is seen by few cameras, so the system is very sparse and block-structured:

\(\mathbf{C}\) is block-diagonal (3×3 per point), so \(\mathbf{C}^{-1}\) is trivial. Solving the reduced camera system is what makes BA over thousands of views feasible. See bundle adjustment.

Discrete optimisation

• Dynamic programming — Exact on a chain. Used for scanline stereo and, along multiple 1D paths, as the approximation inside Semi-Global Matching.

• Graph cuts — Exact for binary submodular energies via max-flow/min-cut; \(\alpha\)-expansion gives a bounded approximation for multi-label problems. Used in segmentation and stereo.

• Belief propagation — Exact on trees, approximate (“loopy”) on grids. Message passing over an MRF; competitive with graph cuts on stereo.

RANSAC and robust model fitting

Sample a minimal set, fit, count inliers, repeat, keep the best. The required number of iterations follows from wanting probability \(p\) of at least one all-inlier sample:

• \(\varepsilon\) — outlier ratio
• \(s\) — minimal sample size (4 for a homography, 5 for essential, 3 for PnP)
• \(p\) — desired success probability, typically 0.99

The exponential in \(s\) is why minimal solvers matter so much: at \(\varepsilon = 0.5\), \(s=4\) needs ~72 iterations while \(s=8\) needs ~1177.

• MLESAC maximises a likelihood rather than counting inliers, so near-threshold points contribute smoothly.
• PROSAC samples from matches ordered by descriptor quality, finding a good hypothesis far sooner in practice.
• LO-RANSAC runs a local optimisation on the inlier set of each promising hypothesis.
• Always refine on all inliers after RANSAC — the minimal-sample estimate is unbiased but high-variance.

Takeaways

1. One template covers most of vision. Filling its five slots specifies the method.
2. The penalty encodes a noise model. Choosing L2 assumes the residuals are Gaussian.
3. Redescending penalties need good initialisation. Hence RANSAC first, robust refinement second — always in that order.
4. Regularizers fix underdetermination, not numerics. Without them many vision problems have no unique solution at all.
5. Structure is what makes large problems tractable. Sparsity and the Schur complement, not raw solver speed.
6. The optimised surrogate is rarely the reported metric. Several of the decade’s results come from reducing that gap.

Multiscale vision and filtering

The single most reused structural idea in the field, wearing a different costume in every decade: image pyramids, feature pyramids, U-Net skips, cost-volume cascades, hash grids, hierarchical transformers.

Convolution and correlation

Convolution flips the kernel; correlation does not. They coincide for symmetric kernels — which is why the distinction rarely matters for Gaussians and always matters for derivative filters. Deep-learning “convolution” layers compute correlation; since the kernel is learned, the flip is absorbed into it.

Two properties carry most of the practical weight:

• Separability. A 2D Gaussian factors as \(G_\sigma(x,y) = G_\sigma(x)G_\sigma(y)\), reducing a \(k\times k\) filter from \(O(k^2)\) to \(O(2k)\) per pixel. The same factorisation motivates depthwise separable convolutions.
• Linearity and shift invariance. Together these make the frequency-domain view available, and make convolution the unique operator commuting with translation.

Gaussian scale space

The scale-space representation convolves the image with Gaussians of increasing width:

Equivalently, \(L\) solves the heat equation \(\partial_t L = \tfrac12\nabla^2 L\) with \(L(\cdot;0)=I\). The Gaussian is the _unique kernel that creates no new extrema as scale increases — the formal statement of “coarser means simpler”, and the reason scale space is built with Gaussians rather than box filters._

Gaussian and Laplacian pyramids

Gaussian pyramid > \(G_0 = I\), and \(G_{\ell+1} = \operatorname{down}_2(G_\ell * g)\). Blur then subsample — the blur is anti-aliasing, not decoration. Skipping it aliases high frequencies into low ones, which is exactly the artefact that makes naive downsampling destroy thin structures.

Laplacian pyramid > \(L_\ell = G_\ell - \operatorname{up}_2(G_{\ell+1})\), with \(L_n = G_n\). Each level holds one octave of bandpass detail, and the decomposition is invertible: \(G_\ell = L_\ell + \operatorname{up}_2(G_{\ell+1})\). This exact reconstruction is what makes it the right structure for multiband blending.

Scale-normalised Laplacian — the blob detector

Without the factor \(t\), response magnitude decays with scale and extrema are never found at coarse scales. Extrema of the scale-normalised response in \((x,y,t)\) give both a location _and a characteristic scale — this is the mechanism behind SIFT’s scale invariance, and the Difference-of-Gaussians is simply a cheap approximation to it._

The frequency-domain view

Convolution theorem

Three things follow that are worth carrying around:

• Blurring is low-pass filtering. \(\mathcal{F}\{G_\sigma\}\) is a Gaussian of width \(1/\sigma\), so a wide blur kernel is a narrow frequency window. Deconvolution is division by this — and division by near-zero values is why deblurring is ill-posed.
• Sampling requires band-limiting. Nyquist: a signal sampled at rate \(f_s\) must contain no content above \(f_s/2\), or aliasing folds it down. Hence blur-before-subsample.
• Spectral bias. Networks fit low frequencies first. This is precisely why NeRF needs positional encoding: an MLP on raw coordinates cannot represent high-frequency detail in reasonable time.

Image derivatives and the structure tensor

Differentiation amplifies noise, so derivatives are always taken of a smoothed image — and by associativity that is a single convolution with the derivative of the Gaussian:

Sobel and Scharr are small discrete approximations to exactly this. Aggregating first derivatives over a window gives the structure tensor:

Structure tensor

Its eigenvalues \(\lambda_1 \ge \lambda_2\) classify local structure: both small → flat; one large → edge; both large → corner. This one matrix underlies Harris corners, the Shi–Tomasi “good features to track” score, and the invertibility condition of Lucas–Kanade — three methods that appear unrelated until the shared depend on \(\mathbf{M}\) being well-conditioned.

Nonlinear and edge-preserving filters

Morphology

Erosion and dilation. Their compositions are opening \(I\circ B = (I\ominus B)\oplus B\), which removes structures smaller than \(B\), and closing \(I\bullet B = (I\oplus B)\ominus B\), which fills holes smaller than \(B\). Both are idempotent, which is why they are safe as post-processing — applying twice changes nothing.

Coarse-to-fine optimisation

The reason pyramids matter beyond efficiency. For a warp-based objective such as photometric alignment, the cost surface has many local minima at fine scale: shifting by one period of a texture is locally as good as the true alignment. Blurring removes that structure.

Coarse-to-fine schedule

At level \(\ell\) a displacement of \(d\) pixels appears as \(d/2^\ell\), so a large motion becomes a small one and falls inside the linearisation’s basin of attraction. Flow and disparity estimates are scaled by 2 when passed down a level.

The structural flaw, and RAFT’s answer A small, fast-moving object disappears at coarse levels and can never be recovered at fine ones — the coarse estimate has already committed to the background motion. RAFT rejects the pyramid entirely: build all-pairs correlation at a single high resolution and refine iteratively with a shared recurrent operator. The lesson generalises — coarse-to-fine buys a larger basin of attraction at the price of small structure, and that trade is not always worth taking.

The same idea, six decades of costumes

Takeaways

1. Blur before subsampling. Everything else about pyramids is downstream of Nyquist.
2. The Gaussian is not an arbitrary choice — it is the unique kernel that creates no new structure as scale coarsens.
3. Scale-normalisation is what makes scale selection possible, and is the mechanism behind SIFT’s invariance.
4. One matrix — the structure tensor — underlies Harris, Shi–Tomasi and Lucas–Kanade.
5. Coarse-to-fine trades small structure for a larger basin of attraction. RAFT showed that trade is optional.
6. Every generation reinvents the pyramid. Recognising it identifies each reinvention as the same construction.

The shared pipeline and the objective template

Almost every vision system, classical or learned, decomposes into the same ten operations — and a large fraction of the field can be written as a single optimisation template.

The ten-step pipeline

This decomposition is deliberately method-agnostic. A 1998 stereo algorithm and a 2025 feed-forward geometry transformer both instantiate it; they differ in which steps are learned and which are hand-specified, not in which steps exist.

1 · Acquire and calibrate

Model lens distortion, camera intrinsics, exposure, synchronization, sensor-to-sensor extrinsics and rolling shutter. Errors introduced here are systematic and cannot be recovered downstream — they show up as bias, not noise.

2 · Normalize and preprocess

Undistort, rectify, denoise, resize, normalize intensity or colour, construct image pyramids, optionally mask invalid regions. Rectification in particular converts a 2D search into a 1D one and is why classical stereo is tractable at all.

3 · Represent

Choose the carrier of information: pixels, gradients, patches, keypoints and descriptors, superpixels, feature maps, tokens, voxels, points, meshes, signed distance fields, or radiance fields. This is the layer deep learning changed most.

4 · Generate hypotheses

Propose matches, boxes, masks, poses, disparities, flow vectors, tracks or scene structure. The move from enumerated hypotheses (sliding windows, anchors, disparity ranges) to learned ones (queries, proposals) is one of the decade’s clearest trends.

5 · Score with data terms

Compare predictions to observations or labels using photometric, geometric, semantic, appearance or distributional residuals. See the data-term table below.

6 · Regularize

Inject priors: smoothness, sparsity, piecewise rigidity, temporal continuity, shape constraints, weight decay. Not decoration — this is what resolves the underdetermination the data term leaves behind.

7 · Reject outliers and resolve ambiguity

RANSAC, robust penalties, gating, confidence thresholds, visibility masks, left–right checks, hard-negative mining, uncertainty weighting. Real residual distributions are contaminated; assuming they are not is the most common practical error.

8 · Optimize

Closed-form estimation, dynamic programming, graph cuts, belief propagation, Gauss–Newton, Levenberg–Marquardt, gradient descent, or end-to-end backpropagation. The choice is dictated by the structure of the objective, not by fashion.

9 · Post-process

Non-maximum suppression, connected components, morphology, CRFs, median filtering, hole filling, track birth/death rules, mesh cleanup. Each of these embodies a prior the network was not asked to learn — which is exactly why removing them took a decade.

10 · Evaluate and calibrate confidence

Select metrics aligned with the application, test robustness, and inspect failure modes, in place of a single average score.

Three structural observations

• Image pyramids and wavelets provide multiscale analysis and accelerate operations — the idea reappears as Gaussian/Laplacian pyramids, FPNs, U-Net skips, coarse-to-fine cost volumes, multiresolution hash grids and hierarchical transformers.
• Geometric warps — rotation, shear, affine, perspective — are the vocabulary of step 4 whenever correspondence is involved.
• Energy minimisation connects many classical algorithms to Bayesian or Markov-random-field estimation, which is what makes step 5 and step 6 formally comparable across methods.

The common objective template

A large fraction of computer vision can be written in one form:

θ^* = arg min_θ Σ_{i ∈ 𝒱} w_i ρ( r_i(θ) ) + Σ_k λ_k R_k(θ)

θ — unknown image, motion, geometry, pose, label or network parameters · r_i — an observation residual · ρ — a robust penalty · w_i — a confidence or visibility weight · 𝒱 — the set of valid (visible, unoccluded, in-bounds) observations · R_k — priors or regularizers with weights λ_k.

Why this template is worth memorising It makes classical and learned methods directly comparable. A network may predict θ instead of an optimiser solving for it, but the loss still encodes what a valid visual explanation should look like. Four questions identify a method: what θ is, what the residual is, what the robust penalty and the visibility set are, and what priors are imposed. Most papers answer only the first and third explicitly — the interesting engineering is usually hidden in w_i and 𝒱.

Data terms

The residual is where the physics enters. Each data term carries an assumption that, when violated, produces a characteristic failure — listed in the last column and expanded on the failure-mode checklist.

The self-supervision insight hiding in this table Rows 1–4 and 8–9 need no human labels — they compare measurements against other measurements under a geometric or physical model. Rows 5–7 need labels. Nearly the whole self-supervised-learning story of 2019–2025 is the field moving work out of the labelled rows and into the unlabelled ones: photometric reprojection replaced labelled depth, rendering residuals replaced labelled geometry, and contrastive/masked objectives manufactured a residual out of the data’s own structure.

Regularizers and priors

Common priors include first- or second-order smoothness, total variation, sparsity, low rank, non-local self-similarity, piecewise planarity, surface-normal consistency, cycle consistency, left–right consistency, forward–backward consistency, rigidity, temporal constancy, entropy control, and shape or topology constraints.

Regularization is not numerical decoration It resolves the underdetermination that remains after the data term. The standard case is Horn–Schunck optical flow: the brightness-constancy equation gives one scalar constraint per pixel, but flow has two unknown components per pixel, so the system is underdetermined by construction — the aperture problem. Smoothness is not a convenience there; it is what makes the problem well-posed at all.

• Spatial priors — Smoothness (1st/2nd order), total variation, piecewise planarity, surface-normal consistency, edge-aware weighting. Assume the world is locally simple, and encode where it is allowed not to be.

• Structural priors — Sparsity, low rank, non-local self-similarity, shape and topology constraints. Assume the signal lives on a much smaller set than the ambient space.

• Relational priors — Cycle, left–right, forward–backward and temporal consistency; rigidity. Assume that several estimates of the same quantity must agree — see the consistency table.

How to read a method with this template

Given a new paper, the template turns a vague “what does it do” into five answerable questions:

1. What is θ? Scene variables, or network parameters, or both in alternation?
2. What is r_i? Which of the nine data terms, and against what — a label, another measurement, or a rendering?
3. What is 𝒱 and w_i? Which observations are excluded, and how is confidence assigned? This is where occlusion, visibility and outlier handling live.
4. What is ρ? Squared error, L1, Charbonnier, Huber, Tukey, or a learned/implicit robustness?
5. What are R_k and λ_k? Which priors, and how are the weights set — tuned, annealed, or learned?

A method that cannot be placed in this template is usually either a pure representation change (a new backbone) or a pure inference change (a new sampler) — both are easier to evaluate once identified as such.

Image level

Restoration and enhancement

The clearest case in vision where the unknown being optimised is the _image itself — and the family in which the gap between the optimised quantity and the perceived one is provably irreducible._

The generic degradation model

Forward model

• \(\mathbf{x}\) — the latent clean image
• \(k\) — blur kernel / point-spread function
• \(\mathbf{S}_s\) — sampling or decimation by factor \(s\)
• \(c\) — camera-response and colour-filter effects
• \(\mathbf{n}\) — noise, generally signal-dependent

Each task is a different instantiation: denoising sets \(k=\delta\), \(s=1\); deblurring keeps \(k\) non-trivial; super-resolution has \(s>1\); demosaicing makes \(\mathbf{S}\) the Bayer pattern; inpainting makes \(\mathbf{S}\) a binary mask. Restoration inverts it under a prior:

This is the general template with \(\theta = \mathbf{x}\). Without \(R\) the problem is ill-posed: the forward operator has a near-null space, so infinitely many \(\mathbf{x}\) explain \(\mathbf{y}\) equally well.

Deconvolution

Wiener filter

The MMSE linear estimator, in the frequency domain. The added noise-to-signal ratio term is what prevents division by near-zero \(K\) — with zero noise it degenerates to naive inverse filtering, which explodes exactly where the blur has killed the signal. This single term is the whole reason regularization exists in restoration.

Richardson–Lucy

Multiplicative EM iteration for Poisson noise; preserves non-negativity and total flux by construction, which is why it dominates in astronomy and microscopy. Amplifies noise as iterations grow — the iteration count is itself the regularizer, and stopping early is the standard practice.

The four classical deconvolution methods differ only in what is known. Wiener: \(k\) known, Gaussian noise, known spectra. Regularized: \(k\) known, explicit \(R(\mathbf{x})\). Richardson–Lucy: \(k\) known, Poisson noise. Blind: \(k\) unknown, so \(k\) and \(\mathbf{x}\) are estimated jointly — which is severely ill-posed, since \((k*\delta)\) and \((\delta*k)\) explain the data identically, and needs priors on both.

Classical image priors

Total variation (ROF)

\(\ell_1\) on gradient magnitude tolerates a few large jumps, so edges survive while noise is removed. The characteristic artefact is staircasing: smooth ramps are flattened into piecewise-constant steps, because the prior literally prefers them.

Non-local means

Weights come from similarity of _patches \(P\), not proximity of pixels. The prior is self-similarity: natural images repeat their own structure, so a noisy patch has clean relatives elsewhere in the same image._

BM3D — the strongest statement of the self-similarity prior Four stages: (1) block matching groups similar patches into a 3D stack; (2) a 3D transform is applied and coefficients are collaboratively shrunk — because similar patches share structure, the true signal is sparse in that basis while noise is not; (3) estimates are aggregated back with weights; (4) the first estimate guides a second Wiener pass. It remained the benchmark to beat for nearly a decade, and learned denoisers only clearly surpassed it once they were trained on realistic rather than Gaussian noise.

Deep objectives

ESRGAN combines perceptual, adversarial and L1 content terms. It improves perceptual realism and simultaneously reduces PSNR — which is not a bug, and is the cleanest illustration of the following.

The perception–distortion tradeoff It is a proven result, not an empirical observation, that distortion (any full-reference metric such as MSE) and perceptual quality (divergence between the output distribution and the natural-image distribution) cannot both be optimal. Improving one past a point necessarily degrades the other.

Consequences: report both a fidelity metric and a perceptual metric; never compare a GAN-based method to an MSE-based one on PSNR alone; and do not deploy a generative restorer in any measurement or safety-critical setting — it is drawing a plausible sample, not recovering a measurement, and it will hallucinate texture that was never in the scene.

Task-specific notes

• Super-resolution — The degradation used in training defines the method. Bicubic-downsample training generalises poorly to real images; “blind SR” instead randomises blur kernels, noise and compression to cover realistic degradations. Sub-pixel (pixel-shuffle) upsampling avoids the checkerboard artefacts of transposed convolution.

• Demosaicing — Recovering three channels from a Bayer mosaic. Best solved jointly with denoising, because independent demosaicing propagates and correlates the noise, after which no denoiser can undo it.

• HDR and tone mapping — Estimate the camera response \(f\), merge bracketed exposures weighted by reliability, then compress dynamic range for display. Merging must happen in linear light; tone mapping is a perceptual, not a physical, step.

• Inpainting — Classical: diffusion for small gaps, patch-based synthesis (PatchMatch) for texture. Modern: masked diffusion. Large holes are generation, not restoration — the information is simply absent.

Engineering tricks

• Work in linear light when the physics demands it — merging, blending and noise modelling are all wrong in gamma space.
• Use raw sensor data where possible; the ISP has already applied irreversible non-linearities.
• Model heteroscedastic noise, \(\sigma^2(I) = aI + b\). Constant-\(\sigma\) Gaussian training under-performs on real data.
• Tile with overlap and blend, to avoid seams on large images; pad reflectively rather than with zeros.
• Augment with realistic degradations — random kernels, JPEG artefacts, sensor noise — not just additive Gaussian.
• Report fidelity and perception separately.
• Prevent hallucinated detail wherever the output feeds a measurement.

Metrics and failure modes

Failure modes: staircasing from TV; over-smoothing from L2; hallucinated texture from adversarial losses; ringing from deconvolution near strong edges; colour shifts from per-channel processing; and — most consequentially — train/test degradation mismatch, where a model trained on bicubic downsampling collapses on real photographs.

Takeaways

1. All restoration tasks are one forward model with different \(k\), \(s\) and \(\mathbf{S}\).
2. The prior is not optional — the forward operator has a near-null space, so the data alone does not determine the answer.
3. Classical deconvolution methods differ in what is known, not in their objective structure.
4. BM3D is the self-similarity prior taken seriously, and it set the bar for a decade.
5. Perception and distortion trade off provably. Report both; never rank across the tradeoff with one number.
6. A generative restorer invents plausible detail. That is a feature for photography and a defect for measurement.

Features and visual primitives

Learned descriptors displaced SIFT and ORB for matching — but the _structure of this pipeline survived intact into learned feature matching, and NMS, invented here, outlived the family by a decade._

The classical pipeline

Six stages, in this order, for every classical detector–descriptor:

1 · Smooth at one or more scales

Differentiation amplifies noise; every derivative is taken of a Gaussian-smoothed image. See image derivatives.

2 · Compute derivatives or local comparisons

Gradients for edges and corners, Laplacian/DoG for blobs, intensity comparisons for binary tests (FAST, BRIEF).

3 · Score candidate points

A cornerness, edgeness or blobness response per pixel and per scale.

4 · Non-maximum suppression

Keep only local maxima of the response. This is where NMS was invented, and it took until DETR to remove it from detection.

5 · Refine to subpixel

Fit a quadratic to the response around the maximum and take its vertex.

6 · Assign scale and orientation, then describe

Normalising for scale and dominant orientation is what makes the descriptor invariant.

Edges: Canny

Canny is an optimality argument as well as an algorithm: it maximises a criterion combining good detection, good localisation and a single response per edge. Four stages:

1. Gaussian smoothing at scale \(\sigma\).
2. Gradient magnitude and orientation.
3. Non-maximum suppression along the gradient direction — thins ridges to one-pixel width. This is the step that distinguishes Canny from a plain gradient threshold.
4. Hysteresis thresholding with \(\tau_{\text{low}} < \tau_{\text{high}}\): seed edges above \(\tau_{\text{high}}\), then extend through pixels above \(\tau_{\text{low}}\) connected to a seed. Two thresholds give continuity that one cannot.

Corners: Harris and Shi–Tomasi

Consider the weighted SSD of a patch against itself under a small shift \(\Delta\mathbf{u}\):

with \(\mathbf{M}\) the structure tensor. A corner is a point where \(E\) is large in _every direction, i.e. both eigenvalues of \(\mathbf{M}\) are large._

Corner responses

Harris avoids an eigendecomposition by using \(\det = \lambda_1\lambda_2\) and \(\operatorname{tr} = \lambda_1+\lambda_2\), with \(\kappa \approx 0.04\)–\(0.06\). Shi–Tomasi thresholds \(\lambda_{\min}\) directly and is the better predictor of whether a patch is _trackable — which is why it is called “good features to track” and why it pairs with Lucas–Kanade, whose solvability requires exactly this matrix to be well-conditioned._

The aperture problem, stated as linear algebra > \(\lambda_2 \approx 0\) means \(\mathbf{M}\) is rank-deficient: there is a direction along which the patch looks the same, so motion along it is unobservable. That is an edge. Corner detection and the aperture problem are the same statement about the conditioning of one matrix.

Blobs and scale selection: DoG and SIFT

Difference of Gaussians

A cheap approximation to the scale-normalised Laplacian. Extrema of \(D\) over the 3D neighbourhood in \((x,y,\sigma)\) give both position _and characteristic scale — the mechanism behind scale invariance._

SIFT in full:

1. Build a Gaussian/DoG pyramid over octaves and sub-scales.
2. Find \((x,y,\sigma)\) extrema; refine by fitting a 3D quadratic.
3. Reject unstable extrema: low contrast \(\lvert D\rvert < \tau\), and edge-like responses using the curvature ratio

\(\mathbf{H}\) is the 2×2 Hessian of \(D\). The same trace/determinant trick as Harris, reused to discard points that lie along an edge and are therefore poorly localised in one direction.

1. Assign dominant orientation from a 36-bin histogram of gradient orientations weighted by magnitude; duplicate the keypoint for each secondary peak above 80% of the maximum.
2. Describe: a \(4\times4\) grid of 8-bin orientation histograms, computed relative to the dominant orientation, giving a 128-D vector; normalise, clip at 0.2, renormalise to reduce the influence of large gradients from illumination change.

Fast and binary: FAST, BRIEF, ORB

• FAST — A pixel is a corner if \(n\) contiguous pixels on a radius-3 Bresenham circle of 16 are all brighter than \(I_p+t\) or all darker than \(I_p-t\). A high-speed test on pixels 1, 5, 9, 13 rejects most candidates in a few comparisons.

• BRIEF — A binary string from \(n\) intensity comparisons at fixed sampling pairs: \(b_i = \mathbb{1}[I(\mathbf{a}_i) < I(\mathbf{b}_i)]\). Matching becomes a Hamming distance — an XOR and a popcount.

• ORB — Oriented FAST + rotated BRIEF. Orientation from the intensity centroid, \(\vartheta = \operatorname{atan2}(m_{01}, m_{10})\); the BRIEF pattern is steered by \(\vartheta\), and the pairs are learned to be decorrelated and high-variance.

Binary descriptors are a hardware argument, not an accuracy one A 256-bit ORB descriptor is matched with XOR + popcount — a handful of CPU instructions — versus 128 floating-point multiply-adds for SIFT. The accuracy is lower; the throughput is one to two orders of magnitude higher. That trade is why ORB, not SIFT, sits inside real-time SLAM.

Lines and regions: Hough and MSER

Hough transform

Each edge pixel votes for every \((r,\vartheta)\) line through it, tracing a sinusoid in parameter space; collinear pixels produce intersecting sinusoids, so peaks in the accumulator are lines. The \((r,\vartheta)\) parameterisation is used instead of \(y = mx+c\) precisely because the latter has unbounded \(m\) for vertical lines. Generalises to circles and arbitrary shapes at the cost of accumulator dimensionality.

MSER sweeps an intensity threshold and tracks connected components; a region is maximally stable where its area changes slowly with threshold:

Affine-covariant and robust to monotonic illumination change, since only the _ordering of intensities matters. Heavily used in text detection, where characters are stable blobs by construction._

Matching and distances

Lowe’s ratio test

\(d_1, d_2\) are the distances to the nearest and second-nearest neighbours. The insight: an _absolute distance threshold is meaningless because descriptor scale varies, but a match that is much better than the runner-up is distinctive. On SIFT this removes ~90% of false matches while discarding ~5% of correct ones._

• Mutual (cross-check) matching: keep \((a,b)\) only if \(b\) is \(a\)’s nearest neighbour and \(a\) is \(b\)’s.
• Distances: L2 for SIFT-like float descriptors, Hamming for binary, cosine for normalised learned embeddings.
• Geometric verification with RANSAC is mandatory — the ratio test removes ambiguity, not repeated structure.

What replaced this, and what did not

The interesting entry is matching. Classical matching treats each descriptor independently; SuperGlue conditions on the whole set of keypoints in both images via attention, so the assignment respects global consistency — closer in spirit to set prediction than to nearest-neighbour search. Note also that VGGT dispenses with explicit matching altogether.

Failure modes and takeaways

1. One matrix — the structure tensor — explains edges, corners and trackability.
2. Scale selection needs scale-normalised responses, otherwise coarse scales never win.
3. Normalise for orientation and scale before describing. That is the source of the invariance.
4. The ratio test works because it is relative, not absolute.
5. Binary descriptors trade accuracy for throughput, deliberately.
6. The pipeline outlived its operators. Detect → refine → normalise → describe → match → verify is still the shape of learned matching.

Correspondence and registration

Estimating the transformation that aligns two images. The upstream family for stitching, SfM, SLAM, localisation and loop closure — which is why an improvement here propagates everywhere.

Two routes to the same answer

Feature-based (sparse, indirect) Detect → describe → match → reject ambiguous → robustly fit a model → refine on inliers.

Minimises a reprojection residual on a sparse set. Robust to illumination change and large displacement; needs texture and discards most of the image.

Direct (dense, photometric) Initialise a warp → transform → compute photometric residuals → linearise → solve → iterate coarse-to-fine.

Minimises a photometric residual over all valid pixels. Uses all image gradient, sub-pixel accurate; needs a good initialisation and photometric consistency.

The same split reappears one level up as feature-based vs. direct visual odometry. It is the single most consequential architectural choice in geometric vision.

The transform hierarchy

Choosing the model is choosing how many degrees of freedom the data must support. Each row is a subgroup of the one below it.

Choose the simplest model the data supports An over-parameterised warp fits noise and produces alignment that looks plausible and is wrong — the most reliable way to ruin a stitching pipeline. A homography is exact only for a planar scene or a purely rotating camera. Use it on a translating camera viewing a 3D scene and parallax will make it fail, no matter how many inliers RANSAC reports.

Estimating a homography

Cross-multiplying linearises this; each correspondence gives two equations, so four points determine \(\mathbf{H}\) up to scale:

Hartley normalisation is mandatory Raw pixel coordinates are \(O(10^3)\) while homogeneous entries are \(O(1)\), so the columns of \(\mathbf{A}\) differ by three orders of magnitude and the smallest singular vector is dominated by round-off.

Apply a similarity \(\mathbf{T}\) centring the points and scaling their mean distance to \(\sqrt{2}\), solve for \(\hat{\mathbf{H}}\), then un-normalise: \(\mathbf{H} = \mathbf{T}'^{-1}\hat{\mathbf{H}}\mathbf{T}\). The identical argument applies to the eight-point algorithm for \(\mathbf{F}\) and to camera DLT.

The algebraic (DLT) solution minimises a quantity with no geometric meaning. Always refine it by minimising the symmetric transfer error on the inliers:

Direct alignment: Lucas–Kanade

Minimise photometric error over warp parameters \(\mathbf{p}\):

Linearising in \(\Delta\mathbf{p}\) gives a Gauss–Newton step:

For a pure translation \(\mathbf{W}\), \(\partial\mathbf{W}/\partial\mathbf{p} = \mathbf{I}\) and \(\mathbf{H}\) becomes exactly the structure tensor — so LK is solvable precisely where Shi–Tomasi says the patch is trackable.

Inverse compositional: why it is ~10× faster In the forward formulation \(\nabla I\) and hence \(\mathbf{H}\) must be recomputed every iteration, because the warp moves. The inverse compositional trick swaps the roles — the increment is applied to the template and inverted, \(\mathbf{W}(\mathbf{x};\mathbf{p}) \leftarrow \mathbf{W}(\mathbf{x};\mathbf{p})\circ\mathbf{W}(\mathbf{x};\Delta\mathbf{p})^{-1}\) — so \(\nabla T\), the Jacobian and \(\mathbf{H}\) are all constant and precomputable. Same convergence, a fraction of the per-iteration cost. This is standard in any real-time tracker.

Similarity criteria for direct methods

Mutual information is the one that enables cross-modal registration: it assumes only that the two modalities are statistically dependent, not that bright maps to bright. That is why it dominates in medical registration, where a bone is bright in CT and dark in some MR sequences.

Engineering tricks

• Mutual nearest-neighbour plus Lowe’s ratio test before any model fitting.
• Normalise points before DLT — conditioning, not cosmetics.
• RANSAC / MLESAC / PROSAC, then refine on all inliers with a robust kernel.
• Coarse-to-fine pyramid alignment for direct methods; inverse compositional updates for speed.
• Mask moving objects — a dynamic region is an outlier population, not noise.
• Gain and bias compensation, or use NCC/ECC, whenever exposure varies between frames.
• Validate the model choice: if a homography’s inlier count collapses as the camera translates, the scene is not planar.

Failure modes and takeaways

1. Feature-based minimises reprojection; direct minimises photometric. Everything else about the two families follows from that.
2. Model selection is the highest-leverage decision. Simplest valid transform, always.
3. Normalise before DLT.
4. Algebraic solutions are initialisations, not answers — refine on a geometric error.
5. Inverse compositional makes direct alignment real-time by making the Hessian constant.
6. Mutual information is the cross-modal tool, because it assumes only statistical dependence.

Image stitching and fusion

Registration estimates a transformation. Stitching adds four problems registration never has to solve: which projection, which seam, which exposure, and how to hide the join.

The pipeline

1 · Pairwise matching

Features, matching, and RANSAC homographies between overlapping pairs. See registration.

2 · Topology and verification

Build a match graph; verify pairs probabilistically (an overlap is real if inliers \(n_i > \alpha + \beta n_f\) for the features in the overlap). Discard spurious edges and find connected components — one panorama per component.

3 · Global alignment

Bundle-adjust all camera rotations jointly, not pairwise. Pairwise chaining accumulates drift and fails to close the loop on a 360° sweep.

4 · Choose a projection surface

Planar, cylindrical or spherical, depending on field of view.

5 · Exposure compensation

Solve for per-image gains so overlaps agree photometrically.

6 · Seam selection

Cut through regions where images already agree.

7 · Blending

Feathering or multiband, to hide the residual step across the seam.

Why panoramas are a rotation problem

For a camera rotating about its optical centre, with no translation, the mapping between two views is exactly a homography that depends only on rotation:

No depth appears. This is why a panorama can be built with no 3D reconstruction at all — and why it breaks the instant the camera translates, since then parallax makes the required warp depend on scene depth.

Global alignment therefore optimises rotations (and optionally focal lengths) over all images at once:

Global bundle adjustment for panoramas

Rotations are parameterised in \(\mathfrak{so}(3)\) and updated on the manifold. One rotation must be fixed as the gauge (otherwise the whole solution can be rotated freely). A Huber \(\rho\) absorbs surviving mismatches.

Choosing the projection surface

The projection choice is forced by field of view A planar panorama cannot exceed 180° even in principle — points at 90° project to infinity. A full sweep requires either curved lines (cylindrical) or polar distortion (spherical). There is no projection that preserves straight lines and covers a hemisphere; this is a statement about differential geometry, not about software.

Exposure compensation

Auto-exposure and vignetting make overlapping images disagree in brightness. Solve for per-image gains \(g_i\) minimising the disagreement over overlaps, with a prior keeping gains near 1 (otherwise \(g_i \to 0\) is a trivial optimum):

• \(\bar I_{ij}\) — mean intensity of image \(i\) in the region overlapping \(j\)
• \(N_{ij}\) — number of overlapping pixels — weights larger overlaps more
• \(\sigma_N,\sigma_g\) — expected intensity error and expected gain deviation

Linear least squares in \(\{g_i\}\). Gain alone cannot fix vignetting, which is spatially varying — hence block-wise gain compensation or an explicit \(\cos^4\) vignetting model.

Seam selection

Rather than blending everywhere, cut where the images already agree. Formulate as a labelling problem: assign each output pixel a source image \(\ell(\mathbf{p})\), minimising

\(D\) penalises using a pixel outside an image’s valid region; the pairwise term penalises a seam where the two candidate images _disagree: \(V = \lVert I_{\ell(\mathbf{p})}(\mathbf{p}) - I_{\ell(\mathbf{q})}(\mathbf{p})\rVert + \lVert I_{\ell(\mathbf{p})}(\mathbf{q}) - I_{\ell(\mathbf{q})}(\mathbf{q})\rVert\). Solved by graph cut (\(\alpha\)-expansion), or by Dijkstra for a simple two-image strip._

Seam finding is how moving objects are handled A person who walked through the scene between two shots appears in one image and not the other. Blending averages them into a ghost; a well-placed seam routes around them so only one version survives. This is why seam selection precedes blending, and why “deghosting” is mostly a seam problem rather than a blending problem.

Blending

Feathering (alpha blending) Weight each image by distance to its valid-region boundary: \(I(\mathbf{p}) = \frac{\sum_k w_k(\mathbf{p})\,I_k(\mathbf{p})}{\sum_k w_k(\mathbf{p})}\)

Cheap. Hides exposure steps well, but blurs where alignment is imperfect: averaging misregistered detail destroys it.

Multiband (Laplacian pyramid) blending Blend each frequency band over a different spatial extent: \(L^{\ell}\_{\text{blend}} = \sum_k W^{\ell}\_k \cdot L^{\ell}\_k\) with the mask \(W\) blurred more at coarse levels \(\ell\).

Low frequencies blend over a wide band (hiding exposure differences), high frequencies over a narrow one (preserving sharpness). Reconstruct by collapsing the pyramid.

Multiband blending is the reason Laplacian pyramids still matter: the decomposition is invertible, so bands can be manipulated independently and the image reconstructed exactly. Gradient-domain (Poisson) blending goes further, solving \(\nabla^2 f = \nabla\!\cdot\!\mathbf{v}\) to match gradients rather than intensities, which removes visible seams entirely at higher cost.

Failure modes

Takeaways

1. A panorama is a rotation estimation problem. No depth appears — and that is exactly why translation breaks it.
2. Align globally, not pairwise. Chaining drifts and will not close a loop.
3. Field of view forces the projection surface; no surface preserves straight lines over a hemisphere.
4. Exposure compensation needs a prior on the gains, or the trivial all-black solution minimises the objective.
5. Seam finding, not blending, is how moving objects are handled.
6. Multiband blending works by giving each frequency its own blend width.

Semantic understanding

Classification, retrieval and recognition

The family that produced the backbones everything else uses — and the one where the split between _closed-set classification and open-set embedding matters far more than any architecture choice._

Two output types, two loss families

Closed set → a distribution over \(C\) classes Classification, multi-label recognition. Trained with cross-entropy family losses. Cannot express “none of these” — softmax always sums to one, which is why a closed-set classifier is confidently wrong on unseen categories.

Open set → an embedding \(\mathbf{f}\in\mathbb{R}^d\) Retrieval, face verification, person Re-ID, place recognition. Trained with metric-learning losses. Handles identities never seen in training, because the decision is a distance, not a class index.

Cross-entropy and its variants

Softmax cross-entropy

Label smoothing

Prevents the logit gap from diverging. Hard targets push \(z_y \to \infty\) relative to the others, producing over-confident, poorly calibrated models; smoothing caps the optimal gap at a finite value. Typically \(\varepsilon = 0.1\). It improves accuracy and calibration but _tightens class clusters, which measurably hurts downstream retrieval and distillation._

Focal loss

The modulating factor collapses the contribution of already-confident examples: at \(\gamma=2\), a sample with \(p_t = 0.9\) contributes \(100\times\) less than at \(\gamma = 0\). Designed for dense detection, where easy background dominates by orders of magnitude, but useful for any severe imbalance.

Class-balanced reweighting (effective number)

Weighting by \(1/n_c\) over-corrects, because samples within a class are correlated and redundant. The “effective number” \(\,(1-\beta^{n_c})/(1-\beta)\) interpolates between no reweighting (\(\beta=0\)) and inverse frequency (\(\beta\to1\)).

Metric-learning losses

Contrastive — an absolute criterion

Pulls positives to zero distance, pushes negatives beyond a margin \(m\). The weakness is that \(m\) is a global constant in a space whose scale is arbitrary.

Triplet — a relative criterion

Only the _ordering must hold: the positive must be closer than the negative by \(m\). Far less sensitive to the embedding’s global scale, which is why triplet largely displaced contrastive for retrieval._

Angular margin (ArcFace)

Normalise both weights and features onto a hypersphere of radius \(s\), then add an additive angular margin \(m\) to the target class before the softmax. CosFace instead subtracts a cosine margin, \(\cos\vartheta_y - m\). The key advantage over triplet: it needs no pair or triplet sampling — every sample compares against all class centres at once, which removes the entire mining problem.

InfoNCE — the self-supervised workhorse

A \((K{+}1)\)-way classification among one positive and \(K\) negatives. \(\tau\) controls how sharply the loss concentrates on hard negatives: small \(\tau\) makes it nearly a max over negatives. This is the objective behind MoCo, SimCLR and CLIP.

Sampling and hard-negative mining

With \(N\) samples there are \(O(N^3)\) triplets, almost all of which are already satisfied and contribute zero gradient. Mining is therefore not an optimisation — it is what makes training possible at all.

Semi-hard mining exists because hardest-negative mining fails The highest-loss examples are frequently mislabelled ones, so a scheme that up-weights the highest-loss samples also up-weights the label noise. The same tension is why robust losses and hard mining pull in opposite directions and must be balanced deliberately.

Augmentation: Mixup and CutMix

\(\textbf{Mixup:}\quad \tilde{\mathbf{x}} = \lambda\mathbf{x}\_a + (1-\lambda)\mathbf{x}\_b, \qquad \tilde{y} = \lambda y_a + (1-\lambda) y_b, \qquad \lambda\sim\mathrm{Beta}(\alpha,\alpha)\) \(\textbf{CutMix:}\quad \tilde{\mathbf{x}} = \mathbf{M}\odot\mathbf{x}\_a + (\mathbf{1}-\mathbf{M})\odot\mathbf{x}\_b, \qquad \tilde{y} = \lambda y_a + (1-\lambda)y_b,\quad \lambda = \frac{\lvert\mathbf{M}\rvert}{HW}\)

Mixup interpolates images and labels globally; CutMix pastes a rectangular patch and mixes targets _by area. CutMix keeps local statistics natural (no ghostly superpositions) and forces the model to use partial evidence, which is why it transfers better to localisation tasks._

Confidence calibration

A model is calibrated if, among predictions made with confidence \(p\), a fraction \(p\) are correct. Modern networks are systematically over-confident.

Expected Calibration Error

Bin predictions by confidence and measure the gap between accuracy and confidence in each bin.

Temperature scaling

A single scalar, fitted post hoc. It does not change the arg-max, so accuracy is unchanged, and it removes most of the miscalibration. It is the strongest baseline in the area and should be the default before anything more elaborate. But it does not survive distribution shift — a \(T\) fitted in-domain is wrong out-of-domain, which is the open problem.

Classical and deep pipelines

Classical Normalise → handcrafted features (colour histograms, LBP, HOG, SIFT) → encode (bag-of-visual-words, Fisher vectors, VLAD) → train an SVM or logistic regression. The encoding step aggregates local descriptors into a fixed-length vector — Fisher vectors do so via gradients of a GMM likelihood, and remained competitive until roughly 2012.

Deep Augment → encode with a CNN or ViT → pool (global average, or GeM \(\big(\frac1n\sum f^p\big)^{1/p}\) for retrieval) → classify or L2-normalise into an embedding → calibrate → optionally retrieve nearest neighbours with an ANN index.

Metrics

Failure modes and takeaways

1. Closed-set and open-set are genuinely different problems. Pick the output type before the architecture.
2. Relative criteria score above absolute ones — triplet over contrastive, for reasons of scale invariance.
3. Angular-margin losses removed the mining problem by comparing against class centres rather than sampled pairs.
4. The hardest examples are not the most useful. Aggressive mining amplifies label noise.
5. Calibration is orthogonal to accuracy, cheap to fix in-domain, and unsolved out-of-domain.
6. This family supplies everyone else’s backbone, which is why its advances propagate field-wide.

Object detection

The family where the decade’s clearest structural progression happened: dense enumeration → set prediction → open vocabulary. Three hand-designed components removed, one per stage.

The detection objective

The weights are a persistent source of irreproducibility: they interact with the label-assignment rule, the batch size and the image resolution, and are rarely transferable between datasets.

Anchors and box parameterisation

An anchor is a prior box of fixed scale and aspect ratio tiled over the feature map. The network regresses an offset rather than absolute coordinates, because offsets are scale-normalised and therefore learnable with a single set of weights across the image:

R-CNN box encoding

The \(\log\) on size makes the target symmetric — halving and doubling are equal-magnitude errors — and guarantees positive width and height after decoding. Subscript \(a\) denotes the anchor.

Smooth L1 (Huber on box offsets)

Quadratic near zero for stable gradients on good boxes, linear far away so a badly-placed anchor cannot dominate the batch.

Anchor-free alternatives predict the four distances from a point to the box sides (FCOS), or a pair of corner keypoints grouped by embedding (CornerNet), or a centre plus size (CenterNet). These remove the anchor hyperparameters — scales, ratios, and the IoU thresholds that assign them — but not duplicate suppression.

Label assignment — the quiet centre of the field

Label assignment fixes which predictions are responsible for which ground-truth object. It accounts for more of the COCO AP gained between 2019 and 2022 than the backbone does.

One-to-many versus one-to-one One-to-many assignment gives rich supervision and fast convergence, but produces duplicates that must be filtered. One-to-one produces no duplicates but supervises sparsely and converges slowly — the reason DETR needed 500 epochs. YOLOv10’s resolution is to train both heads with a consistent matching metric so they rank candidates identically, then discard the one-to-many head at inference. Set prediction reaches real-time latency this way.

Class imbalance and focal loss

A dense detector scores \(\sim10^5\) locations per image; the overwhelming majority are trivially negative. Summed, they dominate the gradient even though each term is individually small.

Later refinements make the classification target reflect localisation quality rather than a binary label: Quality Focal Loss regresses a soft target equal to the IoU with the matched box, and Varifocal Loss weights positives and negatives asymmetrically. Both address the same mismatch — the ranking used by NMS is a classification score, but the quantity that matters is box quality.

The IoU loss family

The flaw: if the boxes do not overlap, \(\mathrm{IoU} = 0\) regardless of how far apart they are, so the gradient is zero and the box receives no signal about which way to move.

GIoU — restores gradient on disjoint boxes

\(C\) is the smallest axis-aligned box enclosing both. The penalty term grows as the boxes separate, so \(\mathrm{GIoU}\in[-1,1]\) and the gradient never vanishes.

DIoU and CIoU — faster convergence, aspect-ratio awareness

\(\mathrm{DIoU} = \mathrm{IoU} - \frac{\rho^2(\mathbf{b},\mathbf{b}^{gt})}{c^2}, \qquad \mathrm{CIoU} = \mathrm{DIoU} - \alpha v\) \(v = \frac{4}{\pi^2}\left(\arctan\frac{w^{gt}}{h^{gt}} - \arctan\frac{w}{h}\right)^{2}, \qquad \alpha = \frac{v}{(1-\mathrm{IoU}) + v}\)

\(\rho\) is the centre distance and \(c\) the enclosing box’s diagonal. DIoU pulls centres together directly (GIoU can waste iterations enlarging the box instead); CIoU adds an aspect-ratio consistency term \(v\).

Why this family exists at all It is the clearest instance in the field of deliberately closing the gap between the surrogate loss and the evaluation metric. Smooth L1 optimises coordinates; the metric is IoU; the two disagree, especially across scales, because an equal coordinate error means very different IoU for a small and a large box. See objective–metric mismatch.

Non-maximum suppression, and its removal

Sort by score; greedily keep the top box; delete every box with \(\mathrm{IoU} > \tau\) against it; repeat.

Soft-NMS — decay instead of delete

Hard NMS deletes a genuinely distinct object that happens to overlap — the failure case for crowds. Soft-NMS decays its score instead, so a correct but overlapping detection can still survive if its score was high enough.

• Class-aware vs. class-agnostic: suppressing across classes removes duplicate detections of the same object under different labels, but also deletes genuinely overlapping objects of different classes.
• NMS is a latency cost as well as an accuracy choice: it is sequential and data-dependent, which is poor for accelerators. Removing it therefore has a deployment consequence as well as an accuracy one.
• The removal path: DETR (bipartite matching, 2020) → YOLOv10 (consistent dual assignment, 2024) → YOLO26 (natively one-to-one, 2025).

Set prediction

Bipartite matching

\(\hat{\sigma} = \arg\min*{\sigma\in\mathfrak{S}\_N}\sum*{i=1}^{N}\mathcal{L}_{\text{match}}\big(y_i, \hat{y}_{\sigma(i)}\big)\) \(\mathcal{L}_{\text{match}} = -\mathbb{1}_{\{c*i\neq\varnothing\}}\hat{p}*{\sigma(i)}(c*i) + \mathbb{1}*{\{c*i\neq\varnothing\}}\Big[\lambda*{L*1}\lVert b_i - \hat{b}*{\sigma(i)}\rVert*1 + \lambda*{\text{giou}}\mathcal{L}\_{\text{giou}}\Big]\)

Solved in \(O(N^3)\) by the Hungarian algorithm. Because the assignment is a bijection, two predictions cannot claim the same object, so duplicate suppression is trained into the loss. Ground truth is padded to \(N\) with \(\varnothing\).

Metrics

Average Precision

COCO’s headline metric averages AP over IoU thresholds \(0.50\!:\!0.05\!:\!0.95\) and over classes, and additionally reports \(\mathrm{AP}_S\), \(\mathrm{AP}_M\), \(\mathrm{AP}_L\) by object area. The size breakdown is the informative part: a single AP number aggregates over the small-object regime, where the methods differ most.

Tricks and failure modes

Tricks that reliably help

• FPN neck; multi-scale training and testing
• Mosaic and copy-paste augmentation; scale jitter
• EMA of weights; longer schedules with strong augmentation
• Tiled inference for small objects in large images
• Ignore regions for ambiguous or crowd-labelled areas

Failure modes

• Small objects — too few positive anchors; resolution lost to striding
• Crowds — NMS deletes true positives
• Score/quality mismatch — a confident but badly localised box is the one NMS keeps
• Closed vocabulary — everything unlisted is background
• Domain shift — anchor priors and thresholds are dataset-specific

Takeaways

1. Label assignment, not the backbone, drove most of the AP gains from 2019 to 2022.
2. Focal loss fixed a loss problem, not an architecture problem. One-stage detectors were never structurally inferior.
3. The IoU loss family exists to close the surrogate–metric gap, and each member fixes a specific gradient pathology of the previous one.
4. One-to-many converges fast but needs NMS; one-to-one needs no NMS but converges slowly. Consistent dual assignment resolves the dilemma.
5. NMS is a latency cost as well as an accuracy compromise.
6. Read AP by object size. The aggregate number hides where the methods differ.

Segmentation and matting

Four output types that look similar and are optimised very differently — and the one family where the evaluation metric was eventually adopted _as the loss._

Four tasks, four outputs

The four loss families

1 · Pixel-distribution losses

Decomposes over pixels, so it is easy to optimise and well-behaved — but it is blind to shape: a prediction missing a thin structure entirely may have a lower loss than one that gets it roughly right, because the thin structure is a handful of pixels.

2 · Region-overlap losses

Soft Dice

A differentiable relaxation of the Dice coefficient, using soft probabilities rather than a threshold. Self-normalising: the denominator scales with object size, so a tiny object contributes as much as a large one. This is why Dice is the default in medical imaging, where the target may occupy 0.1% of the volume.

Tversky — asymmetric control of FP vs FN

\(\alpha=\beta=0.5\) recovers Dice; \(\beta > \alpha\) penalises false negatives more, which matches screening applications where a miss costs more than a false alarm.

Lovász–Softmax

The _Lovász extension is the tight convex surrogate of a submodular set function — and the Jaccard (IoU) loss is submodular. So this is, in a precise sense, the correct convex relaxation of IoU rather than an ad-hoc one. Computed by sorting the errors and taking a dot product with the gradient of the extension._

3 · Boundary losses

Weights each prediction by its distance from the true boundary, so errors far from the contour cost more than errors adjacent to it. Overlap losses under-weight thin structures and boundary precision; this family compensates. Related: Hausdorff-distance losses and topology-aware (persistent-homology) losses that preserve connectivity.

4 · Compound and task-specific

In practice almost every strong system uses a compound: mask BCE + Dice is the standard for instance masks; CE + Lovász or CE + boundary for semantic segmentation. The rationale is that CE provides well-conditioned early gradients while the overlap term aligns the objective with the metric.

Panoptic quality

RQ is exactly the \(F_1\) score over segments. Matching at \(\mathrm{IoU} > 0.5\) is provably unique — at most one prediction can exceed half overlap with a given ground-truth segment — which is what makes the metric well-defined without a greedy tie-break.

Classical algorithms

• Thresholding & morphology — Otsu’s method picks the threshold maximising between-class variance. Followed by connected components and morphological opening/closing to clean up.

• Watershed — Treat intensity as topography and flood from markers. Over-segments badly without marker control — the classic remedy is marker-controlled watershed on a gradient image.

• Active contours / level sets — Evolve a curve to minimise an energy of image and smoothness terms. Level sets handle topology changes (splitting, merging) naturally by representing the curve implicitly.

Graph cuts and GrabCut

For two labels with a submodular pairwise term this is solved exactly by max-flow/min-cut. GrabCut iterates: fit GMM colour models to current foreground/background, run the cut, refit. The contrast-sensitive pairwise term encodes the prior that label boundaries coincide with intensity edges — the same prior a CRF imposes post hoc.

Dense CRF refinement

With Gaussian kernels over position and colour, mean-field inference is \(O(N)\) per iteration via the permutohedral lattice, making a _fully connected CRF over every pixel pair tractable. It was the standard post-process for DeepLab v1/v2, and was dropped once networks (atrous convolution, decoders, stronger backbones) learned sharp boundaries directly — a clean example of a post-processing prior being absorbed into the model._

Matting

The compositing equation

Severely under-determined: 3 equations (RGB) and 7 unknowns (\(\alpha\), \(F\), \(B\)) per pixel. Hence the need for a trimap marking definite foreground, definite background and the unknown band — or a strong learned prior.

Closed-form matting assumes \(F\) and \(B\) are locally constant, which makes \(\alpha\) locally linear in the image and yields a sparse “matting Laplacian” \(\mathbf{L}\); \(\alpha\) is then the solution of a sparse linear system with the trimap as boundary conditions. Deep matting adds:

• Alpha loss \(\lVert\hat\alpha - \alpha\rVert_1\) (or Charbonnier).
• Compositional loss \(\lVert \hat\alpha F + (1-\hat\alpha)B - I\rVert_1\) — enforces that the predicted matte actually reproduces the observed image.
• Gradient and Laplacian-pyramid losses for crisp hair and fibre detail.

Tricks

• Preserve rare classes in crop sampling. Uniform random crops remove the rare classes; sample crops conditioned on containing one.
• Ignore labels for ambiguous pixels (boundaries, void regions) so they contribute no gradient rather than a wrong one.
• Online hard-example mining or top-\(k\) loss over pixels.
• Auxiliary losses at intermediate resolutions (deep supervision) to stabilise deep decoders.
• Atrous / dilated convolution and pyramid pooling for context without resolution loss.
• Sliding-window inference with overlap at test time; average logits, not labels.
• Point-sampled loss (as in Mask2Former) instead of dense mask loss — a large memory saving at equal accuracy.

Failure modes and takeaways

1. Choose the loss family by what the metric rewards. Dice for imbalance, Lovász for IoU, boundary for thin structure.
2. Lovász is the principled IoU surrogate, not a heuristic — it is the tight convex extension of a submodular function.
3. Compound losses win because CE conditions the early gradients and the overlap term aligns with the metric.
4. PQ works because the \(\mathrm{IoU}>0.5\) match is unique.
5. CRFs encoded a prior the network later learned. A post-processing stage is a prior that has not been trained.
6. Matting is under-determined by construction — 3 equations, 7 unknowns — so it always needs an external constraint.

Keypoints and pose

Localising semantic points rather than corners — and the family where a single representational choice, heatmap versus coordinate, determined a decade of accuracy.

Corners versus semantic keypoints

A different problem from feature detection A Harris corner is defined by local image structure and is repeatable but anonymous — any corner will do. A semantic keypoint (“left elbow”, “nose tip”) is defined by meaning, may sit in a textureless region, and must be identified consistently across subjects and viewpoints. The first is found by an operator; the second must be learned.

Heatmaps versus direct regression

Gaussian heatmap target

\(v_k\) is a visibility flag — occluded joints must not be supervised, or the network learns to hallucinate them at the mean position. \(\sigma\) trades localisation precision against gradient coverage: too small and almost every pixel is background, too large and the peak is imprecise.

Why heatmaps replaced direct regression The output stays spatial: the network never has to convert a spatial activation into a number, a mapping convolutions are structurally poor at. Every pixel receives gradient, and multi-modal ambiguity is representable. The cost is quantisation to the heatmap grid and \(O(K \cdot HW)\) memory.

Why regression came back Direct coordinate regression is cheap, resolution-independent and end-to-end differentiable — which matters when pose feeds a downstream geometric loss. Soft-argmax closed most of the accuracy gap by making the two nearly equivalent.

Soft-argmax / integral regression

The expectation of position under the normalised heatmap. Differentiable, sub-pixel by construction, and it lets a coordinate loss be applied to a spatial representation — combining both advantages. \(\beta\) controls sharpness; as \(\beta\to\infty\) it approaches a hard arg-max. Caveat: an expectation is a poor summary of a bimodal heatmap, e.g. left/right ambiguity, where it returns a point between the two modes that is correct for neither.

Standard refinements: quarter-pixel offset correction using the local gradient of the heatmap around the peak; unbiased encoding–decoding (DARK) that corrects the systematic bias introduced by discretising the Gaussian.

Top-down versus bottom-up

Part Affinity Fields — grouping by line integral

\(\mathbf{L}_c\) is a 2-channel vector field for limb \(c\), pointing along the limb. The integral scores how well the field between two candidate joints agrees with the direction connecting them; bipartite matching on these scores assembles skeletons. The alternative, associative embedding, predicts a scalar tag per joint and groups by tag similarity — simpler, with no explicit limb model.

From 2D to 6-DoF: PnP

Given \(n \ge 3\) correspondences between known 3D model points \(\mathbf{X}_i\) and image points \(\mathbf{u}_i\), recover \((\mathbf{R},\mathbf{t})\):

P3P gives up to four solutions from three points — a fourth disambiguates. EPnP solves the general case in \(O(n)\) by expressing points in a basis of four virtual control points. In practice: P3P inside RANSAC for the hypothesis, then nonlinear refinement on all inliers. The minimal sample of 3 is what makes RANSAC cheap here.

6-DoF object pose and its metrics

ADD — average distance for distinguishable objects

ADD-S — for symmetric objects

A pose is counted correct if the distance is below 10% of the object diameter.

Symmetry is not a detail — it breaks the loss For a rotationally symmetric object (a bowl, a bottle, a bolt) many distinct rotations produce an identical image. A naive rotation loss punishes a correct prediction that happens to pick a different-but-equivalent representative, so the network is trained to be wrong and converges to the mean of the equivalent poses. The fix is to make the loss symmetry-aware — the \(\min\) in ADD-S — or to restrict the target to a canonical fundamental domain.

Rotation geodesic loss

The angle of the relative rotation — the true geodesic distance on \(SO(3)\), unlike an \(\ell_2\) distance between quaternions or matrix entries, which is not.

3D human pose

• MPJPE — mean per-joint position error after aligning the root joint. PA-MPJPE aligns by Procrustes (similarity), removing global rotation, translation and scale; it is the fairer number for monocular methods, which cannot recover scale.
• Lifting 2D to 3D is ambiguous up to depth reflection per joint. Priors that resolve it: bone-length constancy, joint-angle limits, temporal smoothness, and a learned pose prior (e.g. a body model such as SMPL).
• Reprojection loss \(\lVert \pi(\mathbf{X}_{3\text{D}}) - \mathbf{u}_{2\text{D}}\rVert\) allows weak supervision from 2D annotations alone — the same reprojection residual used in calibration and SfM.

Bone-length prior

OKS: the keypoint analogue of IoU

• \(d_i\) — distance between predicted and ground-truth keypoint \(i\)
• \(s\) — object scale (square root of segment area)
• \(\kappa_i\) — per-keypoint falloff constant, calibrated from human annotator variance
• \(v_i\) — visibility flag

The \(\kappa_i\) are the interesting part: they are measured from _inter-annotator disagreement, so the metric tolerates more error on joints humans themselves localise inconsistently (hips) than on precise ones (eyes). AP is then computed over OKS thresholds exactly as detection AP is over IoU._

Tricks and failure modes

Tricks

• Gaussian heatmap targets with visibility masking
• Soft-argmax plus quarter-pixel offset correction
• Flip testing — average predictions over a horizontal flip, remembering to swap left/right joint indices
• Multi-stage refinement with intermediate supervision
• RANSAC-PnP, then ICP refinement when depth is available
• Kinematic constraints and temporal smoothing

Failure modes

• Left/right confusion — the classic bimodal heatmap; flip augmentation without index swapping causes it
• Occluded joints hallucinated at the dataset mean
• Crowds — top-down crops contain two people; bottom-up grouping fails
• Symmetric objects — non-symmetry-aware loss trains toward the mean pose
• Truncation — joints outside the crop have no valid heatmap target
• Scale/depth ambiguity in monocular 3D — report PA-MPJPE

Takeaways

1. Keeping the output spatial is why heatmaps score above regression; soft-argmax then recovered regression’s advantages.
2. Soft-argmax is an expectation, so it fails silently on genuinely bimodal predictions.
3. Top-down buys accuracy with linear cost; bottom-up buys constant cost with grouping risk.
4. PnP is the bridge from 2D keypoints to 6-DoF pose, and its 3-point minimal sample makes RANSAC cheap.
5. Symmetry must be built into the loss, or the loss actively trains the wrong answer.
6. OKS calibrates tolerance to human annotator variance — a metric design idea worth borrowing elsewhere.

Text and document vision

A pipeline distinct enough to deserve separate treatment: the output is a _sequence, not a label or a box, and the alignment between image and sequence is unknown._

Why this differs from detection plus classification

The structural difference Detection outputs an unordered set; classification outputs one label. Text recognition outputs an ordered sequence of variable length, with no supervision about which image column produced which character. That missing alignment is the whole technical problem, and it is why CTC and attention decoders — borrowed from speech — are the core machinery here rather than anything from object detection.

Text detection

Text is not a generic object: instances are extreme-aspect-ratio, densely packed, arbitrarily oriented and often curved. Three representations, in order of generality:

Segmentation-based detectors face one specific problem: adjacent text lines merge into one blob. Two standard answers — shrink masks (predict a shrunken kernel per instance, then dilate to recover the full extent, as in PSENet/DBNet) and link prediction (predict whether neighbouring pixels belong to the same instance).

Differentiable binarization (DBNet)

A hard threshold is not differentiable, so the threshold map \(T\) cannot be learned. Replacing the step with a steep sigmoid (\(k\approx50\)) makes the binarization differentiable, so the network learns an _adaptive, per-pixel threshold jointly with the probability map \(P\). A small idea with a large effect on curved-text accuracy._

MSER remains relevant here: characters are, by construction, regions that stay stable over a range of thresholds, which is exactly what MSER detects.

Recognition I: CTC

The image is sliced into \(T\) frames left-to-right; the network emits a distribution over the alphabet plus a blank at each frame. CTC marginalises over every alignment consistent with the target string.

CTC objective

• \(\boldsymbol{\pi}\) — a frame-level path over alphabet \(\cup\ \{\text{blank}\}\)
• \(\mathcal{B}\) — the collapsing map: remove repeats, then remove blanks

The blank is what allows repeated characters: “ll” is emitted as l–blank–l, which survives collapsing, whereas l–l would collapse to a single “l”. The sum over exponentially many paths is computed in \(O(T\lvert\mathbf{y}\rvert)\) by the forward–backward algorithm.

CTC’s conditional-independence assumption CTC factorises \(p(\boldsymbol{\pi}\mid\mathbf{x}) = \prod_t p(\pi_t\mid\mathbf{x})\) — outputs are independent given the input. So the model has no internal language model and cannot use “q is followed by u” unless the visual evidence says so. This is why CTC decoding is usually combined with an external n-gram or neural language model via beam search, and why attention decoders outperform it on noisy text.

Recognition II: attention and transformers

Autoregressive: each character is conditioned on the previous ones, so the decoder learns an implicit language model. Strictly more expressive than CTC, at the cost of sequential decoding and a tendency to hallucinate fluent-but-wrong text when the image evidence is weak — the language prior overrides the pixels.

Attention drift is the characteristic failure: on long or low-quality text the alignment wanders and the decoder either repeats or truncates. Standard mitigations are a monotonic-alignment penalty, a CTC auxiliary head to anchor the alignment, and coverage penalties borrowed from machine translation.

Geometric rectification

Scene text is rarely fronto-parallel. Rectifying before recognition is worth several points, and is done with a learned thin-plate spline:

Thin-plate spline warp

_An affine part plus a sum of radial basis functions at \(K\) control points \(\mathbf{c}_i\). It is the interpolant that minimises bending energy \(\iint (f_{xx}^2 + 2f*{xy}^2 + f*{yy}^2)\). A localisation network predicts the control points, the grid is sampled differentiably, and the whole rectifier trains end-to-end with the recogniser (the STN/RARE recipe)._

For document images the analogous step is dewarping — undoing page curl and perspective, typically by predicting a dense backward-mapping flow field.

Document layout and structure

• Layout analysis — Detect and classify regions: title, paragraph, list, figure, caption, table, header/footer. Treated as ordinary object detection or instance segmentation, but with a strong reading-order prior.

• Table structure — Two sub-problems: detection (where is the table) and structure recognition (rows, columns, spanning cells). Output is a grid or an HTML/LaTeX string — so it is a sequence task again, with a tree-shaped target.

• Formula recognition — Image-to-LaTeX. Strictly a structured sequence task: the target is a token sequence whose validity is governed by a grammar, so decoding benefits from constrained beam search.

2D positional encoding for documents

The LayoutLM family’s core idea: a document token carries _where it is on the page as well as what it says. Reading order alone is insufficient — a two-column paper, a form, or an invoice is only interpretable with spatial layout, and 1D sequence position actively misleads._

OCR-free document understanding

The pipeline detect → recognise → layout → reason compounds errors: a missed character propagates to a wrong field value. OCR-free models (Donut, Pix2Struct, and modern VLMs) read the page image directly and emit structured output — JSON, an answer, a table — with no explicit text-detection stage.

Where this family currently stands Document understanding is the strongest capability of modern VLMs: Qwen2.5-VL reports 96.4% on DocVQA, and the same models exceed 90% on chart understanding. That sits in sharp contrast with their ~30–45% on fine-grained shape and pattern recognition. The pattern is consistent — these models read text in images extremely well and reason about geometry in them poorly. See the 2024–2025 consolidation.

Metrics

Failure modes and takeaways

1. The unknown image-to-sequence alignment is the core problem, and CTC and attention are two different answers to it.
2. CTC has no language model; attention has too much of one. Their failure modes are exact opposites.
3. Rectify before recognising — learned TPS warping is worth several points on scene text.
4. Documents need 2D position; reading order alone destroys the information that makes forms and tables interpretable.
5. OCR-free models remove error compounding and are now the strongest capability VLMs have.
6. Check the evaluation protocol — scene-text accuracy is usually case-insensitive and alphanumeric-only, which flatters it.

Motion and geometry

Optical flow and scene flow

One scalar equation per pixel, two unknowns per pixel. Everything in this family is a different way of supplying the missing constraint.

Brightness constancy

Assume a scene point keeps its intensity between frames:

A first-order Taylor expansion gives the optical flow constraint equation:

Optical flow constraint

• \(I_x, I_y\) — spatial image gradients
• \(I_t\) — temporal derivative
• \((u,v)\) — the unknown flow

Note what the linearisation costs: it is valid only for small displacements, since \(I\) must be locally well-approximated by its tangent plane. Large motion must be handled by warping or by an explicit correspondence search.

The aperture problem

One equation, two unknowns — per pixel The constraint determines only the flow component parallel to the gradient: \(v\_{\perp} = \frac{-I_t}{\lVert\nabla I\rVert}\) The component along an edge is completely unconstrained. Viewed through a small aperture, a moving edge gives no evidence of motion along its own direction.

This is the same statement as a rank-deficient structure tensor. Corner detection, trackability and flow observability are one question asked three ways.

Horn–Schunck: global smoothness

The Euler–Lagrange equations give a linear system solved by Jacobi iteration:

\(\bar u, \bar v\) are local averages. Each pixel’s flow is pulled toward its neighbours’ average and corrected along the gradient — information _propagates from textured regions into textureless ones. That propagation is what makes the underdetermined problem solvable, and it is why the quadratic smoothness term also blurs motion boundaries._

Lucas–Kanade: local constancy

Assume flow is constant in a window \(W\) and solve the resulting over-determined system by weighted least squares:

\(\mathbf{M}\) is exactly the structure tensor. It is invertible precisely where Shi–Tomasi says the patch is a good corner — which is why KLT tracks corners and not edges, and why “good features to track” is literally the name of that criterion.

Horn–Schunck vs. Lucas–Kanade is the dense-global vs. sparse-local split: HS regularises with a smoothness prior and yields flow everywhere; LK regularises by assuming local constancy and yields flow only where \(\mathbf{M}\) is well-conditioned.

Making the data term robust

Robust variational flow

Three upgrades over Horn–Schunck: (1) the residual uses the full warped image rather than its linearisation, so large motion is representable; (2) a gradient-constancy term adds invariance to additive brightness change; (3) \(\rho\) is a robust penalty — typically Charbonnier \(\sqrt{r^2+\epsilon^2}\) — on both data and smoothness, so occlusions and motion boundaries are not over-penalised.

The census transform goes further on illumination robustness: encode each pixel by the bit pattern of comparisons with its neighbours, and compare by Hamming distance. Only the local ordering of intensities matters, so any monotonic photometric change is absorbed.

Cost volumes and correlation

Correlating learned features rather than raw pixels makes the matching cost explicit and differentiable. A 2D flow search over displacements \(\mathbf{d}\) gives a 4D volume \(H\times W\times (2R{+}1)\times(2R{+}1)\) — expensive, which is why classical deep flow (FlowNet, PWC-Net) restricted \(\mathbf{d}\) to a small range and used a coarse-to-fine pyramid to cover large motion.

RAFT: all-pairs correlation plus a recurrent operator Coarse-to-fine has a structural flaw: a small, fast-moving object disappears at coarse levels, and the pyramid has already committed to the background motion before the fine level ever sees it.

RAFT builds all-pairs correlation once, at a single high resolution, pools it into a 4-level pyramid over displacement (not over image scale), and then iteratively refines a flow field with a shared GRU that looks up correlation at the current estimate: \(\mathbf{f}^{(k+1)} = \mathbf{f}^{(k)} + \Delta\mathbf{f}^{(k)}, \qquad \Delta\mathbf{f}^{(k)} = \mathrm{GRU}\big(\mathbf{h}^{(k)},\ \mathbf{C}(\mathbf{f}^{(k)}),\ \mathbf{f}^{(k)}\big)\) One operator applied many times instead of a cascade of different ones. The result was a large accuracy gain and unusually strong cross-dataset generalisation. The same structural move appears in SlowFast and in diffusion samplers.

Sequence loss over iterations

Every iterate is supervised, with exponentially increasing weight. This is what makes the recurrent operator converge rather than drift, and it is the same trick as DETR’s auxiliary per-layer decoder losses.

Occlusion and consistency

Forward–backward consistency check

Warp forward, then backward; the result returns to the starting point. Failure indicates occlusion or a mismatch. The threshold is made _relative to flow magnitude because absolute tolerance would reject all fast motion._

Occluded pixels have no correspondence by definition, so their photometric residual is meaningless. They must be excluded from the data term — this is the \(\mathcal{V}\) set in the objective template — via forward–backward checks, a predicted occlusion mask, or range checks for out-of-bounds warps.

Scene flow

Optical flow is the _projection of scene flow. Recovering 3D motion needs depth as well: from stereo (disparity at \(t\) and \(t{+}1\) plus flow), from RGB-D, or from a point cloud pair. For LiDAR the task is point-wise 3D displacement, evaluated with 3D endpoint error and “accuracy strict” (fraction of points within 5 cm or 5%)._

Metrics

Endpoint error

• Fl-all (KITTI): percentage of pixels with EPE \(> 3\) px and \(> 5\%\) of the ground-truth magnitude — an outlier rate rather than an average, which is far more informative for driving.
• Breakdowns matter: report EPE separately for all / noc (non-occluded) / occ, and for small vs. large displacement. A method can win on average while failing entirely on fast motion.

Failure modes and takeaways

1. One equation, two unknowns. Every method is a different missing constraint — global smoothness, local constancy, or a learned prior.
2. The aperture problem and corner detection are the same linear-algebra fact.
3. Linearisation is valid only for small motion; warping and explicit correspondence search are how large motion is recovered.
4. Cost volumes make matching explicit; RAFT showed the pyramid was the expendable part, not the correlation.
5. Occluded pixels must be excluded, not down-weighted. They have no correct answer.
6. Report outlier rates and breakdowns, as well as mean EPE.

Stereo and depth estimation

Depth is unobservable from a single view. Everything here is a way of supplying the missing dimension — a second camera, many cameras, motion, or a learned prior.

Geometry and the disparity relation

After rectification, corresponding points share a row and differ only horizontally:

• \(B\) — baseline — distance between optical centres
• \(f\) — focal length in pixels
• \(d\) — disparity

Why everything is parameterised in inverse depth

Depth uncertainty grows quadratically with range for fixed disparity noise. At 10× the distance, the same sub-pixel matching error produces 100× the depth error. Disparity (equivalently inverse depth) has roughly uniform uncertainty, so it is the right variable for estimation, for smoothness priors, and for probabilistic fusion. This is also why a stereo rig has a hard effective range set by \(fB/\sigma_d\).

Matching costs

Census dominates embedded stereo: it is illumination-robust, needs no multiplication, and reduces matching to XOR + popcount — the same hardware argument that favours binary descriptors.

Semi-Global Matching

The ideal is a 2D Markov random field over the disparity map, which is NP-hard. SGM approximates it by summing exact dynamic-programming solutions along several 1D paths:

SGM path cost

\(L*{\mathbf{r}}(\mathbf{p}, d) = C(\mathbf{p},d) + \min\begin{cases} L*{\mathbf{r}}(\mathbf{p}-\mathbf{r},\, d)\\ L*{\mathbf{r}}(\mathbf{p}-\mathbf{r},\, d\pm1) + P_1\\ \min*{k} L*{\mathbf{r}}(\mathbf{p}-\mathbf{r},\, k) + P_2 \end{cases} \;-\; \min_k L*{\mathbf{r}}(\mathbf{p}-\mathbf{r},k)\) \(S(\mathbf{p},d) = \sum*{\mathbf{r}} L*{\mathbf{r}}(\mathbf{p},d)\)

• \(P_1\) — small penalty for a \(\pm1\) disparity step — permits slanted surfaces
• \(P_2\) — large penalty for any bigger jump — permits genuine depth discontinuities
• \(\mathbf{r}\) — path direction; 8 or 16 directions are typical

The trailing subtraction keeps the accumulated cost bounded. The two-penalty design is the key idea: a single smoothness penalty must either forbid slant or permit noise, whereas \(P_1 \ll P_2\) allows gradual surfaces while still preserving object boundaries. \(P_2\) is usually adapted down where the image gradient is high.

The full classical pipeline: rectify → compute cost volume → aggregate (SGM) → winner-take-all → subpixel parabola fit → left–right consistency check → invalidate occlusions → median filter → hole-fill.

Subpixel refinement by parabola fit

Deep stereo

The modern architecture mirrors the classical pipeline, with each stage learned: shared feature extraction → cost volume by concatenation or correlation across the disparity range → 3D convolutional aggregation → differentiable arg-min.

Soft-argmin — differentiable disparity selection

A softmax over negated costs, then an expectation. Differentiable and sub-pixel — but, being an expectation, it produces a value _between the modes when the cost volume is genuinely bimodal, i.e. at occlusion boundaries and on repeated texture. The same caveat as soft-argmax for keypoints._

Supervised losses

Scale-invariant log loss (Eigen)

With \(\lambda = 1\) the loss is invariant to a global scale factor — exactly the ambiguity a monocular method cannot resolve. Penalising it would train the network to guess an unknowable constant. Setting \(\lambda \in (0,1)\) partially restores scale sensitivity when some absolute cue exists.

BerHu (reversed Huber)

L1 for small errors (fine detail), L2 for large ones (strong gradient where the estimate is badly wrong) — the opposite emphasis to Huber, and empirically better for depth regression.

Also standard: gradient-matching and surface-normal losses to sharpen boundaries, ordinal regression (treat depth as ordered bins — easier than regression and naturally handles the long-tailed depth distribution), and uncertainty-weighted likelihoods.

Self-supervised depth

The reprojection loss replaces ground-truth depth entirely. See photometric consistency for the full treatment; the essentials:

Edge-aware disparity smoothness

The mean-normalisation \(d/\bar{d}\) is not cosmetic: without it the smoothness term is trivially minimised by shrinking all disparities toward zero, so the network learns to predict everything as infinitely far away.

Monodepth2’s three refinements — all to the valid set, not the network

• Per-pixel minimum reprojection: \(\min_{s} pe(I_t, I_{s\to t})\) over source frames instead of the mean. A pixel occluded in one view is usually visible in another, and the minimum selects that view automatically — occlusion handling with no occlusion model.
• Auto-masking: keep a pixel only if \(\min_s pe(I_t, I_{s\to t}) < \min_s pe(I_t, I_s)\), i.e. the warped source explains the target better than the unwarped one. This discards pixels where the camera-motion assumption fails — a static camera, or an object moving at the camera’s velocity.
• Full-resolution multi-scale: upsample each scale’s disparity to full resolution before computing the loss, removing texture-copy artefacts and holes.

Multi-view stereo and monocular priors

MVS Sweep a reference frustum over depth hypotheses, warp source views by the induced homography at each plane, and build a cost volume: \(\mathbf{H}\_i(d) = \mathbf{K}\_i\mathbf{R}\_i\Big(\mathbf{I} - \frac{(\mathbf{t}\_i - \mathbf{t}\_r)\mathbf{n}^{\top}}{d}\Big)\mathbf{R}\_r^{\top}\mathbf{K}\_r^{-1}\) Regularise in 3D, regress depth, fuse across views with geometric consistency checks. Cascade/coarse-to-fine hypothesis refinement makes high resolution affordable.

Monocular depth There is no geometric constraint at all — the output is entirely a learned prior over scene structure. Hence it is relative, not metric, unless calibrated by a known scale. Depth Anything V2 showed the decisive factor is data: a synthetic-label teacher pseudo-labelling 62M real images beat far more expensive diffusion-based approaches by >10×.

Depth Anything 3 unified the whole family behind a single depth–ray target over any number of views, removing the distinction between monocular, stereo and MVS at the interface level.

Metrics

_\(\delta_1 = \delta_{1.25}\) is the standard threshold accuracy. For stereo, D1-all reports the percentage of pixels with disparity error \(>3\) px and \(>5\%\). Monocular results are usually reported after median scaling — the prediction is multiplied by \(\operatorname{med}(Z)/\operatorname{med}(\hat{Z})\) — which hides the scale ambiguity. Always check whether a number is scale-aligned before comparing it to a stereo method._

Failure modes and takeaways

1. Estimate in disparity / inverse depth. Uncertainty is uniform there and quadratic in depth.
2. SGM’s two penalties are what let one prior express both slant and discontinuity.
3. Soft-argmin is an expectation — it fails exactly where the cost volume is bimodal.
4. Scale-invariant losses exist because monocular scale is unknowable; penalising it trains a guess.
5. The self-supervised gains came from the valid set, not from the architecture.
6. Check for median scaling before comparing monocular and stereo numbers.

Camera pose, SfM, VO and SLAM

Estimating where the camera was and what the scene looks like, simultaneously — the most mature optimisation pipeline in vision, and the one a single 2025 transformer proved could be replaced by a forward pass.

Two-view geometry

The epipolar constraint

\(\mathbf{F}\) has 7 DoF (9 entries, minus scale, minus \(\det\mathbf{F}=0\)); \(\mathbf{E}\) has 5 (3 rotation + 2 translation direction — translation magnitude is unrecoverable, the origin of monocular scale ambiguity). The constraint says a point in one image lies on a _line in the other, reducing the correspondence search from 2D to 1D._

8-point algorithm Linear in the entries of \(\mathbf{F}\); stack 8 correspondences and take the smallest right singular vector, then project to rank 2 by zeroing the smallest singular value. Requires Hartley normalisation — unnormalised it is numerically useless.

5-point algorithm Uses the calibrated constraints (\(\mathbf{E}\) has two equal singular values, third zero) to work from the minimal 5 correspondences, yielding up to 10 solutions via a 10th-degree polynomial. The smaller minimal sample makes RANSAC dramatically cheaper — and it handles planar scenes, where the 8-point algorithm degenerates.

Pose from E, and the cheirality test

Four combinations; exactly one places triangulated points _in front of both cameras. That is the cheirality check, and it is what resolves the sign ambiguity._

Triangulation

The DLT form: rays never meet exactly, so solve in least squares via SVD. This minimises an _algebraic error; the statistically correct answer minimises reprojection error in both images (the optimal two-view method solves a degree-6 polynomial). Always refine. Triangulation is ill-conditioned at small parallax — two nearly parallel rays intersect at a poorly determined depth, which is why keyframe selection enforces a minimum baseline._

Bundle adjustment

The joint objective

_\(\mathcal{O}\) is the set of observations (camera \(i\) sees point \(j\)), \(\Sigma_{ij}\) the measurement covariance, \(\rho\) a robust kernel (Huber). This is the maximum-likelihood estimate under Gaussian pixel noise, and it is the gold standard against which every learned alternative is measured._

Exploiting structure — the Schur complement

_\(\mathbf{C}\) is block-diagonal with 3×3 blocks (one per point), so \(\mathbf{C}^{-1}\) is trivially parallel. The reduced system has dimension \(6\times N_{\text{cam}}\) instead of \(6N_{\text{cam}} + 3N_{\text{pts}}\) — typically thousands instead of millions. This one algebraic step is what makes BA tractable at all._

Gauge freedom The objective is invariant to a global similarity transform: rotate, translate and scale the whole reconstruction and every reprojection is unchanged. The Hessian is therefore rank-deficient by 7 (6 for a calibrated stereo/metric setup). Fix the gauge by holding one camera fixed, or add a weak prior, or use a pseudo-inverse — otherwise the solver drifts along the null space and the covariance is meaningless.

The two pipelines

Feature-based (indirect) Detect and match → essential matrix with RANSAC → recover pose, check cheirality → triangulate → register new views with PnP-RANSAC → add landmarks → local and global BA.

Minimises reprojection error. Robust to illumination change and large baselines; discards most of the image; needs texture.

Direct Select high-gradient pixels → warp by candidate depth and pose → minimise photometric error → marginalise old states → keyframes → local photometric BA.

Minimises photometric error. Uses all image gradient (works on faint texture), sub-pixel accurate; needs photometric calibration and a good initialisation.

The distinction is exactly the one from registration, one level up. Semi-direct methods (SVO) use direct alignment for tracking and feature-based BA for mapping, taking both advantages.

What SLAM adds over VO

Visual odometry is locally consistent and drifts without bound. SLAM adds the machinery to recover global consistency:

Map management

Keyframe insertion and culling; landmark creation, merging and removal; a covisibility graph linking keyframes that see common points.

Place recognition

Bag-of-words over visual vocabularies (DBoW) or learned global descriptors (NetVLAD). Must be fast and recall-oriented; candidates are verified geometrically afterwards.

Loop verification

A geometric check (enough RANSAC inliers for a consistent rigid transform) plus temporal consistency across several frames. A false positive here is catastrophic — it welds two unrelated places together and corrupts the whole map.

Pose-graph optimisation

Distribute the loop-closure error over the trajectory.

Pose-graph optimisation

• \(\mathbf{Z}_{ij}\) — measured relative pose between keyframes \(i\) and \(j\)
• \(\log(\cdot)^{\vee}\) — \(SE(3)\) logarithm to a 6-vector — the error is computed on the manifold
• \(\Omega_{ij}\) — information matrix of the constraint

Points have been marginalised out, so this is far cheaper than full BA. Run pose-graph optimisation immediately on loop closure to fix gross drift, then a full BA in a background thread to refine.

Visual–inertial fusion

An IMU supplies metric scale, gravity direction, and high-rate motion — precisely the quantities monocular vision cannot observe. The difficulty is rate mismatch (IMU at ~200 Hz, camera at ~30 Hz).

IMU preintegration

The key idea: integrate IMU measurements into a _relative motion constraint expressed in the body frame, so the result does not depend on the (still-unknown) absolute pose and need not be recomputed every time the optimiser updates the state. Bias updates are handled by a first-order correction rather than re-integration._

Observability: a visual–inertial system needs sufficient rotational and translational excitation to make scale and biases observable. Constant-velocity motion leaves accelerometer bias and scale entangled — which is why VIO initialisation requires the device to be moved.

The 2025 alternative: feed-forward geometry

VGGT and Depth Anything 3 > VGGT replaces the entire pipeline — features, matching, RANSAC, triangulation, BA — with a single forward pass of a transformer that alternates frame-wise and global attention and jointly predicts cameras, depth, point maps and tracks. Depth Anything 3 does the same with a plain backbone and a unified depth–ray target. Both beat the specialised optimisation pipelines on their benchmarks, and both are also useful as initialisers for a subsequent bundle adjustment — which is how they are most often deployed in practice.

Tricks and failure modes

Tricks

• 5-point RANSAC; ratio test plus geometric verification; cheirality
• Robust Huber/Tukey kernels; information-matrix weighting
• Keyframe selection on parallax and tracked-feature ratio
• Local sliding-window BA; marginalisation of old states into a prior
• Schur complement; sparse Cholesky ordering
• Constant-velocity motion prior for initialisation
• Rolling-shutter and photometric (gain/exposure) models

Failure modes

• Pure rotation — no parallax, so triangulation and scale fail entirely
• Planar / low-parallax scenes — essential matrix degenerate; use the homography branch
• Dynamic objects — corrupt pose unless masked
• Scale drift in monocular systems
• False loop closure — irrecoverable map corruption
• Repetitive environments (corridors, car parks) — place recognition aliasing
• Motion blur and rolling shutter under fast motion

Metrics and takeaways

ATE (absolute trajectory error) aligns the estimated and ground-truth trajectories with a similarity transform — Sim(3) for monocular, SE(3) when scale is observed — and reports RMSE of positions. RPE (relative pose error) measures drift over fixed sub-sequences and is the better diagnostic for odometry, since ATE can be dominated by a single early error.

1. The epipolar constraint reduces search from 2D to 1D. Everything two-view follows from it.
2. Translation magnitude is unrecoverable from two calibrated views — the root of monocular scale ambiguity.
3. Bundle adjustment is the MLE, and the Schur complement is what makes it computable.
4. Gauge freedom means the Hessian is singular by construction. Fix it deliberately.
5. VO drifts; SLAM closes loops. That single addition is the entire difference.
6. A false loop closure is worse than no loop closure — verify geometrically and temporally.

Visual tracking

Maintaining identity over time. The detection component is largely solved. _Association and lifecycle management account for most of the remaining error._

Four distinct problems

The Kalman filter

A constant-velocity motion model with state \(\mathbf{x} = [u, v, s, r, \dot{u}, \dot{v}, \dot{s}]^{\top}\) (centre, scale, aspect, velocities):

Predict

Update

The Kalman gain \(\mathbf{K}_k\) interpolates between prediction and measurement according to their relative uncertainties: confident prediction (\(\mathbf{P}\) small) ⇒ trust the model; noisy measurement (\(\mathbf{R}\) large) ⇒ trust the model. During an occlusion no measurement arrives, \(\mathbf{P}\) grows with each predict step, and the gate widens, which is the required behaviour and is where a filter differs from linear extrapolation.

A particle filter generalises this to non-Gaussian, multi-modal posteriors by representing the distribution with weighted samples — necessary when the state is genuinely ambiguous (e.g. in clutter), at much higher cost.

Data association

Given \(N\) tracks and \(M\) detections, build a cost matrix \(\mathbf{C}\in\mathbb{R}^{N\times M}\) and solve a linear assignment problem:

Assignment

Solved exactly in \(O(n^3)\) by the Hungarian algorithm. The constraint that each track takes at most one detection and vice versa is a _bipartite matching — structurally the same as DETR’s loss, applied across time instead of across queries._

Cost terms

Mahalanobis (motion) distance

_Distance normalised by the predicted covariance, so it is measured in standard deviations rather than pixels. Under a Gaussian assumption it is \(\chi^2\)-distributed with 4 DoF, giving a principled gate: reject pairs with \(d^{(1)} > \chi^2_{0.95} = 9.4877\)._

Appearance (cosine) distance and the combined cost

\(\mathcal{R}_i\) is a _gallery of the last ~100 appearance descriptors for track \(i\); taking the minimum over the gallery makes re-identification robust to temporary appearance change. Motion and appearance are complementary: motion is reliable short-term, appearance survives long occlusions._

Three trackers, three ideas

ByteTrack’s observation is worth isolating A low-confidence detection is frequently a real object that is occluded, not a false positive. Earlier trackers removed these at the first threshold, which produced identity switches at the moment of occlusion. Keeping them, and using them only in a second association pass where a track is already hypothesised, recovers most of that loss at negligible added cost. The gain here comes from removing a threshold, not from adding a model.

Track lifecycle

The unglamorous part that determines real-world behaviour:

Birth

An unmatched detection creates a tentative track. It must be confirmed by \(n_{\text{init}}\) consecutive matches (typically 3) before being output — this suppresses flickering false positives.

Confirmation

Only confirmed tracks are reported. A detector’s false positives rarely persist for three frames in a consistent location; genuine objects do.

Occlusion

Unmatched but confirmed tracks continue to be predicted, with growing covariance. They stay alive for up to \(a_{\max}\) frames (the “lost buffer”).

Termination

Deleted after \(a_{\max}\) unmatched frames. Too short causes ID switches through occlusion; too long causes ghost tracks and identity theft.

Interpolation

Offline only: fill short gaps after the fact, which measurably improves MOTA and is legitimate for post-hoc analysis but not for online use.

Metrics — and why MOTA is misleading

MOTA

Dominated by detection. FN and FP are typically orders of magnitude more numerous than identity switches, so MOTA measures association weakly: an accurate tracker on a weak detector scores below a weak tracker on an accurate detector. It can also be negative.

IDF1 — identity-centric

Computed after a _global bipartite matching between ground-truth and predicted trajectories, so it measures how consistently identity is preserved over the whole sequence rather than per frame._

HOTA — the decomposition

Explicitly factors detection accuracy from association accuracy and takes their geometric mean, so the two contribute equally and can be reported separately. Report HOTA with its DetA/AssA breakdown: of the three metrics it is the one that separates detector error from tracker error.

Tricks and failure modes

Tricks

• Gate before assigning — set impossible pairs to infinity so the Hungarian solver cannot choose them
• Matching cascade by track age
• Appearance galleries with EMA feature updates
• Camera-motion compensation — estimate a global homography between frames and warp track predictions; essential on moving platforms
• Two-threshold (ByteTrack) association
• Class-consistent association; separate Kalman parameters per class

Failure modes

• ID switches at crossings — two similar objects swap under mutual occlusion
• Fragmentation — long occlusion exceeds \(a_{\max}\)
• Identity theft — a lost track absorbs a new object’s detection
• Ghost tracks — coasting predictions persist after the object left
• Camera motion breaks the constant-velocity assumption
• Crowds — NMS in the detector deletes the occluded person the tracker needed

Takeaways

1. Association is a bipartite matching problem — the same structure as DETR’s loss, applied across time.
2. The Kalman covariance is doing real work: it widens the gate exactly when the object is occluded.
3. Mahalanobis distance gives a principled gate; pixel thresholds do not.
4. Motion and appearance are complementary, not redundant — short-term versus long-term.
5. ByteTrack removes a threshold. Low-confidence detections are frequently occluded objects.
6. Report HOTA with DetA/AssA. MOTA measures the detector more than the tracker.

Video understanding

_Adding a temporal axis multiplies compute and adds almost no labels. Every architecture here is a different answer to: _which temporal structure is worth paying for?__

The task family

Classical: dense trajectories

Before deep learning, the strongest approach was improved dense trajectories: sample points densely, track them with optical flow for \(L\approx15\) frames, and describe the tube around each trajectory with HOG (appearance), HOF (flow), and MBH — motion boundary histograms, the gradient of the flow field.

Differentiating the flow field cancels any _constant flow component — which is exactly camera motion. MBH is therefore camera-motion invariant by construction, and that is why it was the single strongest descriptor of the era. The same idea reappears in modern systems as explicit camera-motion compensation._

Deep architectures

Two-stream (2014)

One CNN on RGB frames (appearance), one on stacked optical flow (motion), fused late. It works because flow is a hand-computed motion feature the network would otherwise have to learn from limited data. Its weakness is cost: flow must be precomputed, and it accounts for most of the compute budget.

3D CNNs and inflation (I3D, 2017)

Replace \(k\times k\) kernels with \(t\times k\times k\). The inflation trick makes them trainable: take a 2D ImageNet-pretrained kernel \(\mathbf{W}\in\mathbb{R}^{k\times k}\) and initialise the 3D kernel as \(\mathbf{W}_{3D}[i] = \mathbf{W}/t\) for all \(i\), so a constant video reproduces the 2D network’s response exactly. Without this, 3D CNNs were hopeless — there is nowhere near enough labelled video to train them from scratch.

Factorised 3D (R(2+1)D, P3D)

Decompose \(t\times k\times k\) into a spatial \(1\times k\times k\) followed by a temporal \(t\times1\times1\). Fewer parameters, an extra non-linearity between the two, and easier optimisation — the same factorisation argument as depthwise separable convolutions.

SlowFast (2019)

Semantics and motion change at different rates, so sample them at different rates. A Slow pathway at low frame rate with high channel capacity captures what; a Fast pathway at \(\alpha\!=\!8\times\) the frame rate with \(\beta\!=\!1/8\) the channels captures how it moves; lateral connections fuse Fast into Slow. The thin fast pathway keeps total cost near a single 3D CNN — the same heavy/cheap asymmetry as mechanism 5.

Video transformers (2021)

Joint space-time attention costs \(O\big((THW)^2\big)\) — prohibitive. TimeSformer’s divided space-time attention factorises it: attend over space within a frame, then over time at the same spatial location, reducing cost to \(O\big(T\cdot(HW)^2 + HW\cdot T^2\big)\). ViViT and MViT explore related factorisations and pooling.

Self-supervised video: VideoMAE

Tube masking

Masking the _same spatial locations across all frames is essential. Random per-frame masking leaks the answer: a patch masked at \(t\) is usually visible at \(t{\pm}1\), and because adjacent frames are nearly identical, the model can reconstruct by copying rather than by understanding. Tube masking removes that shortcut, which is why the viable mask ratio is far higher than image MAE’s 75%._

The general lesson: temporal redundancy is both the opportunity and the trap in video self-supervision. Any pretext task must be designed so that the trivially-copyable solution is unavailable.

Temporal localisation

Structurally the 1D analogue of object detection, and it inherits the same machinery:

• Anchor-based: multi-scale temporal anchors, classification plus boundary regression, then temporal NMS.
• Boundary-based (BMN, BSN): predict start and end probabilities per frame, then score all start–end pairs with a boundary-matching confidence map.
• Query-based: DETR-style set prediction over intervals, with Hungarian matching — removing temporal NMS exactly as DETR removed spatial NMS.

The metric is mAP averaged over tIoU thresholds (typically 0.5:0.05:0.95 on ActivityNet, 0.3:0.1:0.7 on THUMOS). Boundaries are intrinsically ambiguous — human annotators disagree substantially about when an action starts — which caps achievable tIoU and should temper how seriously high-threshold numbers are read.

Anomaly detection

Anomalies are rare, diverse and poorly defined, so supervised classification is unworkable. Two standard formulations:

Multiple-instance ranking (weakly supervised)

Video-level labels only. The _highest-scoring segment of an anomalous video should outrank the highest-scoring segment of a normal one. The two regularisers impose temporal smoothness and sparsity — anomalies are brief._

The alternative is reconstruction- or prediction-based: train an autoencoder or future-frame predictor on normal video only, and score by reconstruction error. The known failure is that a sufficiently powerful autoencoder generalises to anomalies too and reconstructs them well — hence memory-augmented and constrained variants.

Sampling and aggregation

Video benchmark numbers are rarely comparable as published Accuracy depends on frame rate, clip length, the number of test views, input resolution and pretraining corpus — and these vary freely between papers. A “+1.5%” gain frequently reflects 30 test views against 10. Always check views × crops and the pretraining set before comparing.

Failure modes and takeaways

1. Inflation is what made 3D CNNs trainable — there was never enough labelled video to do it from scratch.
2. Factorisation is the recurring answer to temporal cost: (2+1)D convolution, divided space-time attention, SlowFast’s two rates.
3. MBH was camera-motion invariant by construction, which is why differentiating the flow field scores above describing it.
4. Temporal redundancy is a trap for self-supervision; tube masking exists to close the copying shortcut.
5. Check whether the task requires the temporal dimension. Many “video” benchmarks are solvable from one frame.
6. Test-time view counts make published numbers incomparable. Check the protocol first.

3D and generative

Point clouds and 3D perception

Unordered, irregular, non-uniformly dense data. The representation choice — points, voxels, pillars, range images, graphs — determines which operator is even available, and it is made before anything else.

Why point clouds need their own machinery

Three properties that break image operators

• Unordered. A set of \(N\) points has \(N!\) equivalent orderings. Any function of the cloud must be permutation invariant, which a convolution over an array is not.
• Irregular. There is no grid, so there is no neighbour-by-index. Neighbourhoods must be queried geometrically (kd-tree, ball query, hashing).
• Non-uniform density. LiDAR returns are dense near the sensor and sparse far away, so a fixed-radius neighbourhood contains wildly varying point counts.

Five representations

PointNet: permutation invariance done minimally

The universal symmetric form

Apply a shared MLP \(h\) to every point independently, aggregate with a symmetric function (element-wise max), then apply \(\gamma\). Max is permutation invariant by construction, and the paper proves this form can approximate any continuous set function to arbitrary accuracy given enough width.

What max-pooling computes, and its limitation Each output channel records the single point that maximised it, so the global feature is determined by a small set of “critical points” — effectively a learned skeleton of the shape. This gives strong robustness to missing or corrupted points, since deleting a non-critical point changes nothing.

It also means PointNet has no local neighbourhood structure at all: it sees each point in isolation and then globally. PointNet++ fixes this by applying PointNet hierarchically — sample centroids with farthest-point sampling, group by ball query, apply PointNet to each group, repeat — recovering exactly the multi-scale hierarchy that every family eventually rediscovers.

Farthest-point sampling

Greedily picks the point furthest from everything chosen so far. Gives far better spatial coverage than random sampling at \(O(NK)\) cost, and is what keeps thin structures represented after downsampling.

Sparse convolution

A dense 3D convolution over a \(1000^3\) voxel grid is impossible, and pointless: LiDAR occupies well under 1% of it. Sparse convolution stores only occupied voxels in a hash map and computes outputs only there.

• Submanifold sparse convolution restricts output to voxels that were already occupied. This is the critical variant: ordinary sparse convolution dilates the occupied set at every layer, so after a few layers the “sparse” tensor is dense and the advantage is gone.
• Regular sparse convolution is still needed where receptive field must genuinely grow — typically at downsampling layers only.

Registration: ICP and its variants

Point-to-point ICP

Alternate: (1) find correspondences \(\kappa\) by nearest neighbour, (2) solve for the transform in closed form. Step 2 has an exact solution via SVD of the cross-covariance (the Kabsch/Procrustes algorithm): centre both clouds, take \(\mathbf{H} = \sum \tilde{\mathbf{p}}_i\tilde{\mathbf{q}}_i^{\top} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top\), then \(\mathbf{R} = \mathbf{V}\operatorname{diag}(1,1,\det(\mathbf{V}\mathbf{U}^\top))\mathbf{U}^{\top}\) — the determinant term prevents a reflection.

Point-to-plane ICP — converges far faster

Penalise only the component of the residual _along the surface normal. Sliding along a flat surface is then free, which is exactly right — two planar patches genuinely do not constrain in-plane motion. Typically converges in an order of magnitude fewer iterations. Linearised for small angles, it becomes a \(6\times6\) linear system per iteration._

ICP is local — coarse alignment must come first ICP converges to the nearest local minimum of a highly non-convex cost, and it does so confidently. A global initialisation is mandatory: RANSAC over FPFH descriptor matches, 4PCS, or a learned registration front end. This is the same ordering rule as RANSAC before robust refinement — redescending objectives need a good starting point.

FPFH (Fast Point Feature Histogram) describes a point by histogramming the angular relations \((\alpha,\phi,\theta)\) between its normal and those of its neighbours, then convolving with neighbours’ histograms. It is rotation-invariant by construction, which is what makes descriptor-based global registration possible at all.

Set-to-set losses

Chamfer distance

Cheap (\(O(N\log N)\) with a kd-tree) and differentiable. Its weakness is systematic: it is minimised by clumping points in high-density regions, because nothing forces a one-to-one correspondence. Chamfer-optimal reconstructions look blurry and lose thin structure.

Earth Mover’s Distance

\(\phi\) is a _bijection, so every point must be used exactly once — which is precisely what prevents clumping. The cost is \(O(N^3)\) exactly, or \(O(N^2)\) with a Sinkhorn approximation, and it requires equal cardinality. EMD gives visibly better distributions; Chamfer is what most people can afford._

Eikonal regularizer for implicit surfaces

A true signed distance field satisfies \(\lVert\nabla f\rVert = 1\) everywhere. Without this term a learned implicit function has the right zero level set but meaningless values off-surface, which breaks sphere tracing, normal computation and any downstream geometric use.

3D object detection

Output is a 7-DoF box \((x,y,z,l,w,h,\vartheta)\) — yaw only, since objects on a ground plane do not roll or pitch. Three architectural families:

• Voxel-based — VoxelNet, SECOND: voxelise, sparse-convolve, detect in BEV. Accurate; the sparse convolution is the cost.

• Pillar-based — PointPillars: collapse the vertical axis into pillars, then use a plain 2D CNN. Substantially faster, and the standard real-time choice.

• Point-based — PointRCNN: generate proposals directly from points, refine in canonical coordinates. No quantisation loss; slower.

Yaw regression — avoiding the wrap-around

Regressing \(\vartheta\) directly is broken at the \(\pm\pi\) discontinuity, where a tiny angular error produces a huge loss. The \(\sin\) form is smooth and periodic, but cannot distinguish a box from its 180° flip — acceptable for cars (a symmetric box), handled elsewhere by a separate direction classifier.

Tricks and failure modes

Tricks

• Voxel downsampling to equalise density before anything else
• Ground removal by RANSAC plane fitting — removes most points and most false positives
• Normal estimation by local PCA (smallest eigenvector of the neighbourhood covariance)
• Ground-truth sampling augmentation: paste extra object point clouds into scenes — the single strongest augmentation for 3D detection
• Class-balanced voxel sampling; range-dependent neighbourhood radii
• Correspondence rejection by normal compatibility and distance ratio

Failure modes

• ICP converges to a confident wrong pose from poor initialisation
• Sparse distant objects — a car at 70 m may return ~10 points
• Symmetric and featureless geometry — a cylinder or flat wall does not constrain pose
• Chamfer clumping in reconstruction
• Reflective and transparent surfaces — no return, or a spurious one
• Sensor-specific overfitting — a model trained on 64-beam LiDAR fails on 32-beam

Takeaways

1. Permutation invariance is the defining constraint, and a symmetric pooling function is the minimal way to satisfy it.
2. PointNet has no locality; PointNet++ adds the hierarchy back. Multiscale structure is rediscovered here as everywhere.
3. Submanifold sparse convolution exists to stop sparsity dilating away.
4. Point-to-plane ICP converges faster because it does not penalise legitimate sliding.
5. ICP is local and must be initialised globally — FPFH + RANSAC, then refine.
6. Chamfer is affordable and biased; EMD is correct and expensive. The choice selects which artefact appears.

3D reconstruction and completion

Estimating continuous geometry rather than labels or boxes. Deliberately separate from 3D perception: that family predicts what and where, this one predicts _surface._

Choosing a surface representation

Implicit versus explicit — the trade that keeps recurring An implicit field is resolution-free, handles arbitrary topology, and is compact. An explicit mesh or point set is directly renderable, editable and measurable. The decade’s 3D story is largely a swing from explicit (TSDF, meshes) to implicit (NeRF, DeepSDF) and then partly back to explicit (3D Gaussian Splatting) — mechanism 6.

TSDF fusion

Per-voxel signed distance from one depth map

Positive in front of the surface, negative behind, truncated at \(\pm\mu\). Truncation matters: it confines each depth measurement’s influence to a band around the surface, so a measurement cannot assert anything about distant free space it never observed.

Weighted running fusion

_A running weighted average — the MLE under Gaussian depth noise. Weights encode measurement confidence (angle to the normal, range, edge proximity). Capping \(W\) at \(W_{\max}\) keeps the model adaptive to change instead of frozen by history: without the cap, early measurements dominate forever and the reconstruction cannot correct itself._

Dense voxel grids scale as \(O(n^3)\), so large scenes use voxel hashing — allocate blocks only near observed surfaces — or an octree. This is the same sparsity argument as sparse convolution.

Marching cubes

Extract the zero level set as a mesh. For each cube of 8 voxels, the sign pattern at the corners gives an 8-bit index into a table of 256 cases (15 up to symmetry), each specifying which triangles to emit. Vertices are placed on edges by linear interpolation:

This interpolation is what gives sub-voxel accuracy — without it the mesh is blocky at the grid resolution. The classic caveat is ambiguous cases: some sign patterns admit two valid triangulations, and choosing inconsistently between adjacent cubes creates holes. Marching Cubes 33 and the asymptotic decider resolve this properly.

Poisson surface reconstruction

Given oriented points \((\mathbf{p}_i, \mathbf{n}_i)\), treat the normals as samples of the gradient of an indicator function \(\chi\) (1 inside, 0 outside) and solve for \(\chi\):

\(\vec{V}\) is a smoothed vector field built from the oriented normals. A Poisson equation, solved on an adaptive octree with a multigrid solver; the surface is then the level set at the average value of \(\chi\) at the samples.

Why Poisson is the default for scanned data Because it is a global solve, it is inherently robust to noise — an erroneous normal is outvoted by its neighbours rather than producing a local spike, as it would in any local triangulation method. It also produces a watertight, manifold surface by construction.

The price: it hallucinates surface in unobserved regions, closing over genuine holes with plausible geometry. Screened Poisson adds a positional constraint pulling the surface to the points; the standard practice is to trim the output by sampling density so invented regions are removed. Normal orientation must be globally consistent — flipped normals produce inside-out surfaces, and consistent orientation is itself a non-trivial minimum-spanning-tree problem.

Neural implicit surfaces

DeepSDF-style objective

Clamping concentrates capacity near the surface, where accuracy matters, rather than spending it on distant free space. The Eikonal term enforces \(\lVert\nabla f\rVert = 1\), which is what makes \(f\) a genuine signed _distance rather than merely a function with the right zero set — necessary for sphere tracing, normals and offsetting._

Occupancy networks instead predict \(p(\text{occupied}\mid\mathbf{x})\) with a BCE loss; simpler to train, but the field carries no distance information, so extracting normals requires finite differences. Sampling strategy dominates results in both: uniform sampling wastes capacity, so samples are drawn near the surface with a decaying offset distribution.

Shape completion

Real scans are partial — self-occlusion, limited viewpoints, absorbing or specular materials. Completion fills the unobserved geometry.

Completion is generation, not measurement The back of an object was never observed. Any completion is a sample from a learned prior over plausible shapes, and the model will produce one confidently whether or not it is right. SAM 3D makes this explicit — it predicts complete geometry including occluded regions, with a 5:1 human-preference win rate, while the occluded part remains plausible rather than measured. For any metrology application, report observed and inferred geometry separately.

Metrics and validation

Accuracy, completeness, F-score

\(\mathrm{Acc} = \operatorname*{median}*{\mathbf{p}\in R}\ \min*{\mathbf{q}\in G}\lVert\mathbf{p}-\mathbf{q}\rVert, \qquad \mathrm{Comp} = \operatorname*{median}_{\mathbf{q}\in G}\ \min_{\mathbf{p}\in R}\lVert\mathbf{p}-\mathbf{q}\rVert\) \(F*\tau = \frac{2 P*\tau R*\tau}{P*\tau + R*\tau}, \qquad P*\tau = \frac{\lvert\{\mathbf{p}\in R : \min\_{\mathbf{q}}\lVert\mathbf{p}-\mathbf{q}\rVert < \tau\}\rvert}{\lvert R\rvert}\)

Accuracy penalises invented surface; completeness penalises missing surface. Report both — Poisson reconstruction trades one for the other directly, and a single Chamfer number hides which way. The F-score at a stated threshold \(\tau\) is the standard summary (Tanks&Temples, ETH3D).

Normal consistency \(\tfrac{1}{\lvert R\rvert}\sum \lvert \mathbf{n}_\mathbf{p}\cdot\mathbf{n}_{\mathrm{NN}(\mathbf{p})}\rvert\) measures whether the surface orientation is right, which Chamfer distance ignores entirely — two surfaces can be close yet differently oriented.

Geometric validation

• Manifoldness — every edge shared by exactly two faces; no non-manifold vertices.
• Watertightness — no boundary edges, so the mesh bounds a volume and can be measured.
• Orientation consistency — all face normals point outward; check via the sign of the signed volume.
• Self-intersection — must be absent for simulation, 3D printing or boolean operations.
• Euler characteristic \(V - E + F = 2 - 2g\) — a quick genus check that catches spurious handles from noise.

Failure modes and takeaways

1. Truncation is what makes TSDF sound — a depth measurement should not claim knowledge of far-away free space.
2. Cap the fusion weight, or the reconstruction can never correct itself.
3. Poisson is robust because it is global, and hallucinates for the same reason.
4. The Eikonal term is what makes an implicit field a distance field.
5. Completion is generation. Separate observed from inferred geometry when it matters.
6. Report accuracy and completeness separately, plus normal consistency — one number hides the trade.

Neural rendering and novel views

Make the renderer differentiable and the scene becomes an optimisation variable. The entire family follows from that one move.

The volume rendering integral

Continuous form

• \(\mathbf{r}(t)\) — the ray \(\mathbf{o} + t\mathbf{d}\)
• \(\sigma\) — volume density — differential probability of terminating at \(t\)
• \(\mathbf{c}\) — view-dependent emitted colour
• \(T(t)\) — accumulated transmittance: the probability the ray survives to \(t\)

Discrete quadrature — the computed form

_\(\delta_i = t_{i+1} - t*i\) is the sample spacing. This is exactly front-to-back alpha compositing — the same operation as classical volume rendering and as splatting. Every term is differentiable with respect to \(\sigma_i\) and \(\mathbf{c}_i\), so a photometric loss on rendered pixels propagates gradients into the scene representation.*

The whole field in one sentence Because rendering is differentiable, a photometric residual against the training views optimises the scene itself — no 3D supervision, no correspondences, no explicit reconstruction step. The scene is whatever explains the images.

NeRF

Both the coarse and fine networks are supervised. Note the architectural detail that enforces multi-view consistency: \(\sigma\) is predicted from \(\mathbf{x}\) alone, and only \(\mathbf{c}\) sees the view direction \(\mathbf{d}\). Letting density depend on direction would allow the model to cheat by inventing different geometry per view.

Positional encoding — why it is mandatory

MLPs have a spectral bias: they fit low frequencies first and high frequencies extremely slowly. Without \(\gamma\), NeRF produces blurry mush regardless of training time. Mapping coordinates to a Fourier basis makes high-frequency detail representable in a shallow network — equivalently, it turns the network’s effective kernel from a wide smooth one into a tunable-bandwidth one. \(L=10\) for position, \(L=4\) for direction.

Hierarchical sampling

Uniform sampling along a ray wastes almost all its samples in empty space. NeRF uses a coarse pass to build a piecewise-constant PDF from the coarse weights \(w_i = T_i\alpha_i\), then inverse-transform-samples the fine pass from it — concentrating computation where the ray is likely to terminate. Later work replaces this with occupancy grids (skip empty space entirely) and multiresolution hash encodings (Instant-NGP), which together cut training from days to seconds.

3D Gaussian Splatting

The primitive

The factorisation \(\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^\top\mathbf{R}^\top\) with a rotation \(\mathbf{R}\) (quaternion) and a diagonal scale \(\mathbf{S}\) guarantees \(\boldsymbol{\Sigma}\) stays positive semi-definite under unconstrained gradient descent. Optimising the six entries of \(\boldsymbol{\Sigma}\) directly would produce invalid covariances.

Projection to 2D

\(\mathbf{W}\) is the viewing transform and \(\mathbf{J}\) the Jacobian of the affine approximation to the perspective projection. A projected 3D Gaussian is a 2D Gaussian — which is exactly why the primitive was chosen, since it makes rasterisation closed-form.

Compositing — the same alpha blend as NeRF

Over depth-sorted Gaussians overlapping the tile. Identical in form to NeRF’s quadrature — the difference is that \(\alpha_i\) comes from evaluating an analytic 2D Gaussian rather than from querying an MLP, so there is no network in the inner loop.

Adaptive density control

The part that makes it work. Every ~100 iterations, examine the view-space positional gradient \(\lVert\nabla_{\boldsymbol{\mu}}\mathcal{L}\rVert\):

• Clone — large gradient, small Gaussian ⇒ the region is under-reconstructed; duplicate the Gaussian and move the copy along the gradient.
• Split — large gradient, large Gaussian ⇒ it is covering too much; replace with two smaller ones sampled from its own distribution, scaled by \(1/\phi\) (\(\phi\approx1.6\)).
• Prune — opacity below a threshold, or a Gaussian that has grown implausibly large in world or screen space.
• Opacity reset — periodically set all \(\alpha\) near zero and let them recover, which culls floaters that the photometric loss alone would keep.

The regularizers, and what each prevents

Why floaters exist at all With few views, a small semi-transparent blob placed just in front of a camera can explain that camera’s pixels perfectly while being invisible to the others. The photometric loss is fully satisfied. Nothing in the data rules it out — only a prior can, which is what the distortion and entropy terms supply. This is the clearest example in the site of regularization resolving genuine underdetermination rather than merely smoothing.

Practical machinery

• Appearance embeddings — a per-image latent absorbing exposure and white-balance variation, so the geometry is not forced to explain photometric inconsistency.
• Transient masks — per-image uncertainty fields that down-weight pedestrians and other movers in unstructured photo collections (NeRF-W).
• Camera pose refinement — jointly optimise \(\mathbf{T}_i\) with the scene; SfM poses are rarely accurate enough for sharp results.
• Background models — an inverted-sphere or NDC parameterisation for unbounded scenes (NeRF++, Mip-NeRF 360), since a bounded grid cannot represent the sky.
• Patch-based losses rather than isolated rays, so structural terms such as SSIM are computable.
• Coarse-to-fine frequency schedules — anneal in the high-frequency positional-encoding bands, which markedly improves pose-refinement stability.

Metrics and failure modes

Reported as PSNR / SSIM / LPIPS on held-out views. The same perception–distortion caveat applies: PSNR rewards blur, and a method can win on PSNR while looking worse. For geometry quality, report depth or mesh metrics separately — view synthesis quality does not imply correct geometry, and 3DGS in particular can look photorealistic while being locally wrong by centimetres.

Takeaways

1. Differentiable rendering turns the scene into an optimisation variable. That single move defines the field.
2. NeRF and 3DGS composite identically — front-to-back alpha blending. They differ only in where \(\alpha\) comes from.
3. Positional encoding is not a trick but a necessity, because MLPs are spectrally biased.
4. Density must not depend on view direction, or multi-view consistency is forfeited.
5. Adaptive density control is what makes splatting work; the primitive alone is not enough.
6. Regularizers here resolve genuine ambiguity. Floaters are consistent with the data, and only a prior excludes them.
7. Good renders do not imply good geometry. Evaluate them separately.

Image generation and translation

Learning the image _distribution, as opposed to reconstructing a particular scene. Kept separate deliberately: rendering is constrained by camera geometry, generation is not._

Four model families

VAE

Evidence lower bound

The reparameterisation trick \(\mathbf{z} = \boldsymbol{\mu}_\phi + \boldsymbol{\sigma}_\phi\odot\boldsymbol{\epsilon}\), \(\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I})\), moves the stochasticity outside the network so gradients flow through \(\boldsymbol{\mu}\) and \(\boldsymbol{\sigma}\). With a Gaussian likelihood the reconstruction term is an L2 loss, which is why plain VAEs are blurry — L2 produces the conditional mean of all plausible outputs.

VQ-VAE replaces the continuous latent with a learned discrete codebook, sidestepping posterior collapse and making the latent amenable to an autoregressive or diffusion prior. It is the reason latent diffusion has a usable latent space at all.

GANs

The minimax objective

At the optimal discriminator this minimises the Jensen–Shannon divergence between generated and real distributions. That is the source of the instability: if the two distributions have disjoint support — which is typical early in training, since natural images lie on a low-dimensional manifold — JS is constant at \(\log 2\) and its gradient is _zero. The generator receives no signal about which way to move._

Wasserstein GAN with gradient penalty

The Earth Mover’s distance is finite and differentiable even for disjoint supports, so gradients survive. The Kantorovich–Rubinstein duality requires \(D\) to be 1-Lipschitz; the gradient penalty enforces that softly on interpolates \(\hat{\mathbf{x}}\) between real and fake, which works far better than the original weight clipping.

The StyleGAN line then attacked quality rather than stability: progressive growing (train at \(4^2\), fade in resolution) made high-resolution training stable; StyleGAN added a mapping network \(z\to w\) and per-resolution AdaIN injection for a disentangled latent; StyleGAN2 diagnosed AdaIN-induced blob artefacts and replaced it with weight demodulation, plus a path-length regularizer that also made the generator far easier to invert. See the 2016–2018 era page.

Mode collapse, precisely Nothing in the minimax objective rewards coverage. A generator that produces one perfect image fools a discriminator just as well as one that covers the distribution, so the equilibrium is not unique and training can converge to a degenerate solution. Diffusion does not have this failure mode because its objective is a per-sample regression, not a game.

Diffusion

Forward process — fixed, no learning

\(q(\mathbf{x}_t\mid\mathbf{x}_{t-1}) = \mathcal{N}\big(\mathbf{x}_t;\ \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\ \beta*t\mathbf{I}\big)\) \(q(\mathbf{x}\_t\mid\mathbf{x}\_0) = \mathcal{N}\big(\mathbf{x}\_t;\ \sqrt{\bar\alpha_t}\,\mathbf{x}\_0,\ (1-\bar\alpha_t)\mathbf{I}\big), \qquad \bar\alpha_t = \prod*{s=1}^{t}(1-\beta_s)\)

The closed form for \(q(\mathbf{x}_t\mid\mathbf{x}_0)\) is what makes training cheap: any timestep is reached in one step, \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\boldsymbol{\epsilon}\), instead of simulating the chain.

Training objective — a plain regression

Predict the noise that was added. Stable, has a unique optimum, needs no discriminator, and is a reweighted ELBO. This objective is the whole reason diffusion displaced GANs — not sample quality per se, but that the optimisation is a well-posed regression.

Reverse (sampling) step

Classifier-free guidance

Train one model with random condition dropout, then _extrapolate away from the unconditional prediction at sampling time. One scalar \(s\) trades diversity for prompt adherence; \(s\approx7.5\) is typical. Universally adopted, and the single most important sampling-time knob._

Score-based view

Noise prediction _is score estimation up to scale. This unifies DDPM with score matching and with SDE/ODE formulations, and is what licenses deterministic samplers (DDIM, DPM-Solver) that integrate the probability-flow ODE in 10–50 steps instead of 1000._

Latent diffusion and DiT

Cross-attention conditioning

_The conditioning \(\mathbf{c}\) — text, layout, depth, semantic map — enters as keys and values while the noisy latent supplies queries. Because \(\tau_\theta\) is modality-agnostic, the same architecture accepts any conditioning signal, which is why one model family covers text-to-image, inpainting, super-resolution and layout-to-image._

DiT replaces the U-Net with a transformer over latent patches, conditioned through adaLN-Zero — per-block scale and shift regressed from the timestep embedding, initialised so each block starts as the identity (mechanism 1). Its finding: FID decreases smoothly with transformer Gflops, giving generation a scaling law and making video generation a compute question.

Flow matching and few-step sampling

Flow matching / rectified flow

Regress the _velocity field transporting noise to data along a prescribed — preferably straight — path. Simulation-free training, and straight paths need fewer integration steps than diffusion’s curved ones. Adopted as the training objective of SD3 and Flux._

Consistency models

Train a network mapping _any point on a probability-flow ODE trajectory directly to its origin, with a self-consistency constraint along the trajectory (\(\theta^-\) is an EMA target). Gives one- or two-step sampling. Two routes to the same goal: shorten the path (flow matching) or learn to skip along it (consistency)._

Image-to-image translation

CycleGAN — unpaired translation

With no paired data, an adversarial loss alone is satisfied by _any output in the target domain — the mapping is unconstrained. Cycle consistency forces \(G\) and \(F\) to be approximate inverses, which pins content while allowing style to change. A canonical instance of the consistency-constraint family. Its known pathology: the network can steganographically hide source information in imperceptible high-frequency noise to satisfy the cycle without preserving semantics._

Pix2Pix handles the paired case with a conditional GAN plus an L1 term, and uses a PatchGAN discriminator that classifies \(N\times N\) patches rather than whole images — restricting the discriminator to local texture, since L1 already handles low frequencies. ControlNet (see 2022–2023) is the modern answer: a frozen diffusion base plus a zero-initialised trainable branch accepting edges, depth, pose or segmentation.

Metrics

Fréchet Inception Distance

Fréchet distance between Gaussians fitted to Inception features of real and generated sets. Caveats that are routinely ignored: it is biased by sample count (compare only at equal \(N\), usually 50k); it depends on the Inception backbone and even the resizing implementation; and it cannot evaluate a single image.

• Precision / recall for generative models separate fidelity (samples lie on the data manifold) from coverage (the manifold is covered) — FID conflates them, so a mode-collapsed model can score deceptively well.
• CLIP score for prompt adherence; human preference remains the arbiter, since all automatic metrics correlate imperfectly.

Failure modes and takeaways

1. Diffusion replaced adversarial training because its objective is a stable regression, not because of any single sample-quality result.
2. GAN instability is a divergence problem — JS has no gradient on disjoint supports; Wasserstein does.
3. Noise prediction is score estimation, which is what licenses fast ODE samplers.
4. Classifier-free guidance is the key sampling knob, and it trades diversity for adherence.
5. Working in a latent space is what made generation affordable (mechanism 4).
6. FID is fragile. Equal sample counts, same backbone, and report precision/recall alongside.

Multimodal

Vision–language understanding

Language as an open label space. The family that removed the fixed class list — and imported a new set of failure modes with it.

The task family

Contrastive image–text pretraining

CLIP’s symmetric InfoNCE

Embeddings are L2-normalised, so \(\langle\cdot,\cdot\rangle\) is cosine similarity. \(\tau\) is _learned (as \(\log\tau\), clipped) rather than fixed. The softmax normalises over the whole batch, which is why CLIP needs very large batches — the negatives are the batch._

SigLIP — replacing softmax with sigmoid

Every pair is an independent binary classification, so there is no global normalisation over the batch. Consequences: memory no longer scales with \(N^2\) across devices, training works at modest batch sizes, and the learned bias \(b\) absorbs the severe positive/negative imbalance. This is why SigLIP largely replaced CLIP as a vision tower.

Zero-shot classification by prompt

Prompt ensembling — averaging the normalised embeddings of ~80 templates (“a bad photo of a {c}”, “a cropped photo of a {c}”, …) — reliably adds several points, because it marginalises over rendering style rather than committing to one phrasing.

Connector architectures

How do image tokens enter a language model? This is the main design axis of the entire generative-VLM literature.

Resolution is the recurring practical problem: a fixed \(224^2\) tower destroys small text and fine structure. Answers are AnyRes/tiling (split the image into crops plus a thumbnail) and native dynamic-resolution ViTs with window attention, as in Qwen2.5-VL, which preserve native resolution at bounded cost.

Grounding losses

Region–word alignment

Replaces a detector’s fixed classification head with a similarity against _text embeddings, which is what makes the vocabulary open: adding a class means adding a string. Grounding DINO fuses language at the neck, the query initialisation and the head rather than only at the output, which is why it handles arbitrary referring expressions and not just class names._

The composition Grounding DINO + SAM became the default open-vocabulary segmentation pipeline: one model names and localises, the other produces the mask. That is a direct consequence of SAM’s decision not to classify.

Hallucination and its control

Object hallucination is the characteristic failure A VLM describes objects that are not present, because the language model’s prior over plausible scenes overrides weak visual evidence — “a kitchen” strongly predicts “a refrigerator” whether or not one is visible. Measured by POPE, which asks balanced yes/no questions about object presence, including adversarially-chosen absent objects that co-occur frequently with present ones. Models score far worse on the adversarial split, which is the diagnostic that the failure is prior-driven rather than perceptual.

• Mitigations: contrastive decoding against a blurred or blank image (amplifying what the image contributes); grounding verification, where each claim must be localisable; retrieval augmentation; and explicit abstention training.
• Evaluation caution: CIDEr and BLEU reward fluent, generic captions and barely penalise a hallucinated object — so caption metrics systematically understate this failure.

Where the family stands

The gap is the interface, not the language model These models read text in images superbly and reason about geometry in them poorly — a 50-point spread between two capabilities that both require “understanding the image”. The cause is the training distribution and the vision–language interface: web captions describe objects and text, not angles, counts or spatial relations. Scaling the LLM does not fix it, which is why this is listed among the open problems.

Tricks and takeaways

• Large, diverse paired data accounts for more of the result than any other factor; curation and deduplication matter more than architecture.
• Hard-negative captions — minimally edited captions (swapped attributes, reordered relations) are what force compositional binding rather than bag-of-words matching.
• Frozen versus jointly trained vision tower is the most consequential architectural choice; freezing preserves the tower’s robustness, unfreezing adapts it to the LLM.
• Prompt ensembling and class-name templates for zero-shot.
• Two-stage training: align the connector first with everything else frozen, then instruction-tune.
• Feed OCR tokens alongside pixels when documents matter and the tower’s resolution is limited.

1. Language as label space removed the closed vocabulary — the decade’s third interface change.
2. SigLIP’s sigmoid removes the batch-size coupling that made CLIP expensive to train.
3. The connector is the design axis: token count, resolution handling, and whether the tower is frozen.
4. Grounding is contrastive alignment against text, which is what makes the vocabulary open.
5. Hallucination is the language prior overriding weak pixels, and caption metrics hide it.
6. These models read text accurately and estimate quantities poorly. Which side a task falls on determines whether they apply.

Scene reasoning and embodied vision

Where perception stops being the output and becomes an input to action — which changes what “good enough” means, and makes calibrated uncertainty a requirement.

Scene graphs and relationships

Factorised scene-graph likelihood

A graph \(G = (O, R)\) of objects and \(\langle\)subject, predicate, object\(\rangle\) triplets. Conditioning relations on _both object labels is what lets a strong language prior do most of the work — which is also the problem below._

Scene-graph benchmarks are dominated by predicate frequency “on”, “has” and “wearing” account for the great majority of annotated relations. A model that ignores the image and predicts the most frequent predicate compatible with the object pair scores high on Recall@K. This is why mean Recall@K (averaged over predicate classes) and zero-shot triplet recall are the meaningful numbers — an instance of the general evaluation problem where an aggregate metric is raised by exploiting the label prior.

Affordances

Not what an object is, but what can be done with it: graspable, sittable, pourable, pushable. The representation is usually a per-pixel or per-point heat map per action class, trained from interaction data or video of humans performing the action.

Why affordance is not a relabelling of segmentation Affordance is agent-relative and multi-label. The same surface is sittable for a human and not for a forklift; a mug handle is simultaneously graspable and not pourable while the rim is the reverse. A single-label semantic mask cannot express this, so the output must be a set of overlapping, agent-conditioned maps.

Visual navigation

Success weighted by Path Length

• \(S_i\) — 1 if the episode succeeded, else 0
• \(\ell_i\) — shortest-path length
• \(p_i\) — path taken

Success alone rewards exhaustive search — an agent that visits every room eventually finds the toilet. SPL penalises path length, so the metric measures _navigation and not coverage._

Active vision

The camera is no longer a passive observer: the agent chooses the next viewpoint. The natural criterion is expected information gain:

Next-best-view selection

Choose the action maximising the expected _reduction in entropy over the scene state \(\mathbf{s}\). In practice this is approximated by counting unobserved or uncertain voxels visible from a candidate pose. Active perception is the clearest case where calibrated uncertainty enters the decision: a miscalibrated model selects the wrong next view, as well as reporting the wrong confidence._

Sensorimotor prediction and world models

Latent dynamics

Learn dynamics in a compressed latent rather than in pixels — the same latent-space argument as diffusion. Planning then happens by rolling out the latent model, which is orders of magnitude cheaper than rendering.

“World model” is doing a lot of work in that phrase A model trained to predict future observations learns what video looks like, not what physics is. Sora was explicitly framed as a step toward general-purpose physical simulation, and that framing shaped a great deal of subsequent research — but object permanence, contact and causality remain emergent, partial and inconsistent. For any task where being wrong has physical consequences, treat generated futures as plausible samples, not predictions.

Uncertainty and safety

Decomposing predictive uncertainty

Aleatoric uncertainty is irreducible sensor and scene noise — more data will not help. Epistemic uncertainty is ignorance about the model, and _does shrink with data. The distinction is operational: high epistemic uncertainty means “collect more data or refuse to act”; high aleatoric means “this measurement will never be better, design around it”._

Heteroscedastic regression loss

Predict a variance as well as a mean. The first term down-weights samples the model declares uncertain; the \(\log\sigma^2\) term is what stops it declaring everything uncertain to avoid the first term. This is the objective template’s \(w_i\) made learnable.

What changes when perception feeds control

• Latency becomes correctness. A perfect result that arrives after the decision point is a failure.
• Error asymmetry is real. A missed obstacle and a phantom obstacle have utterly different costs, so a symmetric loss is the wrong objective.
• The tail is the product. Average accuracy is nearly irrelevant; the 99.99th percentile failure is what determines whether the system can be deployed.
• Abstention must be an available output. A closed-set softmax cannot say “I do not know”, which is precisely the answer safety requires. See calibration.
• Distribution shift is the norm, not an edge case — new sites, weather, lighting, and hardware revisions.

Where this family stands

A distinct model category emerged in 2025–2026 targeting embodied agents specifically: mixture-of-experts vision–language models activating a few billion parameters per token, evaluated on embodied perception, physical-world understanding and navigation rather than static VQA. Hy-Embodied-VLM-1.0 ranked first on 19 of 38 such benchmarks, with strong results on R2R-CE vision-and-language navigation and zero-shot object-goal navigation.

The framing shift being claimed is from “vision as a bolt-on adapter” — a pretrained LLM trunk with a vision encoder attached, the LLaVA/Qwen-VL generation — toward models whose trunk is itself a world model that predicts and acts. Whether that delivers is the open question of the current window; the documented geometric-reasoning gap is the main reason for scepticism, since navigation and manipulation are geometric tasks.

Failure modes and takeaways

1. Affordance is agent-relative and multi-label — not a relabelling of segmentation.
2. SPL exists because success alone rewards brute-force search.
3. Active vision makes calibrated uncertainty operationally necessary, not merely desirable.
4. Aleatoric and epistemic uncertainty imply different actions. Separate them.
5. Video prediction learns appearance, not physics.
6. When perception feeds control, the tail is the product — and abstention must be a permitted output.

Cross-cutting

Photometric consistency

One principle — warp and compare — supervises optical flow, stereo, direct visual odometry, self-supervised depth, multi-view stereo and neural rendering. It deserves its own treatment because it is not a formula but an ecosystem.

The warp-and-compare principle

Given a target image I_t, a source image I_s, geometry or depth D, camera intrinsics K and relative pose T_t→s, the source image is sampled at projected coordinates to synthesise the target:

Î_s→t(p) = I_s( π( K T_t→s D_t(p) K⁻¹ p̃ ) )

Back-project pixel p to 3D using its depth, transform into the source camera’s frame, project back to the image plane, and sample there. π is the projection; p̃ is p in homogeneous coordinates.

A basic masked photometric loss then compares the synthesised image against the real one:

ℒ_photo = [ Σ_p M(p) ρ( I_t(p) − Î_s→t(p) ) ] / [ Σ_p M(p) + ε ]

M is a validity mask; ρ is a robust penalty; ε prevents division by zero when the mask is empty. Note that the normalisation is by _mask weight, not pixel count — otherwise the loss rewards masking everything out._

Why this is the most reused idea in geometric vision The gradient of this loss flows into D and T — the geometry and the pose — without any ground-truth depth or pose ever being needed. Every self-supervised depth network, every direct SLAM system and every radiance field is, at bottom, minimising a version of this residual. Cross-view methods warp one view into another using intrinsics and extrinsics, then impose pixel, gradient, structural, depth–flow, view-synthesis or forward–backward consistency on the result.

The robust formulation that became standard

Raw squared error on pixel differences is a poor comparison: it is dominated by illumination changes and insensitive to structural mismatch. The widely used learned formulation mixes L1 with SSIM:

pe(I_a, I_b) = (α/2) ( 1 − SSIM(I_a, I_b) ) + (1 − α) ‖ I_a − I_b ‖₁

Typically α = 0.85. L1 retains luminance and colour fidelity; SSIM compares _local structure and is far more tolerant of smooth brightness variation. Almost always paired with edge-aware depth smoothness._

Unsupervised multi-view stereo commonly extends this further, combining pixel and gradient photometric terms with SSIM and depth smoothness — the gradient term adding invariance to additive brightness offsets that neither L1 nor SSIM fully removes.

What photometric consistency assumes

Six assumptions, each of which is violated routinely in real scenes. The list is the set of causes to check when a depth or flow system fails without an evident reason.

• 1 · Same physical surface — The compared pixels observe the same point in the world. Violated at occlusion boundaries and by independently moving objects.

• 2 · Lambertian reflectance — The surface looks the same from both viewpoints, or the appearance change is explicitly modelled. Violated by specularities, transparency and retro-reflectors.

• 3 · Comparable radiometry — Exposure, white balance, vignetting and response curves are comparable or compensated. Violated by auto-exposure between frames — extremely common.

• 4 · Accurate geometry and calibration — The warp is only as good as K, D and T. Calibration error produces a systematic residual that the optimiser will absorb into wrong geometry.

• 5 · Differentiable, valid resampling — Sampling must be differentiable and must not cross invalid boundaries. Bilinear interpolation supplies both the value and a usable gradient.

• 6 · Bad regions are handled — Occluded, disoccluded, saturated, reflective, transparent or dynamic regions are masked or robustly down-weighted — not silently averaged in.

Traditional and modern tricks

This is the ecosystem. Each entry exists because one of the six assumptions fails in a specific, predictable way.

Monodepth2: three tricks that changed self-supervised depth One paper crystallised three of the entries above, and the combination is still the default recipe:

• Per-pixel minimum reprojection. With several source frames, take the minimum residual per pixel rather than the mean. A pixel occluded in one view is usually visible in another, and the minimum picks that view automatically — no explicit occlusion model needed.
• Auto-masking. Discard pixels where the unwarped source image matches the target at least as well as the warped one. Those are exactly the pixels for which the camera-motion assumption fails: a static camera, or an object moving at the camera’s velocity.
• Full-resolution multiscale reprojection. Compute the loss at full resolution for every scale rather than at each scale’s own resolution, which removes texture-copy artefacts and holes in low-texture regions.

The pattern is worth noting: all three are visibility and validity refinements — improvements to 𝒱 and w_i in the objective template, not to the network or the penalty.

Where photometric consistency shows up in the timeline

• Optical flow — brightness constancy is the photometric residual in its simplest form; Horn–Schunck and Lucas–Kanade differ only in the prior used to make it well-posed. See the atlas.
• Self-supervised depth — the reprojection loss replaces ground-truth depth entirely. This is what made monocular depth trainable at scale.
• Direct visual odometry and SLAM — minimise photometric residuals over pose directly, rather than reprojection residuals over matched features. See the atlas.
• NeRF and 3D Gaussian Splatting — the rendering loss is a photometric residual against the training views. The renderer is differentiable precisely so this gradient exists. See the atlas.
• Multi-view stereo — photometric consistency across views is the classical matching cost and the modern self-supervised objective alike.

The limitation that has not been solved Photometric consistency fails on reflection, transparency, saturation and dynamics — and it fails silently, producing confident wrong geometry rather than an error. Every trick in the table above mitigates a symptom; none removes the underlying assumption that appearance is a reliable proxy for correspondence. This is the single largest reason learned geometry is not yet survey-grade.

Losses and regularization

A consolidated reference for the objective vocabulary used across every other module — organised by _what the loss assumes, since that is what determines when it fails._

Regression losses

The single most useful fact in this module > L2 produces the conditional mean, L1 the conditional median. When the target is genuinely ambiguous — the texture behind an occluder, the appearance of a super-resolved edge — the mean of all plausible answers is a blur that is itself implausible. That is why L2-trained restoration, VAEs and future-frame prediction are blurry, and it is not fixable by more capacity. It is a property of the objective.

Classification losses

\(\mathcal{L}_{\text{CE}} = -\sum_c y_c\log p_c, \qquad \mathcal{L}_{\text{BCE}} = -\big[y\log p + (1-y)\log(1-p)\big]\) \(\mathcal{L}\_{\text{focal}} = -\alpha_t(1-p_t)^{\gamma}\log p_t, \qquad y^{\text{LS}}\_c = (1-\varepsilon)y_c + \varepsilon/C\)

Cross-entropy vs. BCE is a modelling choice, not a preference: softmax CE assumes classes are mutually exclusive; per-class BCE does not, and is therefore correct for multi-label problems and for detection heads where “several things overlap here” is legitimate.

See classification for label smoothing and class-balanced reweighting, and detection for the focal family’s quality-aware successors.

Overlap and boundary losses

\(\mathcal{L}_{\text{Dice}} = 1 - \frac{2\sum \hat{p}y + \epsilon}{\sum \hat{p}^2 + \sum y^2 + \epsilon}, \qquad \mathcal{L}_{\text{Tversky}} = 1 - \frac{TP}{TP + \alpha FP + \beta FN}\) \(\mathcal{L}_{\text{IoU}} = 1 - \mathrm{IoU}, \qquad \mathcal{L}_{\text{GIoU}} = 1 - \mathrm{IoU} + \frac{\lvert C\setminus(B\cup B^{gt})\rvert}{\lvert C\rvert}\)

These exist to close the surrogate–metric gap. Full treatments: segmentation’s four families and the IoU loss family. The Lovász–Softmax is the principled member — the tight convex extension of the submodular Jaccard loss, rather than a heuristic relaxation.

Distributional and adversarial losses

This table explains GAN instability entirely: the standard GAN minimises JS, and early in training the generated and real distributions have disjoint support, so there is no gradient. See generation.

Knowledge distillation

The \(T^2\) factor keeps the gradient magnitude of the soft term comparable to the hard term as \(T\) varies (soft-target gradients scale as \(1/T^2\)). The “dark knowledge” is in the _relative probabilities of the wrong classes, which is why temperature matters._

Regularizers

Reading a regularizer as a claim about the world Every entry asserts something. TV asserts the scene is piecewise constant — true for cartoons and depth maps, false for gradients, which is why TV causes staircasing. Edge-aware smoothness asserts depth discontinuities align with intensity edges — false for a low-contrast object boundary. When a regularizer produces a characteristic artefact, that artefact is the prior showing through.

Multi-task weighting

A compound objective \(\mathcal{L} = \sum_k \lambda_k\mathcal{L}_k\) raises the question of how to set \(\lambda_k\) when the terms have incommensurable units — pixels, radians, logits, metres.

Uncertainty weighting (Kendall & Gal)

Learn a per-task noise scale \(\sigma_k\) jointly with the model. The first term down-weights tasks the model finds noisy; the \(\log\sigma_k\) term prevents the degenerate solution of sending all \(\sigma_k\to\infty\). In practice parameterise \(s_k = \log\sigma_k^2\) for stability. This turns a hand-tuned hyperparameter search into an optimisation.

• GradNorm balances the gradient magnitudes of each task rather than the loss values, which is closer to the actual concern.
• Gradient surgery (PCGrad) projects away the component of one task’s gradient that conflicts with another’s — addressing the case where tasks genuinely disagree rather than merely differ in scale.

Uncertainty

Aleatoric — learned observation noise

Epistemic — via an ensemble

Deep ensembles score above MC-dropout and most variational approximations as an estimator of epistemic uncertainty, at \(M\times\) the cost. The decomposition matters operationally: see embodied vision.

Training objective versus evaluation metric

The gap is a design decision, usually unexamined Metrics are frequently non-differentiable, non-decomposable over samples, or defined only over a whole dataset — so a surrogate is unavoidable. But the choice of surrogate encodes assumptions that should be stated. The decade’s most reliable progress came from papers that closed a specific gap deliberately: focal loss for imbalance, GIoU for disjoint boxes, PQ for panoptic, Lovász for IoU. See evaluation and failure analysis.

Takeaways

1. Every loss implies a noise model. Choosing L2 is asserting Gaussian residuals, which real vision data almost never has.
2. L2 gives the mean, L1 the median. Blur under ambiguity is a property of the objective, not the network.
3. Softmax CE asserts mutual exclusivity. Use BCE when that is false.
4. The minimised divergence determines the failure mode — mode-seeking against mode-covering, gradient or no gradient.
5. Every regularizer is a claim about the world, and its characteristic artefact is that claim being wrong.
6. Multi-task weights can be learned rather than tuned, via observation noise.
7. Name the surrogate–metric gap explicitly. Several of the decade’s results come from reducing it.

Cross-cutting mechanisms and failure modes

The techniques that appear in every task family, and the twelve ways vision systems fail. This is the fifth layer of the five-layer treatment — the one that decides whether a method survives contact with real data.

Robust estimation

Ordinary L2 assumes approximately Gaussian residuals and gives large influence to outliers. Real vision residuals are contaminated by mismatches, occlusion, moving objects, saturation, specularities and label errors — always, not occasionally.

The ordering matters RANSAC first, then robust refinement. A robust penalty has a limited basin of attraction: it can suppress outliers once the estimate is roughly right, but it cannot rescue an initialisation that is grossly wrong, because from there the true inliers look like the outliers. This is the same reason coarse global registration precedes ICP.

Multiscale processing

Pyramids capture large motion, objects at multiple sizes, broad context, and coarse geometry before fine detail. The concept is close to universal — and worth recognising in its many costumes:

• Classical — Gaussian and Laplacian pyramids, wavelets, scale space, coarse-to-fine warping.

• Convolutional — Feature Pyramid Networks, U-Net skip connections, atrous/dilated convolution, pyramid pooling.

• Modern — Coarse-to-fine depth volumes, multiresolution hash grids, hierarchical transformers with patch merging.

The recurring trade is the same in all of them: coarse levels enlarge the optimisation basin and supply context, but destroy small or fast-moving structure. RAFT’s refusal to use a pyramid at all — all-pairs correlation at one resolution, refined iteratively — is the clearest statement that the trade is not always worth taking.

Consistency constraints

Seven relations that let a system check itself without labels. Each is a free supervisory signal derived from redundancy, and together they are the backbone of self-supervised vision.

Consistency is where self-supervision comes from Every row is a statement that two independently computed estimates of the same quantity must agree. Disagreement is a residual; a residual is a training signal. This is the mechanism behind self-supervised depth (rows 1, 2, 5), contrastive and distillation pretraining (row 7), and unsupervised tracking (rows 2, 3, 4) — and it is why the supervision ladder could be climbed at all.

Data and augmentation

Geometry-aware augmentation must transform labels consistently Boxes, masks, keypoints, depth, normals, flow, camera matrices and poses cannot be treated like class labels. A horizontal flip negates the x-component of flow and swaps left/right body keypoints; a crop changes the principal point; a resize changes focal length in pixels. Each of these is a silent correctness bug that produces a model that trains fine and is subtly wrong.

Photometric augmentation can improve robustness but may violate the exact assumptions used by self-supervised photometric losses. If correspondence is the supervision, then colour transforms must be applied identically to both views, or applied analytically so the loss can compensate. Randomly jittering the colour of one view of a stereo pair destroys the very signal being learned from.

Sampling and imbalance

Dense visual problems contain enormous numbers of easy background pixels, anchors or pairs. Without intervention, the loss is dominated by examples that are already correct.

Loss-side remedies

• Focal loss — explicitly designed so easy negatives do not dominate dense one-stage detection
• Class weighting and class-balanced reweighting
• Top-k loss; online hard-example mining

Sampling-side remedies

• Balanced crops; foreground oversampling
• Hard-negative mining
• Stratified ray sampling (neural rendering)
• Uncertainty-guided sampling

Post-processing

Post-processing usually embodies a strong prior that the network was not asked to learn. Reading the list this way explains both why these steps work and why removing them was hard:

• NMS enforces sparse detections — the prior that one object yields one box.
• CRFs align labels to image boundaries — the prior that class changes coincide with intensity edges.
• Connected components enforce object coherence.
• Morphology fills or removes structures below a scale.
• Median filtering rejects impulse errors.
• Left–right checks reject stereo occlusions.
• Bundle adjustment imposes global geometric agreement.

The decade-long arc of removing them Every one of these is a hand-designed component with dataset-specific thresholds. DETR showed that NMS could be replaced by putting the sparsity prior into the loss via bipartite matching; YOLOv10 and YOLO26 apply the same construction at real-time latency. The pattern recurs: a post-processing step is a prior you have not yet found a way to train.

Failure-mode checklist

Twelve failure sources, their symptoms and their defenses. Most debugging of a vision system consists of identifying which row applies.

Note the shape of the table Rows 1–8 are violations of image-formation assumptions — physics, not statistics. No amount of additional training data fixes a specularity or a scale ambiguity, because the information is genuinely absent from the measurement. Rows 9–12 are violations of statistical assumptions, and those are addressable with data and loss design. Diagnosing which half applies determines whether the fix is a better sensor setup or a better training set.

Evaluation and failure analysis

The module that matters most and is taught least. By 2026 the field’s hardest open problems are measurement problems, not capability problems.

Metrics by task family

The pattern in the right-hand column Almost every entry hides the same thing: a distribution collapsed into a mean. The aggregate rewards performance on the common, easy majority and is nearly insensitive to the rare, hard minority, which is frequently the case of interest. The response is the same in each row: report the breakdown, by object size, class frequency, occlusion state, displacement magnitude or domain.

Dataset leakage and biased splits

“Zero-shot” is under-specified when the corpus is undisclosed With web-scale pretraining, “zero-shot” means unseen during fine-tuning, not unseen. Since the pretraining corpus is typically undisclosed and unsearchable, contamination cannot be ruled out — and it grew steadily through the decade as corpora grew. Treat cross-domain transfer benchmarks (RF100-VL, VTAB) as more informative than any single zero-shot number.

Calibration

Calibration is orthogonal to accuracy — a model can be accurate and badly calibrated, or vice versa — so it must be reported separately. Temperature scaling fixes most in-domain miscalibration with one scalar and does not change accuracy. It does not survive distribution shift, which is the unsolved part.

Domain shift and robustness

Kinds of shift

• Covariate shift — \(p(\mathbf{x})\) changes, \(p(y\mid\mathbf{x})\) does not: new camera, weather, lighting
• Label shift — class frequencies change
• Concept shift — \(p(y\mid\mathbf{x})\) changes: the grading standard itself moves
• Open-set — genuinely new categories appear

Standard robustness suites

• ImageNet-V2 — a fresh test set, same protocol: accuracy drops even with no corruption
• ImageNet-C — 15 corruptions × 5 severities; report mCE
• ImageNet-A — natural adversarial examples
• ImageNet-R / Sketch — rendition and abstraction shift
• ObjectNet — controlled viewpoint, rotation and background

Effective robustness

Out-of-distribution accuracy is strongly predicted by in-distribution accuracy along a consistent line \(f\) across a huge range of models. Effective robustness is the residual above that line, which separates a shift in the line from a move along it. Very few interventions produce positive \(\rho\); CLIP-style pretraining was one of the notable exceptions.

Label noise

A fraction of every large dataset is mislabelled — ImageNet’s validation set has a measurable error rate, and segmentation boundary labels are systematically imprecise. Consequences that matter:

• Hard-example mining amplifies it. The highest-loss samples are disproportionately the mislabelled ones — which is exactly why semi-hard mining exists and hardest-negative mining collapses.
• Memorisation is late. Networks fit clean patterns first and memorise noise later, so early stopping is itself a noise-robustness method.
• Benchmark ceilings are noise ceilings. Once model error approaches annotator disagreement, further “improvement” is fitting one annotator’s idiosyncrasies.
• Defenses: soft or smoothed labels, robust losses (symmetric CE, generalised CE), sample reweighting, co-teaching, and explicit ignore zones at ambiguous boundaries.

A metric-design idea worth copying > OKS calibrates its per-keypoint tolerance \(\kappa_i\) from measured inter-annotator variance — so it demands precision on joints humans localise consistently and forgives error on those they do not. Very few metrics do this, and most would be better if they did.

Ablation discipline

The confound that recurred all decade Architecture papers changed the training recipe at the same time as the architecture. ConvNeXt quantified it: +2.7 points from the recipe alone, before any architectural change. EfficientNet’s scaling rule turned out to matter less than its NAS-found baseline.

“Method + baseline + recipe” is one confounded unit unless deliberately separated. Minimum standard: hold the recipe fixed, hold the pretraining corpus fixed, report compute-matched comparisons, and report variance over seeds.

• Report seed variance. A 0.3-point gain within seed noise is not a gain.
• Compute-match, and state throughput on real hardware — FLOPs are a poor proxy, as ConvNeXt’s ~49% throughput advantage over Swin at similar FLOPs shows.
• Controlled diagnostics localise error where aggregate scores do not: synthetic probes with a known answer, sensitivity sweeps over one nuisance factor, and counterfactual inputs (blur the object, remove the context, blank the image).
• Test the null hypothesis. Measure what a single frame scores on the video task, what predicate frequency alone scores on the scene graph, and what the caption prior alone scores on VQA.

Objective–metric mismatch

The recurring structural problem, treated at length in losses and regularization. Symptoms to watch for: validation loss improving while the metric stagnates; a method scoring higher on PSNR while rated lower by observers; AP rising from better label assignment rather than better perception.

Where the decade’s most reliable progress came from Papers that closed one specific surrogate–metric gap deliberately: focal loss for dense imbalance, GIoU for disjoint boxes, PQ for joint stuff-and-things evaluation, Lovász for IoU, HOTA for separating detection from association. Each replaced a proxy that had quietly stopped measuring the thing of interest.

Safety-critical evaluation

• The tail is the product. Mean accuracy is nearly irrelevant; the 99.9th-percentile failure determines deployability.
• Error costs are asymmetric. A missed obstacle and a phantom obstacle have wholly different consequences, so a symmetric metric misranks methods.
• Latency is correctness. A right answer after the decision point is a wrong answer.
• Abstention must be possible. A closed-set softmax cannot say “I do not know”.
• Report runtime, memory, energy and latency alongside accuracy — a model that cannot be deployed has no accuracy in practice.
• Auditability. “The model said so” is not a defensible position in a liability chain; keep the inputs, version and confidence.

Takeaways

1. Aggregate metrics hide the distribution that matters. Always report the breakdown.
2. Split by the unit of correlation — sequence, identity, site — never by the individual sample.
3. Calibration is orthogonal to accuracy, fixable in-domain, and unsolved under shift.
4. Effective robustness separates a shift in the accuracy line from a move along it.
5. Hard-example mining amplifies label noise; benchmark ceilings are annotator-disagreement ceilings.
6. Recipe and architecture are confounded unless deliberately separated.
7. Capability outran evaluation. The hardest open problems in 2026 are measurement problems.

Key innovations

Depth, detection and the first believable images

One of the most productive periods in the field’s history: deeper backbones, mature two-stage and one-stage detectors, the rise of instance segmentation, efficient mobile architectures, and rapid progress in GANs.

What was open, and what this era settled

Open problems entering the window

• Depth degradation — deeper networks were empirically worse, and not because of overfitting
• Detection was accurate or fast, never both
• Dense prediction had no canonical architecture
• GAN training was unstable above 128×128

Vocabulary this era established

• Residual / identity shortcut connections
• Region proposal networks, RoIAlign
• Feature pyramids and lateral connections
• Depthwise separable convolutions, inverted residuals
• Focal loss; anchor-free keypoint detection
• Progressive training for high-resolution synthesis

Every component introduced here is present, largely unmodified, in a 2026 production stack.

ResNet: the degradation problem

CVPR 2016 · He, Zhang, Ren & Sun — Microsoft Research · arXiv:1512.03385 · Mechanism 1

The diagnosis is what makes the paper. A deeper model can always represent a shallower one by setting the extra blocks to identity, so it should never be worse — yet SGD did not find that solution. The failure is in optimisation, not capacity, and batch normalization had already ruled out naive vanishing gradients.

The residual reformulation makes the identity the default state of a block.

y = ℱ(x, {W_i}) + x

• W_i → 0 gives an exact identity at zero cost, so adding layers can no longer hurt.
• The gradient with respect to x contains an unattenuated term flowing through the shortcut.
• ResNet-152: 4.49% top-5 single model; 3.57% as an ensemble. Swept ILSVRC-2015 classification, detection and localization, plus COCO detection and segmentation.

The 2016 follow-up, Identity Mappings in Deep Residual Networks (arXiv:1603.05027), analysed propagation through the block and showed that pre-activation ordering (BN → ReLU → conv) makes the shortcut a genuinely clean identity path — enabling a stable 1001-layer network, reported at 4.62% error on CIFAR-10.

Why this is the single most reused idea on the site A branch initialised to the identity recurs outside depth. It is why ControlNet’s zero-initialised convolutions can add conditioning to a frozen diffusion model without degrading it; why DiT’s adaLN-Zero initialises each block as identity; and why LoRA adapters are safe to attach to a pretrained model. Same mechanism, three different decades of use.

The backbone zoo, 2015–2018

ResNet was the pivot, but the window produced a whole family of structural ideas, most of which factorise a dense operation into a cheaper structured one and spend the savings on depth or width.

Squeeze-and-excitation factorises nothing. It rescales channels by a global statistic, and that operation continues in transformer-era block design as channel attention.

MobileNet: the efficiency lineage

2017 / CVPR 2018 · Howard et al.; Sandler et al. — Google · arXiv:1704.04861 · arXiv:1801.04381

A k×k convolution over C_in → C_out costs k²·C_in·C_out per position. Factor it into a depthwise k²·C_in plus a pointwise C_in·C_out and the ratio becomes 1/C_out + 1/k² — roughly a 8–9× reduction for k = 3.

• V1 added a width multiplier and a resolution multiplier, which define an accuracy/latency curve for one architecture, now the standard way to release a family of models.
• V2 introduced the inverted residual and the linear bottleneck: no ReLU on the projection, because ReLU applied in a low-dimensional space destroys manifold information. ~3.4M parameters at ~72% top-1.

Dense prediction finds its architecture

Classification backbones destroy spatial resolution on purpose — stride is how they buy receptive field cheaply. Every dense-prediction architecture is a different answer to getting resolution back.

• FCN (2015) — Replace the classifier’s fully-connected layers with 1×1 convolutions; upsample with learned deconvolution; fuse coarse and fine strides with skips. The template for everything that follows.

• U-Net (2015) — Symmetric encoder–decoder with concatenative skips at every resolution, trained on very few images with heavy elastic augmentation. Still the default diffusion backbone a decade later.

• DeepLab v1–v3+ (2015–18) — Atrous convolution enlarges the receptive field without losing resolution or adding parameters; ASPP samples several dilation rates in parallel; v3+ adds a decoder and atrous separable convolutions.

Feature Pyramid Networks

CVPR 2017 · Lin, Dollár, Girshick, He, Hariharan & Belongie · arXiv:1612.03144

The backbone already computes a pyramid. Its levels differ in semantic level: fine levels carry localisation and little semantics, coarse levels the reverse. FPN takes the deep coarse maps, upsamples 2×, and adds them laterally to the 1×1-projected shallow maps, repeating down the pyramid. The result is strong semantics at every resolution for marginal extra cost, and it became a default neck for detection, instance segmentation and panoptic models alike.

Faster R-CNN and Mask R-CNN

The R-CNN → Fast R-CNN → Faster R-CNN lineage is one long exercise in absorbing hand-built stages into the network. The RPN figure and its analysis are in the task atlas. Faster R-CNN reached ~0.2 s/image at 73.2% mAP on VOC 07+12, and made detection end-to-end trainable for the first time.

ICCV 2017 · He, Gkioxari, Dollár & Girshick — FAIR · arXiv:1703.06870

Mask R-CNN

Two design details carry most of the gain:

• Decoupled mask and class prediction. Predicting one mask per class, and selecting by the classification head, avoids competition between classes in the mask output.
• RoIAlign. RoIPool quantises RoI boundaries to the feature grid twice over — harmless for boxes, fatal for pixel-accurate masks. RoIAlign samples at exact fractional locations with bilinear interpolation, and is worth several mask AP on its own.

Mask R-CNN swept all three COCO 2017 tracks — instance segmentation, bounding-box detection and person keypoint detection — beating the 2016 challenge winners by about 2 AP with no additional tricks, at roughly 5 fps. Its single-model result reached 47.9 box AP and 42.6 mask AP.

Cascade R-CNN (2018) then chained multiple detection heads at progressively increasing IoU thresholds, fixing the mismatch between the IoU used in training and the IoU demanded at evaluation, for 2–4% mAP in high-precision regimes.

One-stage detection: YOLO, SSD, RetinaNet

CVPR 2016 · Redmon, Divvala, Girshick & Farhadi · arXiv:1506.02640

YOLO — detection as one regression

• ~45 fps at 63.4% VOC mAP; a “Fast YOLO” variant reached 155 fps.
• Global context per prediction produced roughly half the background false positives of Fast R-CNN.
• The weakness is structural: one cell with a limited box budget localises small, clustered objects poorly.

ECCV 2016 · Liu, Anguelov, Erhan, Szegedy, Reed, Fu & Berg · arXiv:1512.02325

SSD — multi-scale single-shot detection

SSD300 reached 74.3% mAP at 59 fps — faster than Faster R-CNN and more accurate than YOLOv1, which is exactly the gap it was designed to close.

ICCV 2017 · Lin, Goyal, Girshick, He & Dollár — FAIR · arXiv:1708.02002

Focal loss — why one-stage detectors were losing

FL(p_t) = − α_t (1 − p_t)^γ log(p_t)

A one-stage detector scores ~10⁵ candidate locations per image, overwhelmingly easy background. Summed over that many terms, the easy negatives account for most of the gradient although each contributes little. At γ = 2, a p_t = 0.9 example contributes 100× less. RetinaNet with a ResNet-101-FPN backbone reached 39.1% COCO AP — the first one-stage detector to match two-stage accuracy. The architecture was never the problem; the loss was.

Detection scoreboard, 2015–2018

What every row still shares Anchors (or a grid), a confidence threshold, and NMS — three hand-designed components with dataset-specific hyperparameters. CornerNet is the first crack: predicting paired top-left and bottom-right corner keypoints and grouping them by embedding removes anchors, but not duplicate suppression. Removing both takes until DETR in 2020, and reaching real-time while doing so takes until 2025.

Generative models reach photorealism

ICLR 2018 · Karras, Aila, Laine & Lehtinen — NVIDIA · arXiv:1710.10196

Progressive Growing of GANs

This made high-resolution training roughly an order of magnitude more stable and produced the first genuinely photorealistic 1024×1024 GAN faces, alongside the CelebA-HQ dataset built for the purpose.

CVPR 2019 (arXiv Dec 2018) · Karras, Laine & Aila — NVIDIA · arXiv:1812.04948

StyleGAN

Replaced direct latent-vector injection with an eight-layer mapping network feeding an intermediate latent space, then injected style codes at every resolution via adaptive instance normalization (AdaIN), plus per-pixel noise for stochastic detail. The mapping network’s purpose is to escape the entanglement forced by sampling z from a fixed Gaussian — the intermediate space w need not match any prescribed distribution. The result was a disentangled, style-mixable latent space; FID improved from ProGAN’s 8.04 to 4.40 on the newly released FFHQ dataset.

WGAN and WGAN-GP (2017) address the same instability through the objective, in place of the architecture, replacing the standard adversarial loss with an Earth Mover’s (Wasserstein) distance enforced by weight clipping or a gradient penalty — yielding more reliable dynamics and, unusually for GANs, interpretable loss curves. BigGAN (2018) pulled a different lever again: scaling parameter count, batch size and dataset breadth for class-conditional generation across all 1,000 ImageNet categories rather than one narrow domain.

Era recap

Solved

• Depth — residual reformulation
• Scale — feature pyramids, atrous convolution
• Resolution recovery — FCN/U-Net skip connections
• Instance masks — a parallel branch plus RoIAlign
• Class imbalance — focal loss
• High-resolution GAN training — progressive growing

Still structurally broken

• Everything requires dense human labels
• Fixed, closed class vocabulary
• Anchors, NMS and thresholds — hand-tuned per dataset
• No representation shared across tasks
• Geometry and recognition share no machinery

Mechanism 1 is now established. The next era addresses the first item on the right, the source of the supervision.

Scale, self-supervision and set prediction

A transitional phase: efficient CNN design matured, self-supervised learning closed the gap with supervised pretraining, panoptic segmentation unified two tasks, and — most consequentially — transformers entered vision via DETR.

Three breakthroughs, three consolidations

Independent breakthroughs

• Scaling became a principled procedure — EfficientNet
• Pretraining stopped needing labels — MoCo, SimCLR, BYOL, SwAV
• Detection stopped needing anchors and NMS — DETR

Consolidations

• Semantic + instance unified as panoptic, with the PQ metric
• Video got a principled two-rate design — SlowFast
• Optical flow got its own architectural reset — RAFT

The three breakthroughs are independent but they compose. DETR plus a self-supervised transformer backbone is, structurally, the 2025 detector — it took five more years only to make it fast.

EfficientNet: compound scaling

ICML 2019 · Tan & Le — Google Brain · arXiv:1905.11946

Depth, width and input resolution are not independent: deeper networks need wider layers to exploit, and higher resolution to feed, the added capacity. The paper ties them to a single coefficient φ:

d = α^φ, w = β^φ, r = γ^φ subject to α · β² · γ² ≈ 2, α, β, γ ≥ 1

The constraint keeps FLOPs growing roughly as 2^φ, because FLOPs scale linearly in depth and quadratically in both width and resolution.

• B0, found by NAS over an MBConv + squeeze-excite space: 5.3M parameters, 0.39B FLOPs, 77.3% top-1.
• B7: 84.3–84.4% top-1, matching GPipe (557M parameters) while being 8.4× smaller and 6.1× faster at inference.
• Transferred well: 91.7% on CIFAR-100 and 98.8% on Flowers with an order of magnitude fewer parameters than comparable networks.

Limitations Later ablations found the NAS-discovered baseline contributed more than the scaling rule: applying compound scaling to a plain ResNet baseline underperforms the EfficientNet family. This is a recurring methodological trap — “method + baseline” is one confounded unit unless the two are deliberately separated, and the field would run into it again with ConvNeXt.

MoCo: unsupervised learning as dictionary lookup

CVPR 2020 · He, Fan, Wu, Xie & Girshick — FAIR · arXiv:1911.05722 · Mechanism 3

InfoNCE accuracy increases with the number of negatives. A single batch bounds that number by its own size:

ℒ_q = − log exp(q·k₊/τ) / Σ_i=0..K exp(q·k_i/τ)

Two ideas solve it:

• A FIFO queue of keys from previous batches, which decouples the negative count K from the batch size.
• A momentum encoder, θ_k ← m·θ_k + (1−m)·θ_q with m ≈ 0.999. Without it the queued keys — computed at different times by a rapidly changing encoder — are mutually inconsistent and training collapses. The slow-moving key encoder is what makes an old key still comparable to a new query.
• Shuffling BN prevents the model from cheating through intra-batch statistics.

First unsupervised pretraining to exceed supervised pretraining on several downstream detection and segmentation transfers.

SimCLR: ablation of the components

ICML 2020 · Chen, Kornblith, Norouzi & Hinton — Google Brain · arXiv:2002.05709 · Mechanism 3

No memory queue and no momentum encoder. Four components: stochastic augmentation, a base encoder, a projection head, and NT-Xent over in-batch negatives. The ablations:

• The composition of augmentations matters, not any single one. Random crop + colour distortion is the critical pair. Crop alone leaves a shortcut: two crops of one image share a colour histogram, so the network can solve the task without learning anything semantic.
• A nonlinear projection head, discarded afterwards, markedly improves the representation kept beneath it. The contrastive objective destroys information (colour, orientation) that the head can absorb, leaving the encoder free to retain it.
• Large batches and long schedules help far more than they do in supervised training.

MoCo v2 subsequently folded SimCLR’s stronger augmentations and MLP projection head back into the MoCo framework, so both components transfer across the two methods.

Beyond negatives: BYOL, SwAV and the collapse question

Contrastive methods were assumed to need negatives, because the constant function satisfies “two views should agree” perfectly. Two 2020 papers showed the assumption was wrong.

BYOL — no negatives at all An online network with a predictor head is trained to match a target network updated as an exponential moving average of the online weights. Collapse is avoided — empirically and robustly — by the asymmetry alone: predictor + stop-gradient + EMA target. Exactly why remained contested for years, which is itself instructive about how much of this era was empirical.

SwAV — online clustering Replace pairwise comparison with cluster assignment: predict one view’s “code” from the other, with a Sinkhorn equipartition constraint preventing collapse by forcing cluster usage to stay balanced. Also introduced multi-crop — several additional low-resolution views — which alone is worth several points and was adopted almost universally afterwards.

BYOL’s asymmetric self-distillation template — student, EMA teacher, stop-gradient — becomes DINO, then DINOv2, then DINOv3. It is the single most durable recipe in self-supervised vision.

Panoptic segmentation and PQ

CVPR 2019 · Kirillov, He, Girshick, Rother & Dollár — FAIR · arXiv:1801.00868

Semantic segmentation handled amorphous stuff; instance segmentation handled countable things. Separate datasets, separate models, separate metrics, and no way to compare them. Panoptic assigns every pixel a class and, for things, an instance id — a non-overlapping, exhaustive partition. The task figure is in the atlas.

The first factor is segmentation quality (how well matched segments overlap); the second is recognition quality (an F₁ over segments). Matching at IoU > 0.5 is provably unique, which is what makes the metric well defined.

The companion paper, Panoptic Feature Pyramid Networks, adds a semantic branch to Mask R-CNN’s existing FPN and reports both tasks from one network.

Video and motion: SlowFast and RAFT

SlowFast (ICCV 2019, arXiv:1812.03982) Sample semantics and motion at their own rates. A Slow pathway at low frame rate with high channel capacity captures what is there; a Fast pathway at high frame rate with ~⅛ the channels captures how it moves; lateral connections fuse Fast into Slow. SOTA on Kinetics, Charades and AVA, with the deliberately thin fast pathway keeping total cost near a single 3D CNN.

RAFT (ECCV 2020 best paper, arXiv:2003.12039) Coarse-to-fine flow pyramids lose small fast-moving objects irrecoverably. RAFT builds all-pairs 4D correlation volumes once at a single high resolution, then iteratively updates a flow field with a shared GRU that looks up correlation at the current estimate. One operator applied many times instead of a cascade of different ones — a large accuracy gain and unusually strong cross-dataset generalisation.

Both replace a hand-designed multi-stage pipeline with a single learned operator applied repeatedly — the same structural move that diffusion samplers and DETR’s decoder layers make.

StyleGAN2: diagnosing the artefacts in its own output

CVPR 2020 · Karras, Laine, Aittala, Hellsten, Lehtinen & Aila — NVIDIA · arXiv:1912.04958

StyleGAN images carried characteristic blob artefacts. AdaIN normalises each feature map to zero mean and unit variance, which discards map magnitude. The generator creates one large localised spike beforehand; the spike sets the mean and variance of its map, so magnitude survives normalisation as a ratio to the spike. The artefact was the model routing around its own architecture.

• Weight modulation / demodulation replaces AdaIN — scale the convolution weights and normalise them analytically. StyleGAN normalises activations; StyleGAN2 scales the weights.
• Drop progressive growing (it caused phase artefacts) in favour of a skip/residual generator and discriminator.
• Path-length regularization penalises variation in the Jacobian norm of the latent-to-image map, smoothing it.

The PPL improvement matters beyond image quality: a smoother latent map makes the generator far easier to invert, which enables reliable attribution of a generated image to its source model and real-image editing. StyleGAN2-ADA (2020) then added adaptive discriminator augmentation, reaching comparable quality with 10–100× less training data by tuning augmentation strength from an overfitting heuristic.

DETR: detection as direct set prediction

ECCV 2020 · Carion, Massa, Synnaeve, Usunier, Kirillov & Zagoruyko — FAIR · arXiv:2005.12872 · Mechanism 2

The loss is the contribution, not the architecture. Pad the ground truth to size N with ∅ and find the permutation that minimises total matching cost:

σ̂ = arg min_σ∈𝔖N Σ_i=1..N ℒ_match( y_i, ŷ_σ(i) )

Solved by the Hungarian algorithm. The box cost mixes ℓ₁ with generalised IoU so it is scale-invariant; unmatched slots are trained toward “no object”.

Because the assignment is a bijection, two queries cannot both claim one object. Duplicate suppression is trained into the loss, so the NMS stage is removed. Auxiliary decoding losses at every decoder layer are required for it to converge at all.

Results

• DETR-R50: 42.0 COCO AP, matching Faster R-CNN
• 86 GFLOPs versus 180 — less than half
• Higher AP on large objects, from global attention

Costs

• Lower AP on small objects
• 500 epochs to converge
• Both follow from dense attention on one low-resolution feature map

ICLR 2021 · Zhu, Su, Lu, Li, Wang & Dai · arXiv:2010.04159

Deformable DETR — sparsity fixes both problems

At initialisation, dense global attention is nearly uniform over all spatial locations; learning to be selective is what consumes hundreds of epochs. Uniform attention also cannot afford high-resolution feature maps — which is exactly why small objects suffered. Deformable attention samples only K learned offsets around a reference point per query per level, giving 46.2 AP with 10× fewer training epochs and closing the small-object gap. The follow-up line (Conditional DETR, DAB-DETR, DN-DETR, DINO) attacks query semantics and denoising, and leads directly to RF-DETR in 2025.

Era recap

The quiet turning point DETR placed a transformer within a vision system. The next era applies the transformer to the backbone as well.

The transformer takeover

The period in which transformers were applied across vision tasks. It opens with DETR applying one to detection, continues with ViT exceeding CNN accuracy on classification above a data threshold, and closes with diffusion reporting lower FID than GANs.

Two shifts at once

Architecture

• ViT — plain transformers exceed CNN accuracy above a data threshold
• DeiT — and below it too, with the right recipe
• Swin / PVT / SegFormer — hierarchy restored for dense prediction

Supervision and generation

• CLIP — language becomes an open label space
• DINO / BEiT / MAE — self-supervised ViTs
• DDPM → guidance → latent diffusion — GANs displaced

Every component of a 2026 system is present in prototype form by the end of this window.

ViT: an image is a sequence of patches

ICLR 2021 · Dosovitskiy et al. — Google Research / Brain · arXiv:2010.11929

ViT-L/16 top-1 on ImageNet, by pretraining corpus:

• ImageNet-1k only: 76.5% — worse than a comparable ResNet
• ImageNet-21k: 85.3%
• JFT-300M: 87.8% — above the highest CNN result at matched parameters and FLOPs

What the crossover measures Locality and translation equivariance are a prior. Priors substitute for data — and therefore cost accuracy once data is abundant, because they constrain the hypothesis space in ways the data no longer requires. The crossover sits somewhere around 10⁸ images. This single result reoriented the field’s research agenda from architecture design to pretraining scale.

DeiT (ICML 2021) then reached competitive ImageNet-1k-only accuracy using strong augmentation, heavy regularisation and a distillation token — so part of the threshold is attributable to the training recipe. This is the same confound that ConvNeXt would quantify a year later.

Swin Transformer: hierarchy and linear cost

ICCV 2021 · Liu et al. — Microsoft Research Asia · arXiv:2103.14030

ViT is unusable for dense prediction: attention is O(N²) in the number of patches, and the resolution never changes. Swin fixes both:

• Windowed attention restricts self-attention to non-overlapping M×M windows (M = 7), making cost linear in image size.
• Shifted windows alternate the partition by ⌊M/2⌋ between consecutive blocks, with a cyclic shift plus masking to keep batches regular.
• Patch merging concatenates each 2×2 token neighbourhood between stages: resolution halves, channels double — a feature pyramid built from transformer blocks, directly compatible with FPN-style heads.

87.3% ImageNet-1K top-1, 58.7 COCO box AP, 53.5 ADE20K mIoU — above the prior reported results on all three. Shifted windows and hierarchical merging were adopted across dense-prediction backbones afterwards.

The hierarchical-transformer wave

• PVT (2021) — Progressive shrinking across four stages with spatial-reduction attention, downsampling keys and values to make attention affordable at high resolution.

• SegFormer (2021) — Hierarchical Mix Transformer encoder plus an all-MLP decoder. Drops positional encodings entirely — using a 3×3 depthwise conv in the FFN instead — so it generalises across test resolutions without interpolation.

• Twins, CSWin, MViT — Variations on the same axis: which tokens attend to which, and how to restore a pyramid. MViT adds pooling attention for video.

The convergence worth noting Every successful “transformer for dense prediction” re-imports the two properties ViT discarded: locality (windows, pooling, spatial reduction) and multi-scale hierarchy (patch merging, staged shrinking). In these designs the inductive bias sits in the attention pattern and the stage layout, where it can be changed without changing the operator.

CLIP: language as the label space

ICML 2021 · Radford et al. — OpenAI · arXiv:2103.00020

Trained on 400 million image–text pairs scraped from the web with a symmetric InfoNCE objective over a learned temperature. At test time the text encoder synthesises a linear classifier: embed “a photo of a {class}” for an arbitrary class list and classify by nearest embedding. The class list is supplied at inference as text. Ensembling over several prompt templates raises ImageNet accuracy by close to 5 points, at no additional training cost.

CLIP was a simplified, massively scaled re-derivation of the earlier ConVIRT approach — the predecessor mattered far less than the scale.

What it became

• The text-conditioning mechanism for DALL·E 2 and Stable Diffusion
• The vision tower of most early multimodal LLMs
• The basis of open-vocabulary detection and segmentation
• A robustness result in its own right — much flatter degradation across ImageNet-V2/R/Sketch than supervised models at equal ImageNet accuracy

Documented weaknesses

• Poor at counting, spatial relations and compositional binding — web captions rarely depend on them
• Weak on fine-grained and specialist domains; zero-shot is not free expertise
• The softmax contrastive loss needs very large batches. SigLIP (2023) replaces it with a pairwise sigmoid loss — no global normalisation, so it trains well at modest batch size and scales better

Self-supervised ViTs: DINO, BEiT, MAE

Three routes to the same goal, with a durable division of labour between them.

• DINO (ICCV 2021) — BYOL-style self-distillation with a ViT: the student matches an EMA teacher’s sharpened output distribution, with centring to prevent collapse, over multi-crop views.
  
  The surprise: the [CLS] token’s attention maps contain clean unsupervised object segmentations, and k-NN on the features alone is strong. Emergent structure nobody trained for.

• BEiT (ICLR 2022) — BERT for images: tokenise the image with a discrete VAE, mask patches, and predict the visual token ids rather than pixels. Establishes masked image modelling but inherits a dependency on a separately trained tokeniser.

• MAE (CVPR 2022) — Drops the tokeniser: predict raw pixels of masked patches, with a very high mask ratio and an asymmetric encoder–decoder. Simpler, faster and better — and the asymmetry is what makes it scale.

The division that persists: contrastive and distillation methods give strong linear-probe features; masked modelling gives features that fine-tune better. DINOv2 later combines both objectives precisely for this reason.

CVPR 2022 · He, Chen, Xie, Li, Dollár & Girshick — FAIR · arXiv:2111.06377 · Mechanism 3

MAE: asymmetry is the mechanism

• Masking 75% of patches, with an ℓ₂ loss on normalised patch targets.
• The encoder cost drops ~4×, giving >3× faster wall-clock training.
• The high ratio is essential: at low ratios the task is solvable by local interpolation and teaches nothing semantic.
• ViT-Huge reached 87.8% on ImageNet-1K using only ImageNet-1K data — no web-scale corpus, no paired text. A direct counterpoint to CLIP’s premise.

Diffusion, in three steps

DDPM (2020) Fix a forward noising process and learn to reverse it. Reparameterise to predict the noise:
ℒ = 𝔼[ ‖ε − ε<sub>θ</sub>(x<sub>t</sub>, t)‖² ]
A stable regression objective — no discriminator, no mode collapse. The cost is hundreds of sequential steps.

Guidance (2021–22) Classifier guidance steers sampling with an external classifier’s gradient. Classifier-free guidance removes the classifier: jointly train conditional and unconditional models via random condition dropout, then extrapolate
ε̃ = ε<sub>∅</sub> + s(ε<sub>c</sub> − ε<sub>∅</sub>)
One scalar trading diversity for fidelity. Universally adopted.

Beating GANs (2021) “Diffusion Models Beat GANs” established diffusion as SOTA on ImageNet synthesis. The remaining cost is hundreds of U-Net evaluations at full pixel resolution. Latent diffusion addresses that cost.

Adversarial training is replaced by a regression loss on the added noise, which has a single global optimum. The GAN stability literature — WGAN, gradient penalties, progressive growing, path-length regularization — was not carried further.

Latent Diffusion: move the work to a small space

CVPR 2022 · Rombach, Blattmann, Lorenz, Esser & Ommer — released as Stable Diffusion · arXiv:2112.10752 · Mechanism 4

Most bits in an image are perceptually irrelevant detail. Spend diffusion capacity only on the semantic part:

• Stage 1 — a VQ- or KL-regularised autoencoder compresses by f = 4–8, trained once with a perceptual plus patch-adversarial loss.
• Stage 2 — diffusion at ~64×64 instead of 512×512, roughly a 48× smaller tensor to denoise.
• Cross-attention layers in the U-Net inject conditioning of any modality.

Released publicly in August 2022 under an open licence, it was the first capable text-to-image model with open weights that ran on consumer GPU hardware and could be freely fine-tuned — widely regarded as the accessibility inflection point for the whole field.

Text-to-image, 2021–2022

DALL·E was a 12-billion-parameter GPT-3 variant: text to BPE tokens, autoregressive generation of 1,024 image tokens, decoding through a discrete VAE, then re-ranking candidates with CLIP. DALL·E 2 moved to diffusion and reported more photorealistic results with fewer parameters (3.5B), so the gain is attributable to the change of model class at reduced scale.

Era recap

The shared shape Every row decouples an expensive stage from a cheap one and scales only the expensive stage once: pretrain once and use many times; compress once and denoise many times. That is the foundation-model thesis, and the next era is its consequence.

Foundation models and the promptable paradigm

The CNN mounts a genuine counterattack, segmentation unifies around universal mask prediction, self-supervision reaches production-grade backbones, and the “promptable” paradigm from language models arrives fully in vision.

Consolidation, then a new interface

Consolidation

• ConvNeXt (and V2) — it was the recipe, not the operator
• Mask2Former — one architecture for all three segmentation tasks
• DINOv2 — frozen features that no longer need fine-tuning

The interface opens

• SAM — promptable, zero-shot segmentation at scale
• Grounding DINO / OWL-ViT — open-vocabulary detection
• ControlNet — structural conditioning without retraining

The defining shift: models stop being trained for a task and start being queried for one.

ConvNeXt: a controlled experiment

CVPR 2022 · Liu, Mao, Wu, Feichtenhofer, Darrell & Xie — FAIR / UC Berkeley · arXiv:2201.03545

Not an architecture paper — an ablation paper, and a direct response to the moment when vision transformers seemed poised to replace convolutional networks entirely. The modernisation proceeded through five stages:

• Training recipe alone (AdamW, 300 epochs, RandAugment, Mixup, stochastic depth): 76.1 → 78.8%, before any architectural change at all.
• Macro design: stage compute ratio 3:3:9:3; a “patchify” 4×4 stride-4 stem, mirroring Swin.
• ResNeXt-style grouped/depthwise convolution with an inverted bottleneck.
• Larger kernels, up to 7×7.
• Micro design: GELU for ReLU, fewer activation and normalization layers, LayerNorm for BatchNorm, separate spatial downsampling layers.

Result: 82.0% versus Swin-T’s 81.3%, with roughly 49% higher GPU throughput because convolutions map better to hardware. Scaled up, the family reached 87.8% top-1, scored above comparable Swin models on COCO and ADE20K, and degraded less on ImageNet-A/R/Sketch.

The uncomfortable first bullet A 2.7-point gain from the training recipe alone, before touching the architecture. Part of the “architecture progress” reported between 2019 and 2021 is recipe progress reported as architecture progress — a confound the field had already met with EfficientNet and would keep meeting. The conclusion ConvNeXt actually supports is that the CNN/transformer gap was training recipe and micro-design, not self-attention.

ConvNeXt V2: giving ConvNets masked pretraining

CVPR 2023 · Woo, Debnath, Hu, Chen, Liu, Kweon & Xie · arXiv:2301.00808

ConvNeXt V1 could not benefit from MAE-style pretraining, for two structural reasons:

• Dense convolution has no notion of a missing token. A masked region is a hole across which the convolution interpolates, so the pretext task is solvable without learning the content. Treat the masked image as sparse and use sparse convolutions during pretraining, collapsing back to dense at fine-tuning time.
• Feature collapse. Under masked training, many channels become redundant. The fix: Global Response Normalization — normalise each channel by the aggregate response across channels, explicitly rewarding feature diversity.

The general point A self-supervised objective is specified together with an architecture. Masked modelling was defined on the transformer’s token structure; applying it to a convolutional operator required a sparse formulation of the masking and an additional normalisation layer. The same lesson recurs whenever a pretraining recipe is moved across architectures — and it is why “just use MAE” is rarely a complete answer.

Mask2Former: all segmentation is mask classification

CVPR 2022 · Cheng, Misra, Schwing, Kirillov & Girdhar — FAIR / UIUC · arXiv:2112.01527 · Mechanism 2

One model, three tasks — the same output read differently:

• Merge same-class masks → semantic segmentation
• Keep “thing” masks separate → instance segmentation
• Take the highest-scoring non-overlapping subset → panoptic segmentation

Its key technical innovation, masked attention, constrains each query’s cross-attention to its own predicted mask region from the previous layer, in place of attending over the whole image, which localises the features and shortens the schedule. Two further efficiency choices matter in practice: high-resolution features fed round-robin to successive decoder layers, and the loss computed on sampled points rather than whole masks, which is a large memory saving.

New state of the art on four benchmarks at once: 57.8 PQ panoptic and 50.1 AP instance on COCO, and 57.7 mIoU semantic on ADE20K — above the specialised architectures including Mask R-CNN and DeepLab-based models, from a single model, and reducing the per-task research and engineering effort by at least threefold.

A contemporaneous counterpoint: SegFormer kept the traditional per-pixel grid-labelling output but rebuilt the encoder as a hierarchical Mix Transformer with a lightweight MLP decoder, becoming the go-to choice for fast, accurate plain semantic segmentation. Two complementary answers to the same reframing — “predict a set of masks” versus “label every pixel, but better”.

Open-vocabulary detection

OWL-ViT (2022) A minimal recipe: take a CLIP ViT, remove pooling, attach a lightweight box head and a classification head that compares each token’s embedding against text embeddings, then fine-tune on detection data. Detection becomes a matching problem against an open label set, and it supports image-conditioned queries too.

Grounding DINO (2023) Fuse language into a DINO-style detector at three points — neck, query initialisation and head — in place of the output alone. Trained jointly on detection, grounding and caption data. Zero-shot COCO transfer is reported, and the text input accepts referring expressions as well as class names.

Grounding DINO + SAM became the standard open-vocabulary segmentation pipeline purely by composition — one model names and localises, the other produces the mask. That compositionality is a direct consequence of SAM’s decision not to classify, below.

Detection inherits CLIP’s open vocabulary — and CLIP’s compositional weaknesses along with it.

DINOv2: features good enough to freeze

TMLR 2024 (released April 2023) · Oquab et al. — Meta AI / FAIR · arXiv:2304.07193 · Mechanism 3

Combines DINO’s image-level self-distillation with iBOT’s patch-level masked prediction, then scales carefully:

• Curated data (LVD-142M), assembled by retrieval-based deduplication and balancing from a large uncurated pool. The paper’s argument is that curation, not raw volume, is what makes self-supervised learning scale.
• A model family from ViT-S (21M) to ViT-g (1.1B), with the largest distilled down into the smaller ones.
• Infrastructure at scale: fully sharded training, FlashAttention, batch sizes around 65,000.

The result that matters: the features work frozen. A linear probe gives competitive classification, segmentation, depth estimation and retrieval with no task-specific fine-tuning. It was the first self-supervised method to match or surpass weakly-supervised (CLIP-style) approaches broadly — and it uniquely learned dense properties like depth that classification supervision never produces.

Segment Anything

ICCV 2023 · Kirillov et al. — Meta AI / FAIR · arXiv:2304.02643 · Mechanism 5

Give it an image plus a prompt — a foreground point, a bounding box, or a rough mask — and it returns a valid segmentation mask, including for objects and image distributions never seen during training. For an ambiguous prompt the decoder emits three nested masks with confidence scores, in place of averaging them into mush.

Three design decisions worth copying

1. Asymmetric cost. The ViT-H encoder runs once per image (~0.15 s); the prompt encoder and mask decoder run in ~50 ms, in a browser. This is what makes interactive use — and the annotation loop below — possible at all.
2. A data engine, not a dataset. Three stages: assisted-manual, then semi-automatic (the model proposes, humans add what it missed), then fully automatic over a 32×32 point grid. Yielded SA-1B: over 1.1 billion masks across 11 million licensed, privacy-respecting images — orders of magnitude beyond any prior segmentation dataset.
3. It refuses to classify. SAM produces high-quality masks and leaves object naming to a downstream classifier, detector or language model. That deliberate incompleteness is exactly what makes it composable into other people’s pipelines.

The generalisable pattern is the data engine It is a bootstrapping argument: a mediocre model plus human correction produces data that trains a better model, which then needs less correction. The same structure reappears in SAM 2’s video engine, SAM 3’s concept engine, and Depth Anything’s pseudo-labelling. This is how the field routed around the annotation bottleneck without ever solving it.

ControlNet: mechanism 1, fifteen years later

ICCV 2023 · Zhang, Rao & Agrawala — Stanford · arXiv:2302.05543 · Mechanism 1

Text prompts control content but not geometry. ControlNet adds conditioning on edges, depth, normals, segmentation maps or pose skeletons — without retraining the base model.

The construction: clone the encoder blocks of a frozen Stable Diffusion, connect the copy through zero-initialised 1×1 convolutions, and add its output back into the frozen path. At step 0 the zero convolutions output exactly zero, so the model is bit-identical to the original — training can only add capability, never degrade the base. And it does learn, because the gradient through a zero convolution is non-zero whenever the input is non-zero.

It works from a few tens of thousands of examples and reports a sudden-convergence step at which the model begins to follow the condition.

SDXL (arXiv:2307.01952) pushed quality further along three axes: an ensemble of text encoders concatenating CLIP and OpenCLIP embeddings for richer conditioning; an ensemble-of-experts approach adding a dedicated refiner model specialised for the final denoising steps; and micro-conditioning that feeds the model signals like original image resolution without extra supervision. Plug-and-Play (CVPR 2023) offered a complementary image-to-image route, using an image condition for coarse structure while the text prompt governs fine detail.

Era recap

Three of these eight are the same mechanism DINOv2, SAM and ControlNet all say: freeze a large pretrained thing, attach something cheap, train only that. The variations are what gets attached — a linear probe, a prompt decoder, a zero-initialised side branch — and what the frozen thing knows.

3D becomes real-time, vision learns to talk

Three converging trends: promptable segmentation extends fully into video, 3D reconstruction shifts from implicit neural fields to explicit primitives, and multimodal language models become a mainstream vision interface.

What changed

3D changes representation

• NeRF — beautiful, implicit, fundamentally slow
• 3D Gaussian Splatting — explicit primitives, real-time
• Depth Anything V2 — monocular depth across indoor, outdoor and aerial imagery

Interface and efficiency

• LLaVA — the open vision + LLM recipe
• SAM 2 — prompting extended to video via memory
• YOLOv9 / v10 — NMS removed from the real-time lineage
• Consistency models, flow matching — few-step generation

NeRF: the implicit baseline

An MLP maps (x, y, z, θ, φ) → (colour, density), rendered by quadrature along camera rays with a differentiable volume-rendering integral. Because the whole path is differentiable, a photometric loss on the training views optimises the scene itself. The pipeline figure and the full loss family are in the task atlas.

• Positional encoding is essential — without it the MLP’s spectral bias produces blurry mush.
• View-dependence gives specularities almost for free.
• The cost: an MLP evaluation per sample per ray, spent uniformly — including on empty space. Days to train, seconds per frame.

3D Gaussian Splatting

SIGGRAPH / ACM TOG 2023 · Kerbl, Kopanas, Leimkühler & Drettakis — Inria / MPI · arXiv:2308.04079 · Mechanism 6

Represent the scene as millions of anisotropic 3D Gaussians — position, covariance (factored as scale × rotation to keep it positive semi-definite), opacity, and spherical-harmonic view-dependent colour. Three contributions:

1. 3D Gaussians initialised from the sparse SfM point cloud already produced by camera calibration, which keeps the properties of a continuous volumetric radiance field and places no primitives in empty space.
2. Interleaved optimisation and adaptive density control — clone Gaussians in under-reconstructed regions, split over-large ones, prune transparent ones, driven by view-space positional gradients.
3. A fast, visibility-aware tile-based rasteriser with per-tile depth sorting and correct differentiable α-blending, which accelerates both training and rendering.

State-of-the-art visual quality at ≥30 fps, 1080p — a combination the prior NeRF-based methods, which query a network per pixel, did not reach at the same time. A 2024 survey records the combination of the reconstruction quality of implicit neural models with an explicit data structure that is dramatically more editable and efficient to render.

Splatting: advantages and limitations

The mechanism

• Compute follows content. Empty space costs nothing because there are no primitives there. NeRF pays for it on every ray.
• Rasterisation, not ray marching. Decades of GPU design are optimised for exactly this operation.
• Explicit data structures are editable — crop, transform, compose, stream. A NeRF’s scene lives in opaque MLP weights.
• Optimisation is well-conditioned — each Gaussian has local influence, so gradients do not fight globally.

The costs

• No surface. A Gaussian cloud is not a mesh and has no well-defined normal; extracting geometry needs extra machinery (SuGaR, 2DGS and successors).
• Optimised for view synthesis, not metric accuracy. It can look photorealistic while being locally wrong.
• Memory-hungry — millions of primitives per scene.
• Anisotropic Gaussians produce their own characteristic artefacts at extrapolated viewpoints.

LLaVA and the VLM architecture template

NeurIPS 2023 Oral · Liu, Li, Wu & Lee — Wisconsin / Microsoft / Columbia · arXiv:2304.08485

The contribution is the training data. Language-only GPT-4 was prompted with existing image annotations, captions and boxes, to generate multimodal instruction-following dialogue. Training then proceeds in two stages:

• Stage 1 — feature alignment on a 558K-image subset of LAION-CC-SBU, training the projection only, with both encoder and LLM frozen.
• Stage 2 — visual instruction tuning on ~150K GPT-generated dialogues plus ~515K academic VQA examples.

The first model reached an 85.1% relative score against GPT-4 on a synthetic multimodal benchmark, and 92.53% on ScienceQA when fine-tuned. LLaVA-1.5 (October 2023) improved the projection to a two-layer MLP and led 11 of 12 benchmarks at release; LLaVA-NeXT (2024) extended to new LLM backbones and higher resolutions.

The design space this opened: connector type (linear, MLP, Q-Former, perceiver resampler); whether the vision tower is frozen or jointly trained; and how to handle native resolution (tiling, AnyRes, dynamic ViT). Every subsequent open VLM is a set of choices in that space.

OpenAI’s GPT-4V, released in 2023, was the leading closed counterpart, scoring 67.7% on the MM-Vet benchmark for integrated vision–language capability — substantially ahead of contemporary open models at the time. Researchers found that visual Chain-of-Thought prompting yielded significant additional improvements on structured visual reasoning such as mathematical reasoning and chart analysis.

SAM 2: an image is a one-frame video

Meta FAIR, July 2024 · Ravi, Gabeur, Hu, Hu, Ryali, Ma, Khedr, Rädle, Rolland, Gustafson et al. · arXiv:2408.00714 · Mechanism 5

Generalises SAM from single images to video by treating an image as a single-frame video. The new task, Promptable Visual Segmentation, lets a user provide points, boxes or masks on any frame and returns the corresponding spatio-temporal masklet across all frames — iteratively refinable with further prompts, and robust to the object temporarily disappearing behind an occluder, with an occlusion head predicting object presence.

• SA-V: roughly 51,000 real-world videos and more than 600,000 masklets collected across 47 countries — 53× more annotations than the previous largest video segmentation dataset, produced by a model-in-the-loop data engine.
• Better video accuracy using 3× fewer user interactions than prior approaches such as SAM combined with XMem++ or Cutie.
• 6–8.4× faster than the original SAM on pure image segmentation, and more accurate.
• Released under Apache 2.0 in four sizes (Tiny, Small, Base Plus, Large), with real-time 30+ fps for all but the largest; SAM 2.1 added multi-object tracking improvements in late 2024.

Note the shape of the result: a memory module plus a data engine converted an image model into a video model without changing the underlying paradigm.

YOLOv9 and YOLOv10: closing the set-prediction loop

NeurIPS 2024 · Wang, Chen, Liu, Chen, Lin, Han & Ding — Tsinghua · arXiv:2405.14458 · Mechanism 2

NMS was the last hand-designed stage in the real-time lineage — and a genuine latency cost, because it is sequential and data-dependent. YOLOv10 removes it entirely through consistent dual assignments: the two heads use a consistent matching metric so they rank candidates identically, which is what lets the one-to-one head inherit the one-to-many head’s fast convergence without inheriting its duplicates.

DETR’s idea, delivered at YOLO latency. Across six variants (N, S, M, B, L, X): YOLOv10-N at 1.84 ms and YOLOv10-S at 2.49 ms are the lowest-latency options; YOLOv10-X reaches 54.4% mAP at 10.70 ms.

YOLOv8 (Ultralytics, 2023) had earlier introduced a CSPDarknet backbone with cross-stage partial connections and SiLU activation, and unified detection, instance segmentation, pose estimation, classification and oriented bounding boxes in one codebase. YOLOv9 (arXiv:2402.13616) introduced Programmable Gradient Information with a reversible auxiliary branch, addressing information lost through deep feedforward stacks, and showed the best balance of precision, recall and mAP in several comparative studies including adverse-weather conditions.

Making diffusion fast

Consistency models (ICML 2023) The probability-flow ODE defines a trajectory from noise to data. Train a network f_θ(x_t, t) that maps any point on a trajectory to its origin, enforcing f_θ(x_t,t) = f_θ(x_t′,t′) along it. Gives one- or two-step sampling, either distilled from a teacher diffusion model or trained standalone. Latent Consistency Models port this to Stable Diffusion.

Flow matching / rectified flow (ICLR 2023) The model regresses a velocity field that transports noise to data along prescribed paths, preferably straight ones:
ℒ = 𝔼[ ‖v<sub>θ</sub>(x<sub>t</sub>,t) − (x₁ − x₀)‖² ]
Simulation-free training, and straight paths mean fewer integration steps. Adopted as the training objective of SD3, Flux and much subsequent work.

Both reduce the number of network evaluations at sampling time: one shortens the trajectory, the other maps points on it directly to the origin. Both address the same property, that diffusion sampling is iterative where GAN sampling is one forward pass.

Era recap

What the frontier question became SAM 2 is a data-engine story. LLaVA is a synthetic-supervision story. Depth Anything, in the next era, is a distillation story. In each the architecture is taken from earlier work and the contribution is the source of the training signal.

Video, depth and concepts

Video generation reaches minute-long coherence, self-supervised backbones scale to billions of parameters and exceed weakly-supervised methods on dense tasks, segmentation extends from geometric prompts to natural-language concepts, and discriminatively trained monocular depth scores above diffusion-based depth.

DiT: giving generation a scaling law

ICCV 2023 · Peebles & Xie · arXiv:2212.09748 · Mechanism 1 · Mechanism 4

Replace latent diffusion’s U-Net with a plain transformer over latent patches, conditioning through adaLN-Zero — regress per-block scale and shift from the timestep/class embedding, initialised so that each block starts as the identity. That last detail is mechanism 1 again: a residual branch that begins as a no-op is safe to add.

FID decreases monotonically with transformer Gflops across the model sizes tested. The extension to video then follows by adding compute.

Sora: video generation as world simulation

OpenAI, February 2024 · Video generation models as world simulators · Mechanism 4

A text-to-video diffusion model capable of generating up to one minute of high-definition video while maintaining visual quality and strong prompt adherence. The architecture is a diffusion transformer:

• Compress raw video into a lower-dimensional latent spacetime representation using a dedicated video compression network.
• Decompose that representation into spacetime patches — the direct analogue of word tokens in a language model — which serve as the transformer’s input.
• Generate by starting from pure visual noise and iteratively denoising, guided by CLIP-like text conditioning.

The key technical insight was that unifying how video and image data are represented as patches allowed training across a much wider range of durations, resolutions and aspect ratios than prior video models could handle. Sample quality improved significantly with additional training compute — diffusion transformers scale for video as effectively as they do for language and images.

Sora builds on the same diffusion architecture underlying DALL·E 3, adapted with additional temporal controls. Open-source efforts such as Open-Sora replicated the approach with a Spatial-Temporal Diffusion Transformer (STDiT) that decouples spatial self-attention within each frame from temporal attention across frames at the same spatial location — making efficient video diffusion training more broadly accessible.

On “general purpose simulators of the physical world” OpenAI explicitly framed the scaling behaviour as suggestive of a path toward world simulation, and that framing shaped a great deal of subsequent research. These models are trained on appearance. Object permanence, contact and causality appear in the samples partially and inconsistently, and are not constrained by the training objective.

Depth Anything V2: data engineering, fixed architecture

NeurIPS 2024 · Yang, Kang, Huang, Zhao, Xu, Feng & Zhao · arXiv:2406.09414 · Mechanism 3

Monocular depth estimation — predicting a full depth map from a single RGB image, with no multi-camera setup — advanced through three data choices, with the architecture held fixed:

1. Replace all labelled real training images with photorealistic synthetic ones for the initial teacher. Synthetic labels are exact; real depth annotations are noisy, sparse at edges, and incorrect on transparent surfaces.
2. Scale up the teacher’s capacity substantially, so it can absorb the synthetic-to-real domain gap.
3. Use that teacher to pseudo-label over 62 million real unlabelled images, and train the released student models on that bridge — never on the noisy real labels.

The resulting models span 25M to 1.3B parameters and are more than 10× faster than depth models built on Stable Diffusion such as Marigold and GeoWizard, at higher accuracy, with finer detail and smaller degradation than Depth Anything V1. The ViT-Large variant achieved roughly 97.1% mean accuracy across indoor, outdoor, transparent, adverse-weather, aerial and underwater imagery. An engineered discriminative pipeline therefore scores above the more computationally expensive generative approaches on this dense-prediction task.

DINOv3: self-supervision at 7B parameters

Meta AI / FAIR, August 2025 · Siméoni, Vo, Seitzer et al. · arXiv:2508.10104 · Mechanism 3

A 7× larger model than DINOv2 (a 6.7-billion-parameter ViT) trained on a 12× larger dataset (roughly 1.7 billion images curated from a pool of about 17 billion), while using only a fraction of the compute that weakly-supervised methods like CLIP require.

Gram anchoring is the key technical innovation. In very long, large-scale self-supervised runs, global features keep improving while dense, patch-level features degrade — the patch similarity structure drifts and becomes noisy, which silently destroys exactly what makes the backbone useful for segmentation and depth. Gram anchoring regularises the Gram matrix of patch features toward an earlier checkpoint’s, preserving dense quality without freezing the representation outright.

Evaluated as a frozen backbone — no fine-tuning — across roughly sixty benchmarks and fifteen distinct vision tasks, DINOv3 is reported by its authors at state-of-the-art performance. It is the first purely self-supervised model to score above weakly-supervised models such as SigLIP 2 on a broad range of dense-prediction probing tasks including object detection, semantic segmentation, depth estimation and video object tracking. Released under a commercial licence with backbones from 21M to 6.7B parameters, plus dedicated ConvNeXt variants and a specialised satellite-imagery backbone trained on MAXAR data.

Why this closes a loop opened in 2019 > MoCo made unsupervised pretraining competitive with supervised. DINOv2 made it competitive with weakly-supervised on many tasks. DINOv3 scores above it on a broad set of dense tasks, with a correction to a failure mode, dense-feature drift, that appears only at scale. On these benchmarks the reported results require neither labels nor paired text.

SAM 3: promptable concept segmentation

Meta AI / FAIR, November 2025 · Segment Anything with Concepts · Mechanism 5

SAM 1 and SAM 2 answered “segment the thing I am pointing at” — one object per prompt. SAM 3 extends the lineage beyond geometric prompts to natural-language concept prompts, introducing the task of Promptable Concept Segmentation (PCS): given a short noun phrase such as “yellow school bus”, or an image exemplar, return segmentation masks and unique identities for every matching object instance simultaneously.

• Architecturally, an image-level detector and a memory-based video tracker share a single backbone.
• Recognition and localisation are deliberately decoupled via a dedicated presence head, which separates “is this concept present?” from “where exactly is it?” — and measurably boosts detection accuracy by letting each question be answered by the machinery suited to it.
• Trained via a scalable data engine producing a dataset with 4 million unique concept labels, including deliberately constructed hard negatives, spanning both images and video.
• 2× accuracy gain over existing systems on both image and video promptable concept segmentation, while also improving on previous SAM capabilities for standard visual segmentation.
• Open-sourced alongside a new benchmark, Segment Anything with Concepts (SA-Co), built specifically to evaluate the task.

This unifies single-image, video, interactive refinement and concept-driven detection under one backbone — transforming SAM from a geometric segmentation tool into a concept-level vision foundation model.

YOLO11: continued real-time refinement

Ultralytics, September 2024

Two headline architectural changes: compact C3k2 CSP bottleneck blocks replacing the prior C2f blocks for improved efficiency, and a new C2PSA module combining cross-stage-partial design with spatial attention for more robust feature aggregation — particularly benefiting small-object detection. Multi-task support spans detection, instance segmentation, pose estimation, classification and oriented bounding boxes in one framework.

YOLO11 reports up to 42% fewer parameters than YOLOv8 in its large variant at higher mAP, pushing the accuracy–latency Pareto frontier further than any prior YOLO generation. YOLO11x reaches 54.7% mAP at 11.3 ms TensorRT latency on a T4 GPU; the smallest variant, YOLO11n, achieves 39.5% mAP at just 1.5 ms.

Vision–language models consolidate

By 2025 the open VLM landscape had consolidated around a small number of dominant lineages — most prominently Alibaba’s Qwen-VL series and Shanghai AI Lab’s InternVL series — both of which rapidly approached and in some benchmarks matched closed frontier models.

Qwen2-VL (2024) introduced M-RoPE (multimodal rotary position encoding) and native dynamic-resolution image processing; Qwen2.5-VL refined this into a native dynamic ViT combined with window attention, reducing computational cost while preserving native image resolution and adding support for hours-long video understanding and computer/phone agentic tool use.

The documented gap A 2026 benchmarking analysis found that even leading open VLMs such as InternVL3 and Qwen2.5-VL still show notable weakness on fine-grained shape and pattern recognition — around 30–45% accuracy — relative to their strong performance on document and chart understanding, where they exceed 90%. They read text in images superbly and reason about geometry in them poorly. Visual–language alignment for abstract geometric reasoning remained an unsolved gap as of this period, and the bottleneck is the vision–language interface and training distribution, not the language model.

Era recap

Any-view geometry, concepts and embodiment

Geometry becomes a single forward pass, segmentation reaches single-image 3D reconstruction, and a transformer detector exceeds 60 AP on COCO in real time.

VGGT: geometry as a single forward pass

CVPR 2025 · Wang, Chen, Karaev, Vedaldi, Rupprecht & Novotny — Oxford VGG / Meta · arXiv:2503.11651

Classical 3D vision is an optimisation pipeline: features → matching → RANSAC → triangulation → bundle adjustment. VGGT replaces the whole thing with one feed-forward transformer.

• Input: one to hundreds of views. Output: camera intrinsics and extrinsics, depth maps, point maps and point tracks — predicted jointly.
• Alternating attention: frame-wise self-attention within a view, interleaved with global attention across all views. This is what lets the model reason about cross-view geometry without an explicit matching stage.
• Inference is a single forward pass, in seconds. The optimisation pipelines take minutes to hours.

Over-parameterised joint prediction is reported above the geometry-aware pipelines built around the explicit equations, and is used as an initialiser for subsequent bundle adjustment, which is how the two are now combined.

Depth Anything 3: one target for any number of views

ByteDance Seed, November 2025 · Lin, Chen, Liew, Chen, Li, Shi, Feng & Kang · arXiv:2511.10647

Generalises monocular depth estimation into a unified spatial reconstruction system capable of predicting spatially consistent geometry from an arbitrary number of visual inputs — single images, multi-view collections, or full videos — with or without known camera poses. Two deliberately minimal design choices:

• A single plain transformer backbone, a DINO-style encoder with no task-specific layers.
• A singular unified “depth–ray” prediction target replaces the multi-task training pipelines prior geometry models used. Depth encodes the distance from a pixel to the camera and the ray encodes that pixel’s projection direction in 3D. Together they determine the geometry required for reconstruction, so one head replaces the separate per-task heads.

On its evaluation benchmark DA3 reports the highest scores across all tested tasks, above VGGT by an average of 35–44% in camera pose accuracy and 23–25% in geometric accuracy, and above its own predecessor Depth Anything V2 on monocular depth specifically — despite DA3’s far broader any-view scope.

The team’s own framing, and why it matters “A plain transformer trained on depth-and-ray targets with teacher–student supervision can unify any-view geometry without ornate architectures.” The claim runs against the trend toward specialised multi-task architectures in the 3D vision literature, and matches ViT’s argument about inductive bias: specialised structure substitutes for data and scale, and bounds accuracy once both are available.

SAM 3D: 3D from a single image

Meta AI / FAIR, November 2025 · SAM 3D: 3Dfy Anything in Images · Mechanism 4

A companion suite of two generative models that reconstruct full 3D geometry, texture and spatial layout directly from a single 2D image.

• SAM 3D Objects — a two-stage architecture of diffusion transformers: a first stage predicting coarse 3D shape and object pose, followed by a refinement stage adding realistic texture and surface detail, with a DINOv2 encoder processing the input image.
• SAM 3D Body — a transformer encoder–decoder predicting 3D human pose and mesh parameters directly from a single image, robust under partial occlusion.

Both were trained via a human-and-model-in-the-loop data engine combining synthetic 3D asset pretraining with real-world image alignment — an approach Meta describes as “breaking the 3D data barrier”, since ground-truth 3D annotations for natural images are otherwise extremely scarce.

• At least a 5:1 win rate over prior leading single-image 3D reconstruction methods in human preference tests.
• 28% reduction in Chamfer Distance and 19% improvement in normal consistency on public benchmarks.
• Unlike prior approaches limited to 2.5D visible-surface reconstruction, it predicts complete 3D geometry that can be re-rendered from any viewpoint — including occluded or unseen parts of an object.

Because reconstructed objects and human meshes can be composed into unified 3D scenes, SAM 3D was rapidly integrated into consumer products including Facebook Marketplace’s “View in Room” feature and Meta’s Quest 3 and Horizon Worlds creation tools.

A different point on the data-efficiency curve — with a caveat Prior 3D-from-images methods — MVS, NeRF, splatting — all require many calibrated views. Single-image reconstruction is qualitatively different, and unlocks 3D from legacy photographs where re-capture is impossible. But these are generative priors as much as reconstructors. The occluded geometry is plausible, not measured. The model is inventing a statistically likely back surface, and it will do so confidently whether or not the invention is right.

RF-DETR: the first real-time detector past 60 AP

ICLR 2026 (released March 2025) · Robinson, Robicheaux, Popov, Ramanan & Peri — Roboflow · arXiv:2511.09554 · Mechanism 2

A transformer-based real-time detector built on a DINOv2 vision-transformer backbone, with weight-sharing neural architecture search used to identify optimal accuracy–latency tradeoffs across model sizes. It became the first real-time model — defined as 25+ FPS on an NVIDIA T4 — to exceed 60 mean Average Precision on Microsoft COCO.

• Six sizes from Nano (30.5M parameters, 384×384 input, 2.3 ms latency) to 2XL (126.9M parameters, 880×880 input).
• RF-DETR-2XL reaches 60.1 AP_50:95; RF-DETR-L reaches 56.5 AP_50:95 at 6.8 ms on TensorRT FP16.
• Also leads RF100-VL, a benchmark that measures transfer to custom domains. COCO measures one fixed category set.
• An RF-DETR Segmentation variant, extending the base architecture with a MaskDINO-inspired segmentation head, previewed in October 2025 with the full size family following in January 2026.

It directly overturned the long-standing assumption that DETR-style architectures were inherently too slow for real-time deployment relative to YOLO-family single-stage detectors — five years after DETR made the idea work at all.

Note the benchmark shift RF-DETR reports RF100-VL alongside COCO. COCO AP has saturated, and transfer to a domain the model was not trained on is the quantity these benchmarks now separate. See what the benchmarks hid.

YOLO26: native end-to-end detection

Ultralytics, September 2025 · Mechanism 2

YOLO26 removes non-maximum suppression through a native end-to-end training strategy, in place of an approximation of one. Whereas earlier detectors relied on NMS to filter duplicate overlapping detections, YOLO26 is trained from the outset to learn a one-to-one relationship between objects and outputs, producing a single definitive box per object instance directly.

Three further innovations underpin the shift:

• Removal of Distribution Focal Loss in favour of a simpler, more hardware-friendly bounding-box parameterisation.
• Small-Target-Aware Label Assignment (STAL) paired with Progressive Loss Balancing (ProgLoss) for training stability, particularly benefiting small-object accuracy.
• A novel MuSGD optimiser — a hybrid of SGD and the Muon optimiser, inspired by techniques from large language model training — for faster, more stable convergence.

The performance gains were pronounced specifically on CPU and edge hardware: up to a 43% reduction in CPU inference latency compared to NMS-based baselines, with the nano variant exceeding 40 mAP at roughly 1.5 ms and the extra-large variant reaching approximately 57.5 mAP at 11.5 ms, surpassing YOLO11x while maintaining real-time throughput. It is the first Ultralytics release to natively unify detection, instance segmentation, classification, pose estimation and oriented bounding boxes in one NMS-free architecture.

A five-year arc, completed A comparative retrospective spanning YOLOv5 through YOLO26 characterises the series’ trajectory as a “deployment-first philosophy”, where each generation removed one more architectural or post-processing stage — first accuracy, then efficiency, and finally, with YOLO26, complete inference-pipeline simplicity. DETR proposed set prediction in 2020. It took until 2025 for the idea to arrive, natively, in the lineage optimised for speed.

Embodied vision–language models

A distinct new category emerged explicitly targeting embodied agents operating in the physical world, moving vision–language models beyond static image-and-text reasoning toward navigation and interaction.

Hy-Embodied-VLM-1.0 (Tencent, July 2026) exemplifies the trend: a Mixture-of-Experts vision–language foundation model activating only about 3 billion parameters per token out of roughly 30 billion total, built on a dedicated language backbone paired with a specialised vision encoder. Evaluated across 38 benchmarks spanning embodied perception, physical-world understanding and embodied reasoning, it ranked first on 19 benchmarks and second on 11 more, achieving state-of-the-art results on vision-and-language navigation tasks such as R2R-CE and strong zero-shot performance on object-goal navigation in Matterport3D environments.

The architectural shift being claimed The field increasingly frames this as a move from “vision as a bolt-on adapter” — a pretrained LLM trunk with a vision encoder attached, the LLaVA/Qwen-VL/GPT-4V generation — toward models whose trunk itself functions as a world model that predicts and acts within physical environments, early-fusing all modalities into a single transformer. Whether that reframing delivers is the open question of the next window.

Era recap

Where the decade lands Recognition is promptable by concept. Geometry is recoverable from any number of views, including one, in a single forward pass. Backbones are self-supervised and used frozen. And the last hand-designed post-processing step — NMS — is gone from both major detection lineages. The synthesis page assembles what this means.

Synthesis and reference

The atlas — 29 modules

Every topic, grouped into six blocks. Each module follows the same shape: what it estimates, the image-formation assumptions, classical and deep pipelines, objectives, tricks, failure modes and takeaways.

Foundations · 1–5

The language every later module depends on. Nothing downstream is safe to read before these.

• 1 · Task map & taxonomy — Twelve families of inverse problems, their inputs and outputs, and the dependency structure that explains why upstream advances propagate field-wide.

• 2 · Image formation & cameras — Pinhole projection, distortion, radiometry, noise, rolling shutter. The forward model every inverse problem inverts.

• 3 · Calibration & alignment — DLT, Zhang’s method, rectification, hand–eye, camera–LiDAR and camera–IMU. The family the decade left untouched.

• 4 · Objectives & optimization — The common template, robust M-estimators and their MAP reading, Gauss–Newton, LM, the Schur complement, RANSAC.

• 5 · Multiscale & filtering — Scale space, pyramids, the structure tensor, edge-preserving filters, morphology, and coarse-to-fine optimisation.

• + The shared pipeline — The ten operations common to classical and learned systems, and where each family’s methods sit within them.

Image level · 6–9

• 6 · Restoration — One degradation model for denoising, deblurring, SR, demosaicing and inpainting — plus the perception–distortion tradeoff.

• 7 · Features & primitives — Canny, Harris, DoG/SIFT, ORB, Hough, MSER — and the pipeline that outlived its own operators.

• 8 · Correspondence — Feature-based vs. direct alignment, the transform hierarchy, DLT normalisation, inverse-compositional LK.

• 9 · Stitching & fusion — Global rotation bundle adjustment, projection surfaces, exposure compensation, seam cuts and multiband blending.

Semantic understanding · 10–14

• 10 · Classification & retrieval — Closed-set vs. open-set outputs, metric-learning losses, mining, Mixup/CutMix, calibration.

• 11 · Object detection — Anchors, label assignment, focal loss, the IoU loss family, NMS and its removal by set prediction.

• 12 · Segmentation & matting — Four output types, four loss families, PQ, dense CRFs, and the compositing equation.

• 13 · Keypoints & pose — Heatmaps vs. regression, soft-argmax, PAFs, PnP, 6-DoF pose and symmetry-aware losses.

• 14 · Text & document — CTC vs. attention decoding, TPS rectification, 2D layout encoding, OCR-free models.

Motion & geometry · 15–19

• 15 · Optical & scene flow — Brightness constancy, the aperture problem, Horn–Schunck, Lucas–Kanade, cost volumes, RAFT.

• 16 · Stereo & depth — Why everything is inverse depth, SGM’s two penalties, soft-argmin, and the self-supervised recipe.

• 17 · Pose, SfM, VO & SLAM — Epipolar geometry, triangulation, bundle adjustment, gauge freedom, loop closure, IMU preintegration.

• 18 · Visual tracking — Kalman prediction, Hungarian assignment, Mahalanobis gating, track lifecycle, HOTA.

• 19 · Video understanding — Inflation, factorised 3D convolution, SlowFast, divided attention, tube masking, temporal localisation.

3D & generative · 20–23

• 20 · Point clouds — Permutation invariance, sparse convolution, ICP variants, FPFH, Chamfer vs. EMD, 3D detection.

• 21 · 3D reconstruction — TSDF fusion, marching cubes, Poisson reconstruction, implicit surfaces, completion as generation.

• 22 · Neural rendering — The volume rendering integral, positional encoding, 3DGS density control, and why floaters exist.

• 23 · Generation & translation — VAE/GAN/AR/diffusion, the divergence table, classifier-free guidance, flow matching, CycleGAN.

Multimodal · 24–25

• 24 · Vision–language — Contrastive pretraining, SigLIP’s sigmoid loss, connector architectures, grounding, hallucination control, and the documented geometric-reasoning gap.

• 25 · Scene reasoning & embodied — Scene graphs, affordances, navigation and SPL, active vision, world models, and what changes when perception feeds control.

Cross-cutting · 26–29

• 26 · Photometric consistency — Warp-and-compare, its six assumptions, the engineering corrections, and Monodepth2’s three refinements.

• 27 · Losses & regularization — The complete objective vocabulary, organised by what each loss assumes; multi-task weighting and uncertainty.

• 28 · Engineering tricks — Robust estimation, multiscale, the seven consistency constraints, augmentation, imbalance, post-processing.

• 29 · Evaluation & failure — Metrics and what they hide, leakage, calibration, effective robustness, label noise, ablation discipline.

Threads that cross the atlas

Several ideas recur in modules that otherwise look unrelated. Following one of these across pages is a faster route to understanding than reading in order.

Six mechanisms, one ladder, and what the numbers hid

Forty papers, six recurring mechanisms. This page sets out what recurred, where the supervision came from, which architectures persisted, and what the benchmarks did not measure.

The six mechanisms

1 · Make the easy path identity

If a block can do nothing at zero cost, adding it can only help. ResNet established this for depth, and it never stopped being useful — because the same argument applies to any modification of a working model.

• ControlNet’s zero convolutions make the augmented diffusion model bit-identical to the original at step 0, so training can only add capability.
• DiT’s adaLN-Zero initialises each transformer block as identity.
• LoRA adapters initialise one factor to zero for the same reason.

2 · Predict a set, not a grid

Move duplicate suppression out of inference and into the loss. DETR’s bipartite matching makes the assignment a bijection, so two predictions cannot claim one object, and the NMS stage is removed.

The five-year lag before this reached real-time latency (YOLOv10, YOLO26, RF-DETR) is the clearest example on this site of an idea being correct long before it was deployable.

3 · Manufacture the supervision

The training signal is constructed from the data itself, by contrast, masking, distillation or synthesis. This is the decade’s largest single theme and gets its own section below.

4 · Work in a latent space

Most bits in an image are perceptually irrelevant. Compress them away with an autoencoder trained once, then spend the expensive iterative computation on a tensor roughly 48× smaller. Latent Diffusion is the clean instance; Sora’s spacetime patches extend it to video, and SAM 3D’s first stage applies it to shape.

The generalisation: find the representation in which the hard problem is small.

5 · Heavy encoder, cheap head

Encode once, answer many queries. SAM’s ViT-H encoder runs once per image at ~0.15 s; the prompt decoder answers in ~50 ms, which is what makes interactive segmentation and the annotation loop possible.

The same split underlies every frozen-backbone-plus-linear-probe setup, every VLM connector, and SlowFast’s thin fast pathway. It is also an economic argument: amortise the expensive computation across many uses.

6 · Return to explicit primitives

Where implicit representations are slow, go back to primitives. 3D Gaussian Splatting replaced NeRF’s MLP-in-the-inner-loop with millions of explicit Gaussians and a rasteriser — cost then scales with occupied volume, empty regions contain no primitives, and the scene is editable because it is data rather than weights.

This is the one mechanism that runs against the decade’s general direction, which is why it is worth stating: more learning is not always the answer, and “differentiable” does not have to mean “neural”.

The supervision ladder

Each rung yields more supervision per unit of human effort than the one before.

Read it as an economic argument, not a technical one Annotation cost was addressed by six successive sources of training signal. In the last two rungs a person corrects model output in place of producing it from scratch, a 10–100× difference in rate, with the kind of human input unchanged. That distinction matters when forecasting: the cost curve bent sharply, but it did not go to zero, and every “zero-shot” model on this site was trained on someone’s labelled data at some remove.

Notice also which data terms each rung uses. Rungs 2–4 manufacture a residual out of the data’s own structure — they are consistency constraints promoted to training objectives. Rung 5 borrows a residual from a simulator. Rung 6 is the only one that still needs a human to look at an image.

Architectural outcomes

Observed outcomes

• Attention became the default operator. ConvNeXt showed a modernised ConvNet matches Swin at equal recipe; ConvNeXt V2 showed it can take masked pretraining too.
• The common factor is the plain, scalable block — uniform, few inductive biases, LayerNorm, GELU, residual, trained with a modern recipe at scale. ViT and ConvNeXt are both instances.
• Hierarchy came back everywhere dense prediction mattered — windows, patch merging, spatial reduction.

Confounds worth naming

• Architecture papers changed the training recipe at the same time. ConvNeXt quantified this: +2.7 points from recipe alone.
• Pretraining corpus is rarely held constant across comparisons.
• Compute-matched comparisons are rarer than they should be, and throughput ≠ FLOPs on real hardware.
• State-space alternatives (VMamba, Vision Mamba) have lower asymptotic complexity and have not replaced attention on the reported benchmarks, which bounds how much the operator accounts for.

The pattern across ViT, ConvNeXt and Depth Anything 3 is consistent enough to state as a rule: specialised structure substitutes for data and scale, and bounds accuracy once both are available. Every one of those three papers makes the argument in a different subfield, and the 2025 geometry models make it again against hand-designed multi-view pipelines.

What the benchmarks hid

Measurement problems

• COCO AP has saturated. Recent increments come largely from label-assignment and training-schedule changes.
• Test-set contamination at web-scale pretraining cannot be measured while the corpus is undisclosed, and the corpora grew through the decade.
• “Zero-shot” is under-specified when the pretraining corpus is undisclosed — it means unseen in fine-tuning, not unseen.
• Single-number reporting hides the distribution that matters: small objects, rare classes, hard negatives.

What the field did about it

• Robustness suites — ImageNet-A/R/Sketch/V2, ObjectNet — became standard alongside in-distribution accuracy.
• Transfer benchmarks (VTAB, RF100-VL) measure accuracy on new domains instead of one fixed taxonomy.
• Frozen-feature evaluation across many tasks (DINOv2/v3’s ~60 benchmarks) resists overfitting to any one of them.
• Human-preference evaluation for generation, where reference-based metrics such as FID are known to correlate poorly.

This is also visible in the objective template: the surrogate loss and the reported metric were never the same object, and the decade’s most reliable progress came when a paper closed that gap deliberately — focal loss for imbalance, GIoU for box regression, PQ for panoptic, Lovász for IoU.

Open problems entering 2026

Representation and reasoning

• Compositional and geometric reasoning in VLMs — 30–45% on fine-grained shape tasks is the headline failure of the current paradigm.
• Calibrated uncertainty under distribution shift; confidence remains a ranking signal, not a probability.
• Video as dynamics, not appearance. Generation is excellent; prediction of physical outcomes is not.
• Long-horizon memory beyond a bounded frame buffer.

Method and practice

• Metric geometry from learned models — depth and splats are relative and plausible, not surveyed.
• Surface extraction from explicit radiance primitives remains bolted on.
• Evaluation of promptable models: an unbounded output space means unbounded, unenumerable failure modes.
• Efficiency of the frontier — 6.7B-parameter backbones are not deployable, and distillation loses exactly the dense quality that was hard to get.

Takeaways

1. 2016 removed the depth ceiling, and identity-initialised branches never stopped being useful. ResNet, ControlNet, adaLN-Zero and LoRA are one idea wearing four costumes.
2. The binding constraint moved: architecture → supervision → interface. The constraint a paper addresses places it in that sequence.
3. Set prediction removed hand-designed post-processing. DETR’s idea needed five years to reach YOLO latency — and it got there.
4. Nobody solved annotation; the field routed around it six times. The supervision ladder is the decade’s real throughline.
5. Geometry and semantics converged onto one backbone family, and geometry became feed-forward.
6. Capability outran evaluation. The hardest open problems are now measurement problems.

The central conclusion, restated Computer-vision methods are combinations of representation + correspondence + constraint + robust optimisation. Deep learning changed the representation and learned parts of matching and inference, but the classical constraints remain plainly visible: photometric warping supervises depth and rendering; reprojection error drives calibration and bundle adjustment; smoothness regularizes flow and geometry; pyramids handle scale; RANSAC and robust losses handle outliers; cost volumes represent correspondence; consistency checks handle ambiguity. Treating these explicitly produces a more durable understanding than studying network names in isolation.

Paper index

Every paper discussed on this page, with era and mechanism columns.