Skip to content

feat(sensors): add the ideal orthographic camera model - #190

Open
janickm wants to merge 1 commit into
NVIDIA:mainfrom
janickm:dev/janickm/ideal-orthographic-camera-model
Open

janickm wants to merge 1 commit into
NVIDIA:mainfrom
janickm:dev/janickm/ideal-orthographic-camera-model

Conversation

@janickm

@janickm janickm commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds an ideal orthographic camera model (ideal-orthographic): a
distortion-free parallel projection that drops the camera-frame depth instead of
dividing by it, so a point's image location does not depend on it.

This is NCore's first non-central camera model. Its rays are parallel and
share no common origin, so a 3d direction no longer identifies a ray:
unprojection returns 6d [origin, direction] rays, matching the layout
WorldRaysReturn already uses. CameraModel.camera_ray_dim reports which
representation a model uses (3 or 6).

Why

Bird's-eye-view raster data is naturally described by a parallel projection, and
NCore had no way to express one: FTheta, pinhole and fisheye are all central
projections. Without it, consumers hand-roll their own projection alongside
NCore's camera models, which then does not participate in the shared pose,
rolling-shutter and serialization machinery.

Orthographic is a standard projection rather than a bespoke one, e.g. OpenUSD's
GfCamera::Projection {Perspective, Orthographic} and OpenGL's glOrtho.

Design notes

The ray interface is asymmetric, which keeps the change small.
camera_rays_to_image_points receives an unnormalized camera-frame point for
every model (for a central model a point and a direction coincide up to scale;
a non-central model projects the point itself). So the world_points_to_*
family and the rolling-shutter solver work unchanged, and rolling shutter
comes for free
. Only unprojection needed generalizing.

Central models are provably unaffected. Per-ray origins are now carried
through the sensor pose as R @ origin + t instead of broadcasting the pose
translation. A central model's origins are zero, so the same expression
reproduces the previous broadcast exactly, with no branch. There is a regression
test asserting this for all three central models.

Two combinations are refused, because a parallel bundle makes them
ill-defined rather than merely unimplemented:

  • External distortion deflects each ray individually and would not preserve
    the parallel bundle. Rejected at construction: a ray-dimensionality guard
    alone would let the pairing survive every forward projection and only surface
    on unprojection.
  • Rectification against a central model has no depth-independent solution, as
    there is no common centre to pivot about. Rectifying two non-central models is
    well-defined but not implemented yet (NotImplementedError).

paraxial_pinhole_geometry() raises via the documented opt-out: an orthographic
camera's focal length is infinite, so it has no pinhole approximation and
IdealPinholeCameraModelParameters.from_source() correctly refuses it.

Naming. pixels_per_unit is the orthographic counterpart of a pinhole's
focal_length, occupying the same place in the projection but differing in
dimension: a pinhole consumes the dimensionless x/z, this consumes x
directly. Named per-unit rather than per-meter because coordinates may be
UNITLESS (e.g. SfM reconstructions). The Ideal prefix mirrors
IdealPinholeCameraModelParameters and leaves the unqualified Orthographic
family name free for a distorted telecentric sibling, avoiding a repeat of the
retired pinhole -> opencv-pinhole identifier alias.

No serialized format changed. The model registers through the existing
register_camera_model / register_camera_model_parameters hooks, and the
discriminator was already written from type().

Also in this PR

  • Renamed the forward projection to camera_points_to_image_points /
    camera_points_to_pixels (old names kept as documented-deprecated
    forwarders). See below.
  • ExternalDistortionModel now guards the ray representation in its public
    methods and dispatches to _impl hooks, mirroring CameraModel. See the note
    below.
  • camera_ray_dim is a ClassVar, since the ray representation is a property of
    the model type rather than of an instance.
  • Extracted the image-domain scale/resolution prelude that was duplicated across
    all four transform() implementations (and dropped a redundant double
    .astype(np.uint64) in the FTheta one).

Renaming camera_rays_to_image_points

Raised in review. The forward method's argument was always an unnormalized
camera-frame point, never a ray. With only central models the distinction
was unobservable, because every central projection is scale-invariant:

Model Forward math Magnitude
ideal / OpenCV pinhole cam_rays[:,:2] / cam_rays[:,2:3] cancels
FTheta / OpenCV fisheye atan2(xy_norm, z), then delta/xy_norm * xy cancels
ideal orthographic cam_rays[:,:2] * pixels_per_unit is the answer

A non-central projection has no such freedom: the point's x and y are
precisely what is being projected, so passing a normalized direction silently
yields a different image location rather than an error. That promotes the name
from merely loose to actively misleading, so it is fixed here rather than after
release:

  • camera_rays_to_image_points -> camera_points_to_image_points
  • camera_rays_to_pixels -> camera_points_to_pixels

Old names remain as forwarders carrying a .. deprecated:: directive, matching
the docstring-only deprecation style already used in this module (there is no
runtime warnings.warn anywhere in the repo, so none was introduced).

Note the unprojection side keeps its name: image_points_to_camera_rays
genuinely returns rays, and for a non-central model those rays are the whole
point of the 6d representation.

Note on the ExternalDistortionModel refactor

The refactor renames the two abstract methods an external distortion model
implements (distort_camera_rays / undistort_camera_rays ->
_distort_camera_rays_impl / _undistort_camera_rays_impl), so that the public
methods can hold the ray-representation guard.

Not flagged as a breaking change: there are no out-of-tree implementations of
the abstract base, and models deriving from a concrete class such as
BivariateWindshieldModel inherit the new hooks unchanged. Were an out-of-tree
subclass of the abstract base to exist, it would fail loudly at instantiation
(Can't instantiate abstract class ...) rather than silently.

Testing

  • Projection: window-fraction correctness across the full extent, depth
    invariance, out-of-window validity, and that points behind the camera stay
    valid (no frustum, unlike every other model).
  • 6d rays: distinct origins with a shared direction, and the
    unprojection/projection round trip.
  • World rays: per-ray origins under a rotated and translated pose, for both the
    static and rolling-shutter paths; plus a regression guard that the three
    central models are bit-identical.
  • Guards: distortion rejected at construction, paraxial_pinhole_geometry() and
    from_source() raising, non-central rays rejected by the distortion model,
    and both Rectificator cases.
  • Serialization round trip, transform(), and factory dispatch.
  • Point semantics: central projections are scale-invariant, the orthographic one
    is not, and the deprecated aliases agree with the renamed methods. The alias
    test covers the orthographic model specifically, since it is the only one
    whose result would change if an alias perturbed its argument in passing.

bazel test //... --config=no-gpu passes (36/36, Python 3.8 and 3.11),
bazel build //... is clean under the ty aspect, and bazel run //:format.check reports no violations.

Docs

New "Central and Non-Central Models" and "Ideal Orthographic Camera Model"
sections in sensor_models.rst, a "Camera Rays" section in conventions.rst
documenting the 3d/6d representations, caveats in the external-distortion and
rectification sections, and formats.rst now lists ideal-orthographic (plus
the previously missing ideal-pinhole).

@copy-pr-bot

copy-pr-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@janickm janickm self-assigned this Sep 18, 2026
Comment thread docs/data/conventions.rst Outdated
Comment thread docs/data/conventions.rst Outdated
Comment thread ncore/impl/sensors/camera.py Outdated
)

@property
def camera_ray_dim(self) -> int:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be a class-wide or even static property?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class-wide, agreed — it never depends on instance state. Now a ClassVar[int], following the precedent at types.py:1568. Verified it works on an nn.Module subclass: readable from both class and instance, doesn't land in state_dict(), survives .to(). Test added.

Comment thread ncore/impl/sensors/camera.py
Adds `IdealOrthographicCameraModel` / `IdealOrthographicCameraModelParameters`,
a distortion-free parallel projection that drops the camera-frame depth instead
of dividing by it. Serialized as `ideal-orthographic`.

This is NCore's first *non-central* camera model: its rays are parallel and
share no common origin, so a 3d direction no longer identifies a ray.
Unprojection therefore returns 6d `[origin, direction]` rays, matching the
layout `WorldRaysReturn` already uses, and `camera_ray_dim` reports which
representation a model uses. Origins and directions alike are relative to the
extrinsic camera frame's origin.

The two directions of the ray interface are asymmetric, which is what keeps the
change small: projection receives an unnormalized camera-frame *point* for every
model, so the `world_points_to_*` family and the rolling-shutter solver work
unchanged, and rolling shutter comes for free. Only the unprojection path needed
generalizing, where per-ray origins are now carried through the sensor pose as
`R @ origin + t` rather than broadcasting the pose translation. Central models
are unaffected: their origins are zero, so the same expression reproduces the
previous broadcast exactly.

Two combinations are refused because a parallel ray bundle makes them
ill-defined rather than merely unimplemented:

- External distortion deflects each ray individually and so would not preserve
  the parallel bundle. Rejected when the model is constructed, since the forward
  path alone would not surface the mismatch.
- Rectification against a central model has no depth-independent solution, as
  there is no common centre to pivot about. Rectifying two non-central models is
  well-defined but not implemented yet.

`paraxial_pinhole_geometry()` raises via the documented opt-out: an orthographic
camera's focal length is infinite, so it has no pinhole approximation.

The intrinsics are `principal_point` plus `pixels_per_unit`, the orthographic
counterpart of a pinhole's `focal_length`, differing in that its input carries
scene units. Named per-unit rather than per-meter because coordinates may be
`UNITLESS`. `window_min()` / `window_max()` / `from_window()` restate the same
intrinsics as the metric window the camera views, for consumers working in
normalized image coordinates.

Renames the forward projection to say what it takes:

  camera_rays_to_image_points -> camera_points_to_image_points
  camera_rays_to_pixels       -> camera_points_to_pixels

The argument was always an unnormalized camera-frame point, never a ray. With
only central models the distinction was unobservable, because every central
projection is scale-invariant: the pinholes divide by depth and the FTheta and
fisheye models take an angle about the projection centre, so any point along a
given ray projects identically and the magnitude is discarded. An orthographic
projection has no such freedom - the point's x and y are precisely the quantity
being projected - so passing a normalized direction silently yields a different
image location rather than an error. The old names are kept as
documented-deprecated forwarders, following the docstring-only deprecation style
already used in this module. Unprojection keeps its name, as
`image_points_to_camera_rays` genuinely returns rays.

Also refactors `ExternalDistortionModel` to guard the ray representation in its
public methods and dispatch to `_distort_camera_rays_impl` /
`_undistort_camera_rays_impl`, mirroring `CameraModel`, and extracts the
image-domain scale/resolution prelude duplicated across all four `transform()`
implementations.

Note the `ExternalDistortionModel` refactor renames the two abstract methods an
external distortion model implements. There are no out-of-tree implementations
of the abstract base, and models deriving from a concrete class such as
`BivariateWindshieldModel` inherit the new hooks unchanged. No serialized format
changed.
@janickm
janickm force-pushed the dev/janickm/ideal-orthographic-camera-model branch from e95d3ba to 75074c1 Compare September 18, 2026 10:04
@janickm
janickm marked this pull request as ready for review September 18, 2026 10:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant