Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 88 additions & 6 deletions docs/builtin_type.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ Builtin objects fall into two serialization families:

| Family | Current types | Storage model | Codec policy |
|--------|---------------|---------------|--------------|
| Byte-backed views | `Image`, `DepthImage`, `PointCloud`, `CompressedPointCloud`, `OccupancyGrid`, `Mesh3D`, `VideoFrame` | Header fields live in the SDK struct; payload bytes live behind `Span<const uint8_t>` plus `BufferAnchor`. | No mandatory canonical codec; preserve zero-copy views over ROS, MCAP, compressed image, point-cloud, or plugin-owned payloads. If conversion is unavoidable, allocate a new payload and anchor it. |
| Owned values | `ImageAnnotations`, `FrameTransforms`, `SceneEntities`, `AssetVideo`, `RobotDescription`; future marker types | SDK structs own their vectors/strings/scalars directly. | Add explicit codecs when canonical bytes are needed. Codecs serialize the owned value to the protobuf-wire payload described by the `.proto` contract, using shared private wire primitives. `RobotDescription` carries source-format text as-is (no canonical codec) — the format hint distinguishes URDF / SDF / MJCF. |
| Byte-backed views | `Image`, `DepthImage`, `PointCloud`, `CompressedPointCloud`, `OccupancyGrid`, `OccupancyGridUpdate`, `Mesh3D`, `VideoFrame` | Header fields live in the SDK struct; payload bytes live behind `Span<const uint8_t>` plus `BufferAnchor`. | No mandatory canonical codec; preserve zero-copy views over ROS, MCAP, compressed image, point-cloud, or plugin-owned payloads. If conversion is unavoidable, allocate a new payload and anchor it. |
| Owned values | `ImageAnnotations`, `FrameTransforms`, `SceneEntities`, `AssetVideo`, `RobotDescription`, `CameraInfo`; future marker types | SDK structs own their vectors/strings/scalars directly. | Add explicit codecs when canonical bytes are needed. Codecs serialize the owned value to the protobuf-wire payload described by the `.proto` contract, using shared private wire primitives. `RobotDescription` carries source-format text as-is (no canonical codec) — the format hint distinguishes URDF / SDF / MJCF. |

Canonical `.proto` files live under `pj_base/proto/pj` and act as the wire
format contract. One file per top-level message, each named after its message
Expand Down Expand Up @@ -120,6 +120,9 @@ annotations, frame transforms, or no builtin object.
| `kSceneEntities` | `PJ::sdk::SceneEntities` | Procedural 3D scene primitives (arrows, cubes, lines, text, …). |
| `kAssetVideo` | `PJ::sdk::AssetVideo` | File-backed video reference plus typed playback metadata. |
| `kRobotDescription` | `PJ::sdk::RobotDescription` | Raw URDF/SDF/MJCF text + format hint. |
| `kCameraInfo` | `PJ::sdk::CameraInfo` | Pinhole camera calibration (intrinsics K, distortion D, rectification R, projection P). |
| `kOccupancyGridUpdate` | `PJ::sdk::OccupancyGridUpdate` | Incremental sub-rectangle patch for a previously-published `OccupancyGrid`. |
| `kLog` | `PJ::sdk::Log` | Textual log message (severity level + text + originating name). |

`BuiltinObject` is `std::any`. Producers store a concrete builtin value in it;
consumers recover the concrete type with `std::any_cast<T>(&object)` or ask
Expand Down Expand Up @@ -315,6 +318,34 @@ renderer cares about cell-to-world placement, not pixel layout.
`pj_base/builtin/occupancy_grid_codec.hpp` serializes and deserializes this
type using the canonical `PJ.OccupancyGrid` protobuf wire format.

## OccupancyGridUpdate

`OccupancyGridUpdate` is the incremental counterpart to `OccupancyGrid`: a
row-major sub-rectangle patch into a previously-published base grid (ROS
`map_msgs/OccupancyGridUpdate`, e.g. `<base>/costmap_updates`).

It deliberately carries **no** `origin` / `resolution` — a patch is not
independently placeable. A stateful consumer pairs the update with its base
grid (by topic-name convention, `<base>/costmap_updates` ↔ `<base>/costmap`)
and positions it at the base's `origin + (x, y) * resolution`. This keeps the
producer stateless and cross-topic-blind; all accumulation / placement lives in
the consumer. The patch is a self-contained snapshot at its own timestamp, so it
stores and decodes like any other object (no replay required at decode time).

| Field | Type | Notes |
|-------|------|-------|
| `timestamp_ns` | `Timestamp` | Timestamp of the update. |
| `frame_id` | `std::string` | Must match the base grid's frame. |
| `x` | `int32_t` | Column offset (cells) of the patch top-left into the base grid. |
| `y` | `int32_t` | Row offset (cells) of the patch top-left into the base grid. |
| `width` | `uint32_t` | Patch width in cells. |
| `height` | `uint32_t` | Patch height in cells. |
| `data` | `Span<const uint8_t>` | Row-major signed-8-bit cells; size must equal `width * height`. |
| `anchor` | `BufferAnchor` | Keeps `data` alive when it references shared storage. |

`pj_base/builtin/occupancy_grid_update_codec.hpp` serializes and deserializes
this type using the canonical `PJ.OccupancyGridUpdate` protobuf wire format.

## CompressedPointCloud

`CompressedPointCloud` carries a point cloud delivered in a format-specific
Expand Down Expand Up @@ -432,8 +463,8 @@ bundles heterogeneous primitives sharing a `frame_id` and timestamp;

Use `SceneEntities` when the value is procedural 3D scene content
expressible as a small set of primitives: arrows, cubes, spheres,
cylinders, line strips/loops/lists, triangles, text labels, or coordinate
axes glyphs.
cylinders, line strips/loops/lists, triangles, text labels, coordinate
axes glyphs, or model (mesh asset) references.

| Field on `SceneEntity` | Type | Notes |
|------------------------|------|-------|
Expand All @@ -442,7 +473,12 @@ axes glyphs.
| `id` | `std::string` | Republishing with the same `(topic, id)` replaces the previous entity. |
| `lifetime_ns` | `int64_t` | `0` means persist until replaced; otherwise expire `lifetime_ns` after `timestamp`. |
| `frame_locked` | `bool` | When true, track `frame_id` as it moves; when false, stamp into the fixed frame at publish time. |
| `arrows` / `cubes` / `spheres` / `cylinders` / `lines` / `triangles` / `texts` / `axes` | `std::vector<…Primitive>` | Heterogeneous primitive lists. |
| `arrows` / `cubes` / `spheres` / `cylinders` / `lines` / `triangles` / `texts` / `axes` / `models` | `std::vector<…Primitive>` | Heterogeneous primitive lists. `models` references a mesh asset by `url` or inline `data`. |

The `SceneEntities` batch also carries `deletions` (`std::vector<SceneEntityDeletion>`):
removal commands that let a snapshot-based producer express the removal half of a
stateful stream (e.g. ROS Marker `DELETE` / `DELETEALL`). A deletion is either
`kMatchingId` (remove the entity with the given `id`) or `kAll` (clear the topic).

Each primitive carries its own `Pose`, geometry-specific size or shape
fields, and color (or per-vertex colors, where applicable). See
Expand Down Expand Up @@ -487,6 +523,51 @@ Design notes:
`<robot>`) before emission. Generic `std_msgs/String` payloads on unrelated
topics should not surface as RobotDescription.

## CameraInfo

`CameraInfo` carries pinhole camera calibration — intrinsics, distortion,
rectification, and projection — for one camera frame (ROS
`sensor_msgs/CameraInfo`). Consumers use it to draw camera frustums, back-project
depth pixels into 3D, and rectify or overlay onto images.

Like `OccupancyGridUpdate`, it is correlated to its image / depth topic by
topic-name convention (`<ns>/camera_info` ↔ `<ns>/image_raw`); the object itself
carries no topic linkage. It is an owned value (small matrices and a distortion
vector, no byte blob), so no `BufferAnchor` is needed.

| Field | Type | Notes |
|-------|------|-------|
| `timestamp_ns` | `Timestamp` | Timestamp associated with this calibration. |
| `frame_id` | `std::string` | Camera optical frame. |
| `width` | `uint32_t` | Image width in pixels. |
| `height` | `uint32_t` | Image height in pixels. |
| `distortion_model` | `std::string` | e.g. `plumb_bob`, `rational_polynomial`, `equidistant`; empty when rectified. |
| `D` | `std::vector<double>` | Distortion coefficients; size depends on the model. |
| `K` | `std::array<double, 9>` | 3x3 row-major intrinsics `[fx 0 cx; 0 fy cy; 0 0 1]`. |
| `R` | `std::array<double, 9>` | 3x3 row-major rectification (identity for monocular). |
| `P` | `std::array<double, 12>` | 3x4 row-major projection / camera matrix. |

Sub-window fields (binning, ROI) from `sensor_msgs/CameraInfo` are intentionally
omitted; they are additive later if a consumer needs them.
`pj_base/builtin/camera_info_codec.hpp` serializes and deserializes this type
using the canonical `PJ.CameraInfo` protobuf wire format.

## Log

`Log` is a single textual log message, for a log/console panel. It mirrors the
core of Foxglove's `Log` schema (and `rcl_interfaces/Log` / `rosgraph_msgs/Log`).

| Field on `Log` | Type | Notes |
|----------------|------|-------|
| `timestamp_ns` | `Timestamp` | Time of the log message. |
| `level` | `Log::Level` | `kUnknown`/`kDebug`/`kInfo`/`kWarning`/`kError`/`kFatal` (values match Foxglove). |
| `message` | `std::string` | Log text. |
| `name` | `std::string` | Originating process / node / logger name. |

Foxglove's source-location fields (`file`, `line`) are intentionally omitted.
`pj_base/builtin/log_codec.hpp` serializes and deserializes this type using the
canonical `PJ.Log` protobuf wire format.

## Conversion Examples

| Source type | Canonical builtin type | Conversion intent |
Expand All @@ -503,7 +584,8 @@ Design notes:
| Detection or tracking message | `ImageAnnotations` | Convert boxes, points, circles, and labels into pixel-space primitives. |
| ROS `tf2_msgs/TFMessage` | `FrameTransforms` | Convert transform batches into named parent/child frame relationships. |
| ROS `std_msgs/String` on `/robot_description` (or matching name) carrying URDF XML | `RobotDescription` | Validate root element matches `format`, then carry the raw text + format hint. No mesh resolution at parse time. |
| ROS `std_msgs/String` on `/robot_description` (or matching name) carrying URDF XML | `RobotDescription` | Validate root element matches `format`, then carry the raw text + format hint. No mesh resolution at parse time. |
| ROS `sensor_msgs/CameraInfo` | `CameraInfo` | Map K / D / R / P plus dimensions; correlate to the image topic by name. Sub-window (binning / ROI) is dropped. |
| ROS `map_msgs/OccupancyGridUpdate` | `OccupancyGridUpdate` | Forward the cell-space patch (`x`/`y`/`width`/`height` + bytes); the consumer pairs it with the base grid and supplies origin/resolution. |

The builtin type is the boundary object. After conversion, consumers should not
need to know which third-party schema produced it.
6 changes: 6 additions & 0 deletions pj_base/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,16 @@

add_library(pj_base STATIC
src/builtin/asset_video_codec.cpp
src/builtin/camera_info_codec.cpp
src/builtin/compressed_point_cloud_codec.cpp
src/builtin/depth_image_codec.cpp
src/builtin/frame_transforms_codec.cpp
src/builtin/image_annotations_codec.cpp
src/builtin/image_codec.cpp
src/builtin/log_codec.cpp
src/builtin/mesh3d_codec.cpp
src/builtin/occupancy_grid_codec.cpp
src/builtin/occupancy_grid_update_codec.cpp
src/builtin/point_cloud_codec.cpp
src/builtin/scene_entities_codec.cpp
src/builtin/video_frame_codec.cpp
Expand Down Expand Up @@ -83,6 +86,9 @@ if(PJ_BUILD_TESTS)
tests/point_cloud_codec_test.cpp
tests/compressed_point_cloud_codec_test.cpp
tests/occupancy_grid_codec_test.cpp
tests/occupancy_grid_update_codec_test.cpp
tests/camera_info_codec_test.cpp
tests/log_codec_test.cpp
tests/mesh3d_codec_test.cpp
tests/video_frame_codec_test.cpp
tests/scene_entities_codec_test.cpp
Expand Down
30 changes: 30 additions & 0 deletions pj_base/include/pj_base/builtin/builtin_object.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,16 @@
#include <string_view>

#include "pj_base/builtin/asset_video.hpp"
#include "pj_base/builtin/camera_info.hpp"
#include "pj_base/builtin/compressed_point_cloud.hpp"
#include "pj_base/builtin/depth_image.hpp"
#include "pj_base/builtin/frame_transforms.hpp"
#include "pj_base/builtin/image.hpp"
#include "pj_base/builtin/image_annotations.hpp"
#include "pj_base/builtin/log.hpp"
#include "pj_base/builtin/mesh3d.hpp"
#include "pj_base/builtin/occupancy_grid.hpp"
#include "pj_base/builtin/occupancy_grid_update.hpp"
#include "pj_base/builtin/point_cloud.hpp"
#include "pj_base/builtin/robot_description.hpp"
#include "pj_base/builtin/scene_entities.hpp"
Expand All @@ -56,6 +59,9 @@ enum class BuiltinObjectType : uint16_t {
kSceneEntities = 11, ///< sdk::SceneEntities — procedural 3D scene primitives.
kAssetVideo = 12, ///< sdk::AssetVideo — file-backed video reference + playback metadata.
kRobotDescription = 13, ///< sdk::RobotDescription — raw URDF/SDF/MJCF text + format hint.
kCameraInfo = 14, ///< sdk::CameraInfo — pinhole camera calibration (K/D/R/P).
kOccupancyGridUpdate = 15, ///< sdk::OccupancyGridUpdate — incremental sub-rectangle patch for an OccupancyGrid.
kLog = 16, ///< sdk::Log — textual log message (level + text + name).
};

/// A-priori classification of a schema. Currently carries only the type;
Expand Down Expand Up @@ -94,6 +100,12 @@ struct SchemaClassification {
return "kAssetVideo";
case BuiltinObjectType::kRobotDescription:
return "kRobotDescription";
case BuiltinObjectType::kCameraInfo:
return "kCameraInfo";
case BuiltinObjectType::kOccupancyGridUpdate:
return "kOccupancyGridUpdate";
case BuiltinObjectType::kLog:
return "kLog";
}
return "kNone";
}
Expand Down Expand Up @@ -140,6 +152,15 @@ struct SchemaClassification {
if (s == "kRobotDescription") {
return BuiltinObjectType::kRobotDescription;
}
if (s == "kCameraInfo") {
return BuiltinObjectType::kCameraInfo;
}
if (s == "kOccupancyGridUpdate") {
return BuiltinObjectType::kOccupancyGridUpdate;
}
if (s == "kLog") {
return BuiltinObjectType::kLog;
}
return std::nullopt;
}

Expand Down Expand Up @@ -189,6 +210,15 @@ using BuiltinObject = std::any;
if (t == typeid(RobotDescription)) {
return BuiltinObjectType::kRobotDescription;
}
if (t == typeid(CameraInfo)) {
return BuiltinObjectType::kCameraInfo;
}
if (t == typeid(OccupancyGridUpdate)) {
return BuiltinObjectType::kOccupancyGridUpdate;
}
if (t == typeid(Log)) {
return BuiltinObjectType::kLog;
}
return BuiltinObjectType::kNone;
}

Expand Down
45 changes: 45 additions & 0 deletions pj_base/include/pj_base/builtin/camera_info.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/**
* @file camera_info.hpp
* @brief Pinhole camera calibration (intrinsics + distortion) for a camera frame.
*
* CameraInfo is a small owned builtin: it carries calibration matrices and
* distortion coefficients directly, with no byte blob, so no BufferAnchor is
* needed. It is correlated to an image / depth topic by convention — a consumer
* pairs `<ns>/camera_info` with `<ns>/image_raw` — so the canonical object
* itself carries no topic linkage. Sub-window fields (binning, ROI) are
* intentionally omitted; they are additive later if a consumer needs them.
*/
// Copyright 2026 Davide Faconti
// SPDX-License-Identifier: Apache-2.0

#pragma once

#include <array>
#include <cstdint>
#include <string>
#include <vector>

#include "pj_base/types.hpp"

namespace PJ {
namespace sdk {

/// Pinhole camera calibration. Mirrors the calibration payload of
/// sensor_msgs/CameraInfo (K/D/R/P + distortion model), expressed as canonical
/// PJ vocabulary rather than a ROS-specific type. Matrices are row-major.
struct CameraInfo {
Timestamp timestamp_ns = 0;
std::string frame_id; ///< Camera optical frame.
uint32_t width = 0; ///< Image width in pixels.
uint32_t height = 0; ///< Image height in pixels.
std::string distortion_model; ///< e.g. "plumb_bob", "rational_polynomial", "equidistant".
std::vector<double> D; ///< Distortion coefficients; size depends on the model.
std::array<double, 9> K{}; ///< 3x3 intrinsics [fx 0 cx; 0 fy cy; 0 0 1].
std::array<double, 9> R{}; ///< 3x3 rectification (identity for monocular).
std::array<double, 12> P{}; ///< 3x4 projection / camera matrix.

bool operator==(const CameraInfo&) const = default;
};

} // namespace sdk
} // namespace PJ
24 changes: 24 additions & 0 deletions pj_base/include/pj_base/builtin/camera_info_codec.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#pragma once
// Copyright 2026 Davide Faconti
// SPDX-License-Identifier: Apache-2.0

#include <cstddef>
#include <cstdint>
#include <string_view>
#include <vector>

#include "pj_base/builtin/camera_info.hpp"
#include "pj_base/expected.hpp"

namespace PJ {

inline constexpr std::string_view kSchemaCameraInfo = "PJ.CameraInfo";

/// Serializes sdk::CameraInfo to canonical PJ.CameraInfo wire bytes
/// (see pj_base/proto/pj/CameraInfo.proto).
[[nodiscard]] std::vector<uint8_t> serializeCameraInfo(const sdk::CameraInfo& info);

/// Decodes canonical PJ.CameraInfo wire bytes into an owned sdk::CameraInfo.
[[nodiscard]] Expected<sdk::CameraInfo> deserializeCameraInfo(const uint8_t* data, size_t size);

} // namespace PJ
44 changes: 44 additions & 0 deletions pj_base/include/pj_base/builtin/log.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/**
* @file log.hpp
* @brief A single textual log message (level + text + originating name).
*
* Log is a small owned builtin — no byte blob, no BufferAnchor. It mirrors the
* core of Foxglove's Log schema (and rcl_interfaces/Log / rosgraph_msgs/Log),
* expressed as canonical PJ vocabulary. The Foxglove source-location fields
* (`file`, `line`) are intentionally omitted.
*/
// Copyright 2026 Davide Faconti
// SPDX-License-Identifier: Apache-2.0

#pragma once

#include <cstdint>
#include <string>

#include "pj_base/types.hpp"

namespace PJ {
namespace sdk {

/// A textual log message at a point in time.
struct Log {
/// Severity level. Values match Foxglove's Log.Level.
enum class Level : uint8_t {
kUnknown = 0,
kDebug = 1,
kInfo = 2,
kWarning = 3,
kError = 4,
kFatal = 5,
};

Timestamp timestamp_ns = 0;
Level level = Level::kUnknown;
std::string message; ///< Log text.
std::string name; ///< Originating process / node / logger name.

bool operator==(const Log&) const = default;
};

} // namespace sdk
} // namespace PJ
24 changes: 24 additions & 0 deletions pj_base/include/pj_base/builtin/log_codec.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#pragma once
// Copyright 2026 Davide Faconti
// SPDX-License-Identifier: Apache-2.0

#include <cstddef>
#include <cstdint>
#include <string_view>
#include <vector>

#include "pj_base/builtin/log.hpp"
#include "pj_base/expected.hpp"

namespace PJ {

inline constexpr std::string_view kSchemaLog = "PJ.Log";

/// Serializes sdk::Log to canonical PJ.Log wire bytes
/// (see pj_base/proto/pj/Log.proto).
[[nodiscard]] std::vector<uint8_t> serializeLog(const sdk::Log& log);

/// Decodes canonical PJ.Log wire bytes into an owned sdk::Log.
[[nodiscard]] Expected<sdk::Log> deserializeLog(const uint8_t* data, size_t size);

} // namespace PJ
Loading
Loading