araponga: estimating 3D orientations from 2D landmarks in R

araponga is for you if you need to estimate the 3D orientations (left—right, up—down) of a directed object from two landmarks labeled in single images or video frames. It identifies the orientations compatible with the observed 2D landmark geometry and any additional information you can provide about the imaging geometry or object, such as view elevation or unprojected length.

The central problem is that a pair of 2D landmarks does not uniquely determine a 3D angle: the same apparent orientation in an image can be produced by many combinations of true orientation, camera position, and viewing angle. Instead of returning a single potentially misleading estimate, araponga returns the set of 3D angles compatible with the observed landmarks and user-provided constraints.

Installation

You can install araponga from CRAN:

install.packages("araponga")

Then load the package:

library(araponga)

You can install the development version from GitHub with:

install.packages("pak")
pak::pak("jocateme/araponga")

The development version may include changes that have not yet been released on CRAN.

Basic workflow

The general workflow is:

2D landmarks -> projected geometry -> constrained 3D search -> compatible 3D angles

The package uses fixed angle conventions for pitch, yaw, and view elevation. To display these conventions at any time, use:

conventions()

First, use two landmarks — a base and a tip — to describe the object’s 2D projection:

observed <- project2d.from.xy(
  x_tip = 10,
  y_tip = 20,
  x_base = 0,
  y_base = 0,
  plot = TRUE
)

observed

The returned araponga2d object contains the landmark coordinates, projected 2D pitch, and projected length.

We can then ask, for example, which 3D pitches (up—down orientations) are compatible with the observation, given assumptions about view elevation, yaw, and landmark labeling error:

possible_pitches <- find.pitch(
  observed2d = observed,
  label_error = 1, # ±1 px labeling error
  candidate_view_elevations = -35:-25, # seen from 25-35° below
  candidate_yaws = -30:0 # side-on to slightly facing camera
)

plot.angles(possible_pitches, type = "pitch")

Using the same observation, we can instead ask which yaws (left—right orientations) are compatible:

possible_yaws <- find.yaw(
  observed2d = observed,
  label_error = 1,
  candidate_view_elevations = -35:-25,
  candidate_pitches = 0:90 # horizontal to straight up
)

plot.angles(possible_yaws, type = "yaw")

An important point is that the candidate_... arguments define which 3D configurations are considered plausible before compatible angles are returned.

If the unprojected length of the object is also known or can be estimated, it can be supplied with full_length. In that case, araponga uses both projected direction and projected length to constrain the compatible 3D orientations:

possible_pitches <- find.pitch(
  observed2d = observed,
  label_error = 1,
  full_length = c(22, 24),
  candidate_view_elevations = -35:-25,
  candidate_yaws = -30:0
)

full_length must be expressed in the same units as the observed 2D coordinates. Thus, when landmarks are measured in pixels, full_length must represent the object’s unprojected length in pixels rather than its physical length directly.

Two helper functions can be used to estimate this quantity. full_length.from.camera() converts physical length to an unprojected image length using camera distance and camera geometry, while full_length.from.reference() estimates it from a reference segment of known physical length and pixel length.

For workflows in which projected measurements have already been calculated, find.3d() and its wrappers can also accept pitch2d and/or length2d constraints directly, without requiring the original landmarks. See the vignette and function documentation for this alternative input route.

Main functions

araponga includes functions for:

• describing observed 2D projections and uncertainty:
  • project2d.from.xy()
  • pitch2d.w.error()
• estimating unprojected image length:
  • full_length.from.camera()
  • full_length.from.reference()
• finding compatible 3D angles:
  • find.3d()
  • find.pitch()
  • find.yaw()
• visualizing angles:
  • plot.angles()
  • conventions()
• working with yaw sets:
  • trim.yaws()
  • summarize.yaws()
• projecting and rotating 3D coordinates:
  • project2d.from.3d()
  • rotate3d(), Rx(), Ry(), Rz()

Learn more

For package conventions and a full worked example, see the vignette:

vignette("araponga")