User Interface

Overview

The GUI consists of a primary (world) view, secondary (camera and depth) views, a frame selection panel, and various other ancillary views and panels. The world view is a three dimensional view that shows the reconstruction results and camera poses in the computed world coordinate system. All other views and information panes are dockable, and may be rearranged, resized, detached or closed (hidden) according to user preference. Closed panes may be reopened via the View menu, or using the context pop-up menu of the main menu bar or title bar of any pane.

The secondary panes are:

Camera Selection:
Provides controls to view and select the active frame, and to cycle through the same in a manner similar to video playback or a slide show.
Camera View:
Shows the imagery from the current frame, along with various two dimensional geometry. If a camera model is available, also shows three dimensional geometry mapped into the camera’s point of view, as well as estimation residuals.
Depth Map View:
Shows the most recent estimated depth map as a two dimensional image.
Metadata:
Shows the metadata values associated with the current frame image.
Ground Control Points:
Shows a list of ground control points and/or camera registration points, as well as details (including geodetic location) of the selected point. Also contains controls for editing or deleting points.
Log Viewer:
Shows the log output for the application, particularly from running algorithms.

World View

The world view provides a three dimensional view showing all three dimensional products in a common coordinate system. It is especially useful for viewing the spatial relationships between cameras and landmarks, ground control points, dense point clouds, or surface meshes.

Landmarks are displayed as points, while cameras are displayed as frustums of semi-arbitrary length, with the active camera highlighted (see Camera Options). The “camera path”, which is a polyline connecting the camera centers, can also be displayed. Ground control points are displayed as “jacks”; three short lines meeting at right angles in 3D.

Additionally, the world view can display a representation of the ground plane (i.e. the local coordinate plane z = 0), and can display the image for the active frame projected to that plane. These projections result in a stabilized view of the video if the scene is largely planar and that plane has been aligned with z = 0. Products from video with geospatial metadata are automatically translated to the z = 0 plane. For data without geospatial metadata, the Estimate Cameras/Landmarks tool attempts to fit a plane to the landmarks as it estimates them to reorient the landmarks to z = 0.

The world view also supports visualization of the depth image for the current frame as either a dense RGB point cloud or a surface mesh. It also can render a three dimensional surface mesh; either a “volume mesh”, extracted from the level set of a volumetric array, or an externally provided mesh. The volume is typically the result of the fusion of several depth maps using a truncated signed distance function or similar operator.

Tool Bar

view-reset Reset View
Resets the view extents so that the entire scene is visible. The keyboard shortcut R provides the same effect. Additional actions are available via the action’s associated pop-up.
view-reset Zoom to Landmarks
Resets the view extents so that all landmarks are visible. This action is available via the view-reset Reset View button’s associated pop-up menu. The keyboard shortcut is Shift+R.
view-reset View to World Top/Left/Right/Front/Back
Resets the view rotation to a “standard” rotation, such that the view axes are parallel with the world axes. These actions are available via the view-reset Reset View button’s associated pop-up menu.
view-reset Perspective
Toggles the world view between perspective and parallel projection. Perspective projection more closely models human vision and is often useful for visualizing depth in the scene. Parallel projection can be useful for viewing the scene in profile. This action is available via the view-reset Reset View button’s associated pop-up menu. The keyboard shortcut is P.
image Show Camera Frame Image
Toggles visibility of the frame image projected onto the ground plane (if an associated camera model is available). The associated pop-up allows the opacity of the same to be adjusted.
camera Show Cameras
Toggles visibility of cameras and related visualizations. The associated pop-up provides additional options; see Camera Options.
landmark Show Landmarks
Toggles visibility of landmarks. The associated pop-up provides additional options; see Point Options.
grid Show Ground Plane Grid
Toggles visibility of the ground plane grid. The ground plane is the z= 0 plane in local 3D coordinates. The grid is centered about x = y = 0, however the grid lines are otherwise strictly aesthetic and do not correspond to any particular values. The size of the grid adjusts to fit the scene.
roi Show/Edit Region of Interest
Toggles visibility of the region of interest selection in the world view. While visible, the ROI may be resized by clicking and dragging on any of the six handles on the faces of the ROI box.
roi Reset Region of Interest
Resets the region of interest to the axis-aligned box containing 80% of the landmark points plus an additional 50% padding. This action is available via the roi Show/Edit Region of Interest button’s associated pop-up menu.
depthmap Show 3D Depth Map
Toggles visibility of the depth map (if available) rendered as a 3D point cloud or mesh; see 3D Depth Map Options.
volume Show Surface from Volume
Toggles the visibility of the surface mesh extracted from volumetric data; see Volume Surface Options.
location Edit Ground Control Points
Toggles editing of ground control points. See Editing Ground Control Points for details. The keyboard shortcut is Ctrl+G.
ruler Enable Measurement Tool
Toggles placing or editing of the ruler measurement tool. Initially — when the ruler has not yet been placed, or after it has been removed using Reset Measurement Tool — a ruler can be placed by clicking two points in the view. The depth of the points is calculated based on landmarks, depth map points, or the surface mesh location under the mouse cursor. Turn off visibility of objects to avoid selecting them. The ground plane is selected if no nearby geometry is found. Once placed, the ruler’s points may be moved freely. Holding down the z key constrains the ruler to vertical measurements. Holding down the x or y keys constraints the ruler to a horizontal plane. Placement of the ruler may be canceled by pressing the Esc key before placing the second point.
ruler Reset Ruler
Removes the currently placed ruler. This action is available via the ruler Enable Measurement Tool button’s associated pop-up menu.

Camera Options

The camera Show Cameras pop-up provides additional controls that can be used to control the display of the cameras in the world view. These allow changing the color of both the active and inactive cameras as well as the camera path, changing the size of the camera frustums, and toggling visibility of the inactive cameras and camera path separate from the overall camera visibility.

The camera scale controls are relative to a “base size” that is computed from the extents of the scene data. The inactive camera scale is relative to the active camera scale, with the maximum allowed value giving active and inactive camera frustums the same size.

Point Options

The landmark Show Landmarks pop-up provides additional controls that can be used to control the display of the landmarks in the world view. (The same controls are also used in the camera view to manipulate the display of feature points and landmarks in that view.) These allow the color of the items to be changed, as well as their size. Feature items (that is, feature points and landmarks) are displayed as dots, with a fixed size-on-screen that is independent of the view.

Several options for color are provided. The simplest is “solid color”, which displays all landmarks in the same, user selected color. “True color” displays landmarks in the color estimated to correspond to the actual color of the point in the real world scene, as computed from the input imagery. “Color by data” uses color to visualize other per-point data, such as the number of individual frames that contributed to (“observed”) each landmark.

In addition to coloring by data, points may be filtered (selectively displayed) according to their respective values of the currently selected data set. Filtering may exclude points above or below selected lower or upper bounds, or that are not within said bounds.

3D Depth Map Options

The depthmap Show 3D Depth Map pop-up provides additional controls on the display of depth maps in the world view. The options allow the depth map to be rendered either as a 3D point cloud (one point per pixel) or a dense triangular mesh (one vertex per pixel). In either case, the rendered depth data is colored by the RGB color values of the corresponding video frame. A filter option is also available to remove depth points based on thresholds on various attributes. Currently, these attributes are the Weight and Uncertainty. Images of these attributes as well as the depth map itself are also shown in the Depth Map View and the filter options selected here apply to that view as well. See Depth Map View.

Volume Surface Options

The volume Show Surface from Volume pop-up provides additional controls on the extraction and coloring of a surface from volumetric data. The “Surface threshold” parameter controls the value of the isosurface at which the surface is extracted from the volume. The “Colorize surface” option, if checked, allows coloring each vertex of the mesh. The “Current frame” mode projects the RGB values from the current frame onto the mesh, while the “All frames” mode combines appearance projected from all frames or a subset of frame sampled at a regular interval. The “Color display” options determine how to color the surface. Options include mean color, median color, surface normal, and number of observations.

Editing Ground Control Points

The location Edit Ground Control Points action allows the user to enter or leave edit mode for ground control points. When not in edit mode, the scene location of ground control points is fixed and cannot be changed, nor can ground control points be selected in the world or camera views.

In edit mode, clicking on a ground control point in either view selects the point in both views as well as the Ground Control Points panel. (Selecting a point in the panel also selects it in both views.) Points may be dragged in either view to change their scene location. Holding the Shift key while moving constrains movement to one of the principle axes.

New points may be added by holding the Ctrl key while clicking. When placing new ground control points in the view, TeleSculptor projects a ray into the scene that corresponds to the location that was clicked and selects a location along this ray based on scene elements in the immediate vicinity. If no nearby landmark points are found, the new point is placed on the ground plane.

Pressing the Del key while in edit mode when one of the views has keyboard focus will delete the currently selected ground control point.

Camera View

The camera view provides a camera space view of detected feature points, computed landmarks and ground control points (all projected to the camera space), camera registration points, and the corresponding input imagery, for the active frame. Additionally, the estimation residuals — the difference between landmarks and feature points which participated in computing their estimated positions — can be displayed as line segments between the feature point location and projected landmark location.

Navigation

To pan the view, hold the middle (wheel) mouse button and drag, or drag while holding Alt and the primary (usually left) mouse button. To zoom, hold the secondary (usually right) mouse button and drag, or rotate the mouse wheel.

Tool Bar

view-reset Reset View
Resets the view to the frame image extents. Additional actions are available via the action’s associated pop-up.
view-reset Zoom Extents
Resets the view extents so that the entire scene is visible. This action is available via the view-reset Reset View button’s associated pop-up menu.
image Show Camera Frame Image
Toggles visibility of the frame image. The associated pop-up allows the opacity of the same to be adjusted.
feature Show Feature Points
Toggles visibility of feature points / trails. The associated pop-up provides additional options; see Feature Options.
landmark Show Landmarks
Toggles visibility of landmarks. The associated pop-up provides additional options; see Point Options.
residual Show Residuals
Toggles visibility of the landmark estimation residuals. The associated pop-up allows the color of the displayed residuals to be changed.
location Edit Camera Registration Points
Toggles editing of camera registration points. See Manual Camera Calibration for details. The associated pop-up allows computing a camera model on the current frame using camera registration points. The keyboard shortcut is Ctrl+R.

Feature Options

In addition to active feature points, which have all the options described in Point Options, the position of feature points on adjacent frames may also be displayed by enabling Trails. For image collections where frames adjacent in the frame list also have spatially similar camera poses (especially when using consecutive video frames as input), these may be useful as an additional means of visualizing camera motion.

The trail color and length (number of adjacent frames to be used) may be changed, as well as whether to show trails only for lower-numbered frames (“historic” mode), or for all adjacent frames (“symmetric” mode). In all cases, trails are displayed only for active feature points.

Depth Map View

The Depth Map View provides an image viewer similar to the Camera View but specialized to display depth map images. Depth map images are loaded from VTK image (.vti) files associated with a particular video frame. Often there are only depth maps on a subset of frames. The active (or most recent) depth map is displayed in this view by mapping depth to color. The Depth Map View can also display an image representation of other attributes associated with the depth map, such as the image color. Some attributes like uniqueness and best cost are associated with the algorithms used to generate the depth values. The same depth maps can be rendered in the World View as a point cloud. Furthermore, depth map filtering options in the World View also apply to the image rendering of the depth map in the Depth Map View.

Navigation functions the same as the Camera View.

Tool Bar

view-reset Reset View
Resets the view to the frame image extents.
view-reset Display Mode
Selects which image mode to display in the in the view: Color, Depth, Best Cost Value, Uniqueness Ratio; see Color Map Options. The depth filters apply regardless of which image is shown.

Color Map Options

In addition to selecting the mode under Display Mode, there is also an option to select the color mapping function for each mode except Color. The mapping function describes how the scalar data field (e.g. depth) is mapped to color. Below the color map option are the minimum and maximum values from the data used in the mapping. The Auto check box, which is checked by default, indicates that the values are determined automatically from the range of values in the image data. By unchecking the Auto checkbox, the minimum and maximum values of the range can be adjusted manually for finer control of the visualization.

Camera Selection

The camera selection panel contains a large slider used to select the active frame, and used to control which frame’s imagery and feature points are displayed in the camera view. If the camera pose for the frame is known, the active camera is also highlighted in the world view, A spin box next to the slider shows the active frame number, and can also be used to select the active frame. Note that the frame numbers need not be consecutive. Some video readers are configured to only read every N-th frame, where N may be 10, for example. This helps cut down on data redundancy in video. The frame sampling rate can be configured by opening the project configuration file (.conf) in a text editor.

The controls to the right of the panel control the application’s slideshow mode. Slideshow mode automatically increments through the loaded frames at a fixed rate. This can be used to view the feature points for each frame image in sequence. Setting the delay between cameras sufficiently low can be used to simulate video playback for image sequences taken from a motion imagery source.

The slideshow action controls are also available via the View menu. The small slider controls the delay between slides. The slider response is logarithmic, with single steps in one-tenth powers of ten. The slider tool tip includes the current delay in human readable units. Several frame filters are also available in the View menu. These filters allow limiting the frames to show to a specific subset, such as key frames or frames with tracking data.

Metadata

The metadata panel displays the collection of video metadata for the current frame, if available. The set of fields is selected from the entire data set; individual frames may be missing some or all fields. The metadata itself is provied by the video reader. For encoded video files, TeleSculptor supports key-length-value (KLV) encoding following the Motion Imagery Standards Board (MISB) 0104 and 0601 standards. Customized video readers can read metadata from other sources, such as supplimentary text files or EXIF data.

Ground Control Points

The ground control points panel displays a list of all ground control points and camera registration points in the current data set, as well as detailed information for the selected point. Points have an automatically assigned ID (which may change between sessions) and an optional user-provided name, which may be assigned or changed by editing that column of the point (by double-clicking or pressing the edit key — usually F2).

Selecting a point in the list will select the same point in the world and camera views. Similarly, selecting a ground control point or camera registration point in either view (when the corresponding edit mode is active) will select the same point in the list.

When a point is selected, changing its geodetic location (as described by the latitude, longitude, and elevation text fields) automatically promotes the point to a “user registered” point. These are points for which the geodetic location has been externally measured and is therefore known to be correct. The geodetic location of points which are not user registered is computed from their scene location and the computed scene to geodetic transformation (if available). User registered points are indicated by a glyph (glyph-surveyed) in the ground control point list.

Note that moving a user registered point in the world or camera views (that is, changing its scene location) does not change its geodetic location.

Tool Bar

copy-location Copy Location
Copies the geodetic location of the selected point to the clipboard. Several options for coordinate ordering and whether or not to include the elevation are provided.
apply Apply Similarity Transform
Estimates a 3D similarity transformation to best align the ground control point locations with the specified geodetic locations. At least three “user registered” ground control points are required. That is, at least three points must have manually specified latitude, longitude, and altitude. The estimated transform is applied to all data (cameras, landmarks, depth maps, etc.).
reset Revert Changes
Reverts user changes to the active ground control point’s geodetic location, such that the point is no longer “user registered”. This has no effect on points that are not user registered. Note also that the geodetic location will not change if a scene to geodetic transformation is not available.
delete Delete Point
Deletes the active ground control point.

Match Matrix View

The match matrix view provides a visualization of the feature point associations across frames. Pixels in the image correspond to values in the “match matrix” representing the number of feature points that feature detection has determined correspond to the same real world feature. Several options are provided to adjust the visualization:

  • Layout controls the position of “identity” values, i.e. values that compare a frame to itself rather than a distinct frame. The default, “diagonal”, simply maps the frame number directly to both the X and Y axes. “Horizontal” skews the image so that the y values are relative to the “identity” values, placing them in a horizontal line at y = 0, with positive y representing “later” frames, and negative y representing “earlier” frames. “Vertical” reverses these axes.
  • Orientation controls which screen direction is considered positive Y. The default, “matrix”, uses down for positive Y, as in textual value tables (e.g. textual listings of matrices, spreadsheets) or images. “Graph” uses up for positive Y, as in most graphical plots.
  • Values controls what values are used for each pixel. The default, “absolute”, uses the raw number of feature point correlations (which, for “identity” values is equal to the total number of feature points on that frame). “Relative (combined)” mode uses the percent of common feature points relative to the total number of distinct feature points on each frame being compared. The other two “relative” modes give the percent relative to the total number of feature points for the frame represented by either the X or Y axis.
  • Scale controls the scaling function that is applied to the values produced according to the value mode. The choices are “linear”, “logarithmic” and “exponential”, and should be self explanatory. In absolute value mode, logarithmic scale uses the maximum value as the logarithm base. Otherwise, the base can be adjusted with the “range” control, which applies a pre-scale to the value before computing the logarithm (thereby allowing the shape of the scaling curve to be adjusted). Exponential scale allows the user to select the exponent.
  • Color provides the set of colors to which scaled values are mapped. Several presets are available according to user taste. Different presets may help emphasize different aspects of the data.

Moving the mouse over the image will display which frames are being compared and the number or percentage of feature correlations in the status bar. The match matrix view also allows the image to be exported to a file.

Data Files

TeleSculptor supports visualization of various data files (landmarks, cameras, etc.) that are computed in other tools. However the recommended workflow for most users is to simply load a video and derive all other product from it. Video files are loaded using File → Import → Imagery….

Before computing any products from video, a “Project” directory is needed to store the results. A project is created with File → New Project which asks the user to provide a path to a working directory. Inside this directory, a “Project File” is created (name matching the directory name plus extension .conf) to store project settings. Various other result files are also written to the project directory. To open an existing project, use File → Open Project… and navigate to an existing .conf file.

Note

When loading cameras or images individually, cameras and images are associated in a first-loaded, first-matched manner. There is no way to load individual camera and image files that allows for cameras without images, or images without cameras, except at the end of the frame sequence. Similarly, frame identifiers are assigned sequentially based on the order in which files are loaded. In order for feature points to be correctly associated with their corresponding frames, the camera/image files must be loaded so that these automatically assigned identifies match those that were assigned by the feature detection/tracking pipeline.