Understanding LeiaPix Depth Maps

If the side-by-side stereo pairs below are too far apart for you to fuse them by parallel viewing, you may be able to zoom out of this web page in your browser. In Chrome, use .

Flowchart

This flowchart shows the files and programs used to convert a 2D image to 3D. The naming convention for the files produced is:-

LPX.D	LeiaPix Converter → depth map
LPX.LR	LeiaPix Converter → Left and Right images side-by-side
LPX.LIF	LeiaPix Converter → Leia Image Format file
LF2.D	LIF2 → depth map
LF2.M	LIF2 → mono image
LF2.C	LIF2 → contour map
LF2.MD	LIF2 → mono image and depth map side-by-side
SPM.LR	StereoPhoto Maker → Left and Right images side-by-side

The LeiaPix depth map LPX.D can be ignored as it is no better than the one in the LIF. The LeiaPix side-by-side LPX.LR can be ignored as it is highly compressed and created with no control over disparity as provided by StereoPhoto Maker. The LIF2 depth map LF2.D and mono image LF2.M can be ignored as they are only provided for convenience if needed without having to separate LF2.MD. The LIF2 contour map LF2.C displays the depth map by overlaying the mono image with contours which show where the depth changes.

Disparity from depth

Of course, all of these depth maps are really disparity maps: the actual depth or object distance is inversely proportional to the disparity. In the depth maps, the values are normalised to the range 0 to 255. These values are still referred to here as depths to distinguish them from the disparities used by StereoPhoto Maker in Edit->Depth map-> Create 3D image from 2D+Depth map. SPM calculates the disparities with parameters supplied in this dialog:-

The formula used by SPM¹ is

giving

where width is the width of the image in pixels and deviation and position are the parameters entered in the dialog. The position specifies the position of the stereo window: pixels at this depth are assigned zero disparity. The disparity is the difference in pixel x-coordinates between the right and left images. For each pixel in the mono image, SPM shifts it disparity/2 pixels left for the right image and disparity/2 pixels right for the left image. Setting deviation to 30 effectively satisfies the editing version of the "1 in 30" rule. It is important to remember, however, that the depth values are normalised to the range 0 to 255, regardless of the range of depths in the converted image. Setting deviation to 30 could be too high, resulting in the dreaded "cardboard cutout" effect.

Depth map contours

Shifting pixels to the left and right usually leaves gaps in the left and right images, which SPM must fill with its gap-filling algorithm. To appreciate how the gaps occur we can examine the depth map contours superimposed on an image.

Each contour shows where there is a change in depth. On most of these contours, the depth is increasing or decreasing by 1. When there is a large change over a small area, the contours become so close together that they join into a black band. These depth changes can be seen more clearly on the cross-section underneath, which shows the depth variation along the yellow mid-line. It's good that there are no sudden depth changes at the edges of objects, as this helps avoid cardboard cut-out effects.

The strip underneath the cross-section shows the trajectory of pixels when they are shifted left for the right image. Only pixels at depth changes are shown. When there is a black band at the left edge of an object, pixels are overlaid by compression in the left shift, as can be seen at the edge of the sleeve on the left. When there is a black band at the right edge of an object, pixels are separated by expansion in the left shift, as can be seen at the edge of the arm on the right.

This makes it easier for the gap-filling algorithm than a sudden disparity change, but causes some distortion. This is one reason why the black bands are offset from the edges of objects. The other reason is that objects are often surrounded by protrusions which are below the resolution of the depth map. The classic example is hair around a head. On the stereo images, this offset results in some of the background being pulled in around the object. This is not noticeable when the background is plain, as in the wall behind the figure.

Here the hair is successfully rendered, except for some longer hairs on the left which have remained at the depth of the background. This image also shows the compression/expansion artefacts suffered by the earring on the left, which falls inside a black band.

When the background is not plain, parts of it are distorted by being pulled in the direction of the object, as shown below behind the sleeve.

These artefacts result from trade-offs made by the LeiaPix converter when creating the depth map. In many cases, they are too slight to be noticed except at extreme enlargement, and the 2D to 3D conversion can be considered successful.

iPhone portrait mode

It is instructive to compare the depth map contours from a LeiaPix depth map with those from a depth map extracted by SPM from an iPhone photo taken in portrait mode.

Here there is not much gap between the black bands and the edges of the face and hair. This result in severe compression/expansion artefacts along these edges.

^{1. Inferred from experiments with SPM.}

Christopher B. Jones 2022-04-28