Stereoscopic Depth

The depth information in a three-dimensional scene in front of us can be acquired by a dual-lens stereoscopic camera as a side‑by‑side pair of images and presented for viewing in the same manner. There are other methods of acquisition: on some iPhones the TrueDepth camera captures accurate face data by projecting and analysing thousands of invisible dots to create a depth map of your face. There are other methods of presentation: row-interlaced images for viewing on a passive 3D TV or monitor, or colour anaglyphs for viewing with red/cyan glasses.

When using a dual-lens stereoscopic camera, it is possible to deviate from the conventional model where acquisition and presentation are essentially inverse operations. A depth map can be created from a stereoscopic pair in StereoPhoto Maker with
Edit->Depth map->Create depth map from stereo pair.
SPM invokes the DMAG5 and DMAG9b apps by Ugo Capeto. From this image:- it created this pair consisting of the left image and the depth map:- For each pixel there is a corresponding value (actually a disparity) in the depth map. This correspondence remains when the left image and the depth map are transformed identically, by clockwise rotation, for example:-

This transformed image can then be reloaded into SPM and used to create a new stereo pair with
Edit->Depth map->Create 3D image from 2D+Depth map, giving:-
Apart from the aspect ratio, this resembles the photograph which might have been taken of the same scene with the camera rotated.

Thus if a more complicated workflow is acceptable, it's not actually necessary to take stereo photographs with the two lenses side‑by‑side. Why might it be advantageous for the two lenses to be one above the other, for example?

  1. To allow a more compact rig:
  2. Taking a hyper stereo shot when sideways access is limited.
  3. Shooting a panorama to get consistent depth information between the left and right edges.
Regarding the second point, it should be noted that the effective separation can be increased when recreating the 3D image, so that less lens separation may be necessary. The third point is the only really compelling reason: one that this article has been leading up to.

Stereo pairs taken with a dual fisheye camera in VR180 mode are intended to be viewed in a VR headset, with which you can look in any direction. They have an inherent problem in that horizontal disparity decreases away from the centre whereas vertical disparity increases. This effect may not be noticeable in sideways views if there are no nearby objects close to the edges. To view the whole width on a normal screen, you can convert the stereo pair from equirectangular to a central cylindrical panorama, but the stereo effect is weak away from the centre:- Perhaps better depth information can be obtained by holding the camera with the lenses separated vertically instead of horizontally?

Here is an equirectangular stereoscopic image taken with an Insta360 EVO camera in VR180 mode when standing vertically instead of horizontally. The camera detected the vertical orientation and rotated the dual fisheye images before converting to equirectangular. Depth cannot be perceived in this pair of images as the disparities are vertical, not horizontal. It can be converted into a pair of central cylindrical panoramas and rotated 90° using this command with the ffmpeg v360 filter
ffmpeg -i familyroom.jpg -vf v360=e:cylindrical:h_fov=360:v_fov=115,scale=w=iw/2:h=ih,transpose=1 -y -update true frpanorama.jpg
This can be read into SPM by selecting the Above/Below Stereo Format. The disparities, which are now horizontal, are too small, however, to extract useful depth information by the above-mentioned depth map method in SPM.

This exercise was repeated with the camera mounted on a tripod, a first shot taken at the base position and a second shot with the camera raised about 10 cm. This allowed the camera to be used in 360° mode and the two views were converted to central cylindrical panoramas and rotated 90° as before, then loaded into SPM:- The disparities, which are now horizontal, are now good enough to extract useful depth information by the above-mentioned depth map method in SPM. When the left image and depth map are now rotated by clockwise rotation, the recreated stereo pair is:- This has quite good stereo depth but unfortunately is marred by depth map errors especially noticeable at the sides of the television screen on the right-hand wall. This depth map was created with the default DMAG5 and DMAG9b parameters in SPM. It is possible to try to improve the depth map by tweaking these parameters but hard to get an acceptably good result.

It's a bit of a cop-out, but better 3D effects can be obtained using a depth map created from a single image by MiDaS. From one of the above images (unrotated) my Python script used MiDaS to create this image:- Here is a Dubois anaglyph version for viewing with red/cyan glasses:- This then seems to be the best way to obtain a stereoscopic 3D panorama for casual viewing on a normal screen.

Even though he wrote DMAG5 and DMAG9B, Ugo Capeto no longer recommends using them and says that nowadays, it's just easier to use AI monocular depth estimation. He recommends MiDaS version 3.1 which rates highly in this excellent review at World of Depth.

Christopher B. Jones, 2024-07-10