I kind of put this project on the backburner for the time being. I'll still occasionally look for new issues and try to reply, but I cannot currently continue development because I'm lacking the necessary hardware. I'll also occasionally check for pull request. So if you feel like adding or improving something, feel wholeheartedly invited to do so!
This ROS package merges the depth images of two depth cameras (RGBD, Time of Flight, Structured Light etc.) into one depth image.
It's actually twofold. One node merges two point clouds and the other converts a point cloud from a certain point of view into a depth image. By chaining multiple merge_pcl nodes/nodelets, it's possible to merge even more cameras. At a cost though.
For simple RGB images it is possible to just distort and stitch two images based on the camera parameters and certain features (edges, corners) that are commonly available in both images to generate a "panorama" like image (see auto_stitch for example). For depth images this is not sufficient as the depth values (i.e. the pixel values) would also have to be adapted to the distortion.
This node/nodelet utilizes point cloud concatenation to generate a depth image. This is done by taking the following steps:
- generate point clouds from the depth images of both cameras
- transform both point clouds into a common coordinate system (e.g. "map" or the coordinate system of the virtual camera; see below)
- concatenate both transformed point clouds into one big point cloud
- project all points of the new point into the image of a pinhole camera model based virtual camera
I needed a way to enhance the field of view of a depth cam. Instead of fiddling with additional lenses (which would probably have worked too), I used what I had at hand: a second depth cam. Furthermore I like the idea of using a point cloud because it makes it possible to place the virtual camera practically arbitrarily and even make it able to move in the virtual space, as long as it and the other cameras see respectively generate enough points of the point cloud to get a reasonable depth image as a result.
This makes it even possible to optimize the camera pose based on some fitness value (e.g. for people tracking this might be how well the person of interest can be seen by the virtual camera).
Well, I could have, but... first of all, to be dead honest, using the point clouds was easier to grasp and implement. If someone comes up with a faster solution, go for it! I won't stop anyone from implementing a pure math/depth image based approach. Quite the opposite. I'd really like to see how much that approach would be faster.
Sure. All cameras (the two real cameras and the virtual camera) should have the same focal length. The pinhole camera furthermore should take a point of view that sees all points of the point cloud and from which the coverage of the point cloud is sufficient.
CPU load is quite high, of course, depending on the input and output resolution.
TBA