1 year ago
#76143
Sami Wood
minimal example of bundle adjustment for image mosaic in OpenCV
Im looking at the following image stitching example in the OpenCV documentation: https://raw.githubusercontent.com/opencv/opencv/4.x/samples/python/stitching_detailed.py, trying to wrap my head around how to use bundle adjustment to estimate homography and warp images. Im having a hard time following what exactly is going in partially because I can't seem to find the docs for many of the functions they are using. A snippet of the code I think I am particularly interested in is below.
estimator = ESTIMATOR_CHOICES[args.estimator]()
b, cameras = estimator.apply(features, p, None)
if not b:
print("Homography estimation failed.")
exit()
for cam in cameras:
cam.R = cam.R.astype(np.float32)
adjuster = BA_COST_CHOICES[args.ba]()
adjuster.setConfThresh(1)
refine_mask = np.zeros((3, 3), np.uint8)
if ba_refine_mask[0] == 'x':
refine_mask[0, 0] = 1
if ba_refine_mask[1] == 'x':
refine_mask[0, 1] = 1
if ba_refine_mask[2] == 'x':
refine_mask[0, 2] = 1
if ba_refine_mask[3] == 'x':
refine_mask[1, 1] = 1
if ba_refine_mask[4] == 'x':
refine_mask[1, 2] = 1
adjuster.setRefinementMask(refine_mask)
b, cameras = adjuster.apply(features, p, cameras)
if not b:
print("Camera parameters adjusting failed.")
exit()
focals = []
for cam in cameras:
focals.append(cam.focal)
focals.sort()
if len(focals) % 2 == 1:
warped_image_scale = focals[len(focals) // 2]
else:
warped_image_scale = (focals[len(focals) // 2] + focals[len(focals) // 2 - 1]) / 2
if wave_correct is not None:
rmats = []
for cam in cameras:
rmats.append(np.copy(cam.R))
rmats = cv.detail.waveCorrect(rmats, wave_correct)
for idx, cam in enumerate(cameras):
cam.R = rmats[idx]
corners = []
masks_warped = []
images_warped = []
sizes = []
masks = []
for i in range(0, num_images):
um = cv.UMat(255 * np.ones((images[i].shape[0], images[i].shape[1]), np.uint8))
masks.append(um)
warper = cv.PyRotationWarper(warp_type, warped_image_scale * seam_work_aspect) # warper could be nullptr?
for idx in range(0, num_images):
K = cameras[idx].K().astype(np.float32)
swa = seam_work_aspect
K[0, 0] *= swa
K[0, 2] *= swa
K[1, 1] *= swa
K[1, 2] *= swa
corner, image_wp = warper.warp(images[idx], K, cameras[idx].R, cv.INTER_LINEAR, cv.BORDER_REFLECT)
corners.append(corner)
sizes.append((image_wp.shape[1], image_wp.shape[0]))
images_warped.append(image_wp)
p, mask_wp = warper.warp(masks[idx], K, cameras[idx].R, cv.INTER_NEAREST, cv.BORDER_CONSTANT)
masks_warped.append(mask_wp.get())
There are several key things I can't seem to find.
estimator.apply() I can't find the docs to this and so I don't fully understand what the function exects as arguments nor what it returns. (Estimator I'm looking at: https://docs.opencv.org/4.x/df/d15/classcv_1_1detail_1_1Estimator.html)
What is the camera object
for cam in cameras: cam.R = cam.R.astype(np.float32)
is this the correct documentation to look at? https://docs.opencv.org/4.x/dc/d3a/classcv_1_1viz_1_1Camera.htmladjuster.apply() also doesn't seem to be a member function of any of the classes. BundleAdjusterBase, BundleAdjusterReproj or others... (maybe I just don't understand c++. Adjuster IM looking at: https://docs.opencv.org/4.x/d5/d56/classcv_1_1detail_1_1BundleAdjusterBase.html)
PyRotationWarper Class Reference states PyRotationWarper.warp() takes two parameters from the camera intrinsics and rotation. Would I be correct in assuming this is performing the bundle adjustment step warping images based on 3D points?
Does this snippet more or less represent a minimal working example of image mosaicing using bundle adjustment? Im not sure what Im doing. If someone would be willing to provide an example of stitching 4 or 5 images and applying bundle adjustment I would be eternally grateful.
PS. I'm not using createStitcher because I want to learn to do it from scratch and eventually use deep learning to estimate camera params and pose/ match feature points.
python
opencv
image-processing
computer-vision
image-stitching
0 Answers
Your Answer