1 year ago

#76143

test-img

Sami Wood

minimal example of bundle adjustment for image mosaic in OpenCV

Im looking at the following image stitching example in the OpenCV documentation: https://raw.githubusercontent.com/opencv/opencv/4.x/samples/python/stitching_detailed.py, trying to wrap my head around how to use bundle adjustment to estimate homography and warp images. Im having a hard time following what exactly is going in partially because I can't seem to find the docs for many of the functions they are using. A snippet of the code I think I am particularly interested in is below.


    estimator = ESTIMATOR_CHOICES[args.estimator]()
    b, cameras = estimator.apply(features, p, None)
    if not b:
        print("Homography estimation failed.")
        exit()
    for cam in cameras:
        cam.R = cam.R.astype(np.float32)

    adjuster = BA_COST_CHOICES[args.ba]()
    adjuster.setConfThresh(1)
    refine_mask = np.zeros((3, 3), np.uint8)
    if ba_refine_mask[0] == 'x':
        refine_mask[0, 0] = 1
    if ba_refine_mask[1] == 'x':
        refine_mask[0, 1] = 1
    if ba_refine_mask[2] == 'x':
        refine_mask[0, 2] = 1
    if ba_refine_mask[3] == 'x':
        refine_mask[1, 1] = 1
    if ba_refine_mask[4] == 'x':
        refine_mask[1, 2] = 1
    adjuster.setRefinementMask(refine_mask)
    b, cameras = adjuster.apply(features, p, cameras)
    if not b:
        print("Camera parameters adjusting failed.")
        exit()
    focals = []
    for cam in cameras:
        focals.append(cam.focal)
    focals.sort()
    if len(focals) % 2 == 1:
        warped_image_scale = focals[len(focals) // 2]
    else:
        warped_image_scale = (focals[len(focals) // 2] + focals[len(focals) // 2 - 1]) / 2
    if wave_correct is not None:
        rmats = []
        for cam in cameras:
            rmats.append(np.copy(cam.R))
        rmats = cv.detail.waveCorrect(rmats, wave_correct)
        for idx, cam in enumerate(cameras):
            cam.R = rmats[idx]
    corners = []
    masks_warped = []
    images_warped = []
    sizes = []
    masks = []
    for i in range(0, num_images):
        um = cv.UMat(255 * np.ones((images[i].shape[0], images[i].shape[1]), np.uint8))
        masks.append(um)

    warper = cv.PyRotationWarper(warp_type, warped_image_scale * seam_work_aspect)  # warper could be nullptr?
    for idx in range(0, num_images):
        K = cameras[idx].K().astype(np.float32)
        swa = seam_work_aspect
        K[0, 0] *= swa
        K[0, 2] *= swa
        K[1, 1] *= swa
        K[1, 2] *= swa
        corner, image_wp = warper.warp(images[idx], K, cameras[idx].R, cv.INTER_LINEAR, cv.BORDER_REFLECT)
        corners.append(corner)
        sizes.append((image_wp.shape[1], image_wp.shape[0]))
        images_warped.append(image_wp)
        p, mask_wp = warper.warp(masks[idx], K, cameras[idx].R, cv.INTER_NEAREST, cv.BORDER_CONSTANT)
        masks_warped.append(mask_wp.get())

There are several key things I can't seem to find.

  1. estimator.apply() I can't find the docs to this and so I don't fully understand what the function exects as arguments nor what it returns. (Estimator I'm looking at: https://docs.opencv.org/4.x/df/d15/classcv_1_1detail_1_1Estimator.html)

  2. What is the camera object for cam in cameras: cam.R = cam.R.astype(np.float32) is this the correct documentation to look at? https://docs.opencv.org/4.x/dc/d3a/classcv_1_1viz_1_1Camera.html

  3. adjuster.apply() also doesn't seem to be a member function of any of the classes. BundleAdjusterBase, BundleAdjusterReproj or others... (maybe I just don't understand c++. Adjuster IM looking at: https://docs.opencv.org/4.x/d5/d56/classcv_1_1detail_1_1BundleAdjusterBase.html)

  4. PyRotationWarper Class Reference states PyRotationWarper.warp() takes two parameters from the camera intrinsics and rotation. Would I be correct in assuming this is performing the bundle adjustment step warping images based on 3D points?

  5. Does this snippet more or less represent a minimal working example of image mosaicing using bundle adjustment? Im not sure what Im doing. If someone would be willing to provide an example of stitching 4 or 5 images and applying bundle adjustment I would be eternally grateful.

PS. I'm not using createStitcher because I want to learn to do it from scratch and eventually use deep learning to estimate camera params and pose/ match feature points.

python

opencv

image-processing

computer-vision

image-stitching

0 Answers

Your Answer

Accepted video resources