python - 过滤图像中补丁位置的最优化方法

Question

所以我的问题是这样的:我有一个 RGB 图像作为尺寸 (4086, 2048, 3) 的 numpy 数组，我将此图像尺寸拆分为 96x96 block 并返回 这些补丁在 numpy 数组中的位置。在每种情况下我总是得到 96x96 补丁。如果图像的尺寸不允许我在 x 或 y 轴上创建“纯”96x96 补丁，我只需向其添加左侧填充，以便最后的补丁与之前的补丁有一点重叠。

现在有了这些位置，我想以最快的方式删除所有三个 channel 中补丁中每个像素的 RGB 值为 255 的 96x96 补丁可能，并且我想取回所有没有此值的补丁位置。

我想知道:

1. 从图像维度提取 96x96 block 位置的最快方法是什么？ (现在我有一个 for 循环)
2. 如何以最佳方式消除纯白色斑 block (3 个 channel 上的值为 255)？ (现在我有一个 for 循环)

我有很多这样的图像需要处理，图像分辨率高达(39706, 94762, 3)，所以我的“for 循环”在这里很快变得低效。感谢您的帮助! (我也采用了利用 GPU 的解决方案)

下面是伪代码，让您了解目前是如何完成的:

patches = []
patch_y = 0
y_limit = False
slide_width = 4086
slide_height = 2048
# Lets imagine this image_slide has 96x96 patches which value is 255
image_slide = np.random.rand(slide_width, slide_height, 3)
while patch_y < slide_height:
    patch_x = 0
    x_limit = False
    while patch_x < slide_width:
        # Extract the patch at the given position and return it or return None if it's 3 RGB
        # channels are 255
        is_white = PatchExtractor.is_white(patch_x, patch_y, image_slide)

        # Add the patches position to the list if it's not None (not white)
        if not is_white:
            patches.append((patch_x, patch_y))

        if not x_limit and patch_x + crop_size > slide_width - crop_size:
            patch_x = slide_width - crop_size
            x_limit = True
        else:
            patch_x += crop_size
    if not y_limit and patch_y + crop_size > slide_height - crop_size:
        patch_y = slide_height - crop_size
        y_limit = True
    else:
        patch_y += crop_size

return patches

理想情况下，我希望将我的补丁位置放在“for循环”之外，然后一旦我有了它们，我就可以测试它们是否是白色的在for循环之外以及尽可能少的情况调用numpy(所以代码在numpy的C层处理，不会来回python)

Answer 1

正如您所怀疑的，您可以矢量化您正在做的所有事情。它大约需要原始图像内存需求的一个小整数倍。该算法非常简单:填充图像，使整数个补丁适合其中，将其切成多个补丁，检查每个补丁是否全是白色，保留其余部分:

import numpy as np                                                                    

# generate some dummy data and shapes
imsize = (1024, 2048)
patchsize = 96
image = np.random.randint(0, 256, size=imsize + (3,), dtype=np.uint8)
# seed some white patches: cut a square hole in the random noise
image[image.shape[0]//2:3*image.shape[0]//2, image.shape[1]//2:3*image.shape[1]//2] = 255

# pad the image to necessary size; memory imprint similar size as the input image
# white pad for simplicity for now
nx,ny = (np.ceil(dim/patchsize).astype(int) for dim in imsize) # number of patches
if imsize[0] % patchsize or imsize[1] % patchsize:
    # we need to pad along at least one dimension
    padded = np.pad(image, ((0, nx * patchsize - imsize[0]),
                            (0, ny * patchsize - imsize[1]), (0,0)),
                    mode='constant', constant_values=255)
else:
    # no padding needed
    padded = image

# reshape padded image according to patches; doesn't copy memory
patched = padded.reshape(nx, patchsize, ny, patchsize, 3).transpose(0, 2, 1, 3, 4) 
# patched is shape (nx, ny, patchsize, patchsize, 3)
# appending .copy() as a last step to the above will copy memory but might speed up
# the next step; time it to find out

# check for white patches; memory imprint the same size as the padded image
filt = ~(patched == 255).all((2, 3, 4))
# filt is a bool, one for each patch that tells us if it's _not_ all white
# (i.e. we want to keep it)

patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
patch_pixel_x = patch_x * patchsize  # proper pixel indices of each pixel
patch_pixel_y = patch_y * patchsize
patches = np.array([patch_pixel_x, patch_pixel_y]).T
# shape (npatch, 2) which is compatible with a list of tuples

# if you want the actual patches as well:
patch_images = patched[filt, ...]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels

如您所见，在上面我使用白色填充来获得一致的填充图像。我相信这符合您想要做的事情的哲学。如果您想精确地复制循环中正在执行的操作，您可以使用边缘附近考虑的重叠像素手动填充图像。您需要分配正确大小的填充图像，然后手动切片原始图像的重叠像素，以便在填充结果中设置边缘像素。

---

由于您提到您的图像很大，因此填充会导致过多的内存使用，因此您可以避免使用一些苦力来填充。您可以使用大图像的切片(不会创建副本)，但是您必须手动处理没有完整切片的边缘。方法如下:

def get_patches(img, patchsize):
    """Compute patches on an input image without padding: assume "congruent" patches

    Returns an array shaped (npatch, 2) of patch pixel positions"""
    mx,my = (val//patchsize for val in img.shape[:-1])
    patched = img[:mx*patchsize, :my*patchsize, :].reshape(mx, patchsize, my, patchsize, 3)
    filt = ~(patched == 255).all((1, 3, 4))
    patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
    patch_pixel_x = patch_x * patchsize  # proper pixel indices of each pixel
    patch_pixel_y = patch_y * patchsize
    patches = np.stack([patch_pixel_x, patch_pixel_y], axis=-1)
    return patches

# fix the patches that fit inside the image
patches = get_patches(image, patchsize)

# fix edge patches if necessary
all_patches = [patches]
if imsize[0] % patchsize:
    # then we have edge patches along the first dim
    tmp_patches = get_patches(image[-patchsize:, ...], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [imsize[0] - patchsize, 0])
if imsize[1] % patchsize:
    # same along second dim
    tmp_patches = get_patches(image[:, -patchsize:, :], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [0, imsize[1] - patchsize])
if imsize[0] % patchsize and imsize[1] % patchsize:
    # then we have a corner patch we still have to fix
    tmp_patches = get_patches(image[-patchsize:, -patchsize:, :], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [imsize[0] - patchsize, imsize[1] - patchsize])

# gather all the patches into an array of shape (npatch, 2)
patches = np.vstack(all_patches)

# if you also want to grab the actual patch values without looping:
xw, yw = np.mgrid[:patchsize, :patchsize]
patch_images = image[patches[:,0,None,None] + xw, patches[:,1,None,None] + yw, :]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels

这也将完全复制您的循环代码，因为我们明确采用边缘补丁，使它们与之前的补丁重叠(没有虚假的白色填充)。不过，如果您想让补丁按给定的顺序排列，您现在就必须对它们进行排序。