python - Tensorflow - 获取像素的邻域

Question

我正在尝试在 Tensorflow/Keras 中实现 Levin 等人 (2004) 的经典图像着色论文的损失函数:

这是权重方程(强度之间的相关性):

y 是 3x3 窗口中 x 的每个相邻像素，w 是每个像素的权重。

权重需要计算每个像素邻域的均值和方差。

我找不到一个函数可以让我以符号方式编写这个损失函数，我想我应该将它写在一个循环中，在循环中计算每个损失函数的 w窗口。

如何在 Tensorflow 中以符号方式或循环编写此损失函数？

非常感谢。

编辑:这是我在 Numpy 中计算权重的代码:

import cv2
import numpy as np

im = cv2.resize(cv2.imread('./Image.jpg', 0), (256, 256)) / np.float32(255.0)

M = 3
N = 3

# Split the image into 3x3 windows
windows = [im[x:x + M, y:y + N] for x in range(0, im.shape[0], M) for y in range(0, im.shape[1], N)]

# Calculate the correlation for each window
weights = [1 + np.corrcoef(tile) for tile in windows]

Answer 1

我认为这段代码计算了公式中的值:

import tensorflow as tf
from itertools import product

SIGMA = 1.0

dtype = tf.float32
# Input images batch
img = tf.placeholder(dtype, [None, None, None])
img_shape = tf.shape(img)
img_height = img_shape[1]
img_width = img_shape[2]
# Compute 3 x 3 block means
mean_filter = tf.ones((3, 3), dtype) / 9
img_mean = tf.nn.conv2d(img[:, :, :, tf.newaxis],
                        mean_filter[:, :, tf.newaxis, tf.newaxis],
                        [1, 1, 1, 1], 'VALID')[:, :, :, 0]
# Remove 1px border
img_clip = img[:, 1:-1, 1:-1]
# Difference between pixel intensity and its block mean
x_diff = img_clip - img_mean
# Compute neighboring pixel loss contributions
contributions = []
for i, j in product((-1, 0, 1), repeat=2):
    if i == j == 0: continue
    # Take "shifted" image
    displaced_img = img[:, 1 + i:img_width - 1 + i, 1 + j:img_height - 1 + j]
    # Compute difference with mean of corresponding pixel block
    y_diff = displaced_img - img_mean
    # Weights formula
    weight = 1 + x_diff * y_diff / (SIGMA ** 2)
    # Contribution of this displaced image to the loss of each pixel
    contribution = weight * displaced_img
    contributions.append(contribution)
contributions = tf.add_n(contributions)
# Compute loss value
loss = tf.reduce_sum(tf.squared_difference(img_clip, contributions))

不会计算沿图像边界的像素损失，因为原则上公式中没有明确定义，尽管您可以根据需要进行一些更改以将它们考虑在内(将卷积更改为“'SAME' "，必要时进行填充等)。