python - 如何从包含路径和标签的txt文件创建 tensorflow 数据集？

Question

我正在尝试加载 DomainNet数据集转换为 tensorflow 数据集。 每个域包含两个分别用于训练和测试数据的 .txt 文件，其结构如下:

painting/aircraft_carrier/painting_001_000106.jpg 0
painting/aircraft_carrier/painting_001_000060.jpg 0
painting/aircraft_carrier/painting_001_000130.jpg 0
painting/aircraft_carrier/painting_001_000058.jpg 0
painting/aircraft_carrier/painting_001_000093.jpg 0
painting/aircraft_carrier/painting_001_000107.jpg 0
painting/aircraft_carrier/painting_001_000088.jpg 0
painting/aircraft_carrier/painting_001_000014.jpg 0
painting/aircraft_carrier/painting_001_000013.jpg 0
...

每张图像一行包含相对路径和标签。我的问题是，tensorflow/keras 中是否已经有某种内置方法来加载这种结构，或者我是否必须手动解析和加载数据？到目前为止，我的 google-fu 让我失望了......

Answer 1

您可以使用tf.data.TextLineDataset一次加载和处理多个txt文件:

import tensorflow as tf
import matplotlib.pyplot as plt

with open('data.txt', 'w') as f:
  f.write('/content/result_image1.png 0\n')
  f.write('/content/result_image2.png 1\n')

with open('more_data.txt', 'w') as f:
  f.write('/content/result_image1.png 1\n')
  f.write('/content/result_image2.png 0\n')

dataset = tf.data.TextLineDataset(['/content/data.txt', '/content/more_data.txt'])
for element in dataset.as_numpy_iterator():
  print(element)

b'/content/result_image1.png 0'
b'/content/result_image2.png 1'
b'/content/result_image1.png 1'
b'/content/result_image2.png 0'

过程数据:

def process(x):
  splits = tf.strings.split(x, sep=' ')
  image_path, label = splits[0], splits[1]
  img = tf.io.read_file(image_path)
  img = tf.io.decode_png(img, channels=3)
  return  img, tf.strings.to_number(label, out_type=tf.int32)

dataset = dataset.map(process)
for x, y in dataset.take(1):
  print('Label -->', y)
  plt.imshow(x.numpy())

Label --> tf.Tensor(0, shape=(), dtype=int32)