您好，TensorFlow friend 。
TL-DR:
我正在 Google collab 上使用 Tensorflow 1.1.4 并进行 eagerexecution。
我的训练在第一个时期的最后一批失败，因为我的训练循环中有一个强制批量大小 tf.unpack，有时无法获得完整的批处理，并且失败:
Epoch 1/5
147/148 [============================>.] - ETA: 0s - loss: 0.0133 - acc: 0.3863
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-133-bba4aeb63f3b> in <module>()
1 steps_per_epoch=tf.ceil(len(all_image_paths)/BATCH_SIZE).numpy()
2
----> 3 history = model.fit(ds, epochs=5, steps_per_epoch=steps_per_epoch)
8 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
InvalidArgumentError: Input shape axis 0 must equal 32, got shape [30]
[[{{node unstack}}]] [Op:IteratorGetNextSync]
我不知道如何解压包含 BATCH_SIZE 文件路径(字符串张量)的张量，我将其解压、加载并预处理到图像的 BATCH_SIZE 中张量:
def load_and_preprocess_image(batch_of_paths, batch_of_labels):
# unpack our images from our batch
list_of_paths = tf.unstack(batch_of_paths, num=BATCH_SIZE)
list_of_images = []
for path in list_of_paths:
image = tf.read_file(path)
if image == None:
print("Unable to load image at path:" + path )
image = preprocess_image(image)
list_of_images.append(image)
# repack our now loaded and processed images into a batch
batch_of_images = tf.stack(list_of_images, 0)
return batch_of_images,batch_of_labels
有没有办法让我的BATCH_SIZE在批处理未满时从数据集传递的变量中动态读取？就我而言，我请求 BATCH_SIZE = 32，但在最后一次传递时得到 30 - 然后是 abo
更长/完整的上下文
我正在 Google collab 上使用 Tensorflow 1.1.4 并进行 eagerexecution。
我正在尝试针对 CSV 数据集训练模型(多类、多标签)，该数据集列出图像文件路径，然后列出正向存在、负向存在或未知的标签列。
我也知道，数据集还包含大量未标记的数据(我们稍后会解决这个问题:)
标题和第一行如下所示:
filepath,color_key_blue,color_key_green,color_key_luma,color_key_matte,color_saturation_desaturated,color_saturation_neutral,color_saturation_pastel,color_saturation_saturated,color_theory_analagous,color_theory_complementary,color_theory_monochrome,color_tones_blackwhite,color_tones_cool,color_tones_warm,exterior,exterior_airplane,exterior_airport,exterior_apartment,exterior_auto_body,exterior_beach,exterior_bicycle,exterior_boat,exterior_bridge,exterior_bus,exterior_bus_stop,exterior_canyon,exterior_car,exterior_castle,exterior_cathedral,exterior_cave,exterior_church,exterior_city,exterior_desert,exterior_farm,exterior_forest,exterior_glacier,exterior_helicopter,exterior_hospital,exterior_house,exterior_industrial,exterior_lake,exterior_library,exterior_mall,exterior_mansion,exterior_monastery,exterior_mosque,exterior_motorcycle,exterior_mountains,exterior_ocean,exterior_office,exterior_palace,exterior_park,exterior_parkinglot,exterior_pier,exterior_plains,exterior_playground,exterior_polar,exterior_port,exterior_restaurant,exterior_river,exterior_road,exterior_ruins,exterior_school,exterior_sidewalk,exterior_sky,exterior_skyscraper,exterior_space,exterior_spacecraft,exterior_stadium,exterior_station_gas,exterior_station_subway,exterior_station_train,exterior_store,exterior_suburb,exterior_synagogue,exterior_temple,exterior_theater,exterior_town,exterior_train,exterior_truck,exterior_tunnel,exterior_warehouse,exterior_wetlands,interior,interior_airplane_cabin,interior_airplane_cockpit,interior_airport,interior_arena,interior_auditorium,interior_auto_repair_shop,interior_bar,interior_barn,interior_bathroom,interior_bedroom,interior_boat,interior_bus,interior_cafe,interior_cafeteria,interior_car,interior_cave,interior_classroom,interior_cloister,interior_closet,interior_command_center,interior_commercialkitchen,interior_conferenceroom,interior_courtroom,interior_crypt,interior_dancefloor,interior_diningroom,interior_dungeon,interior_elevator,interior_factory,interior_foyer,interior_gym,interior_hallway,interior_helicopter,interior_hospital,interior_kitchen,interior_livingroom,interior_lobby,interior_mall,interior_meditation,interior_nave,interior_office,interior_office_cubicle,interior_office_open,interior_prayer_hall,interior_prison,interior_pulpit,interior_restaurant,interior_spacecraft,interior_stage,interior_stairwell,interior_station_bus,interior_station_fire,interior_station_police,interior_station_subway,interior_station_train,interior_store,interior_store_aisle,interior_store_checkout,interior_study,interior_subway,interior_synagogue,interior_throneroom,interior_train,interior_truck,interior_warehouse,shot_angle_aerial,shot_angle_eyelevel,shot_angle_high,shot_angle_low,shot_focus_deep,shot_focus_out,shot_focus_shallow,shot_framing_closeup,shot_framing_extemelong,shot_framing_extremecloseup,shot_framing_long,shot_framing_medium,shot_level_level,shot_level_tilted,shot_lighting_hard,shot_lighting_key_high,shot_lighting_key_low,shot_lighting_silhouette,shot_lighting_soft,shot_subject_animal,shot_subject_location,shot_subject_object,shot_subject_person,shot_subject_person_body,shot_subject_person_face,shot_subject_person_feet,shot_subject_person_hands,shot_subject_text,shot_timeofday_day,shot_timeofday_night,shot_timeofday_twilight,shot_type_master,shot_type_overtheshoulder,shot_type_portrait,shot_type_twoshot,texture_banded,texture_blotchy,texture_braided,texture_bubbly,texture_bumpy,texture_chequered,texture_cobwebbed,texture_cracked,texture_crosshatched,texture_crystalline,texture_dotted,texture_fibrous,texture_flecked,texture_frilly,texture_gauzy,texture_grid,texture_grooved,texture_honeycombed,texture_interlaced,texture_knitted,texture_lacelike,texture_lined,texture_marbled,texture_matted,texture_meshed,texture_paisley,texture_perforated,texture_pitted,texture_pleated,texture_porous,texture_potholed,texture_scaly,texture_smeared,texture_spiralled,texture_sprinkled,texture_stained,texture_stratified,texture_striped,texture_studded,texture_swirly,texture_veined,texture_waffled,texture_woven,texture_wrinkled,texture_zigzagged
path/to/image/-5.jpg,1,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1
我可以通过tf.data.experimental.make_csv_dataset将csv加载到数据集中，如下所示:
# make our data set
BATCH_SIZE = 32
FILE_PATH = ["filepath"]
COLUMN_NAMES = ["filepath", "color_key_blue","color_key_green","color_key_luma","color_key_matte","color_saturation_desaturated","color_saturation_neutral","color_saturation_pastel","color_saturation_saturated","color_theory_analagous","color_theory_complementary","color_theory_monochrome","color_tones_blackwhite","color_tones_cool","color_tones_warm","exterior","exterior_airplane","exterior_airport","exterior_apartment","exterior_auto_body","exterior_beach","exterior_bicycle","exterior_boat","exterior_bridge","exterior_bus","exterior_bus_stop","exterior_canyon","exterior_car","exterior_castle","exterior_cathedral","exterior_cave","exterior_church","exterior_city","exterior_desert","exterior_farm","exterior_forest","exterior_glacier","exterior_helicopter","exterior_hospital","exterior_house","exterior_industrial","exterior_lake","exterior_library","exterior_mall","exterior_mansion","exterior_monastery","exterior_mosque","exterior_motorcycle","exterior_mountains","exterior_ocean","exterior_office","exterior_palace","exterior_park","exterior_parkinglot","exterior_pier","exterior_plains","exterior_playground","exterior_polar","exterior_port","exterior_restaurant","exterior_river","exterior_road","exterior_ruins","exterior_school","exterior_sidewalk","exterior_sky","exterior_skyscraper","exterior_space","exterior_spacecraft","exterior_stadium","exterior_station_gas","exterior_station_subway","exterior_station_train","exterior_store","exterior_suburb","exterior_synagogue","exterior_temple","exterior_theater","exterior_town","exterior_train","exterior_truck","exterior_tunnel","exterior_warehouse","exterior_wetlands","interior","interior_airplane_cabin","interior_airplane_cockpit","interior_airport","interior_arena","interior_auditorium","interior_auto_repair_shop","interior_bar","interior_barn","interior_bathroom","interior_bedroom","interior_boat","interior_bus","interior_cafe","interior_cafeteria","interior_car","interior_cave","interior_classroom","interior_cloister","interior_closet","interior_command_center","interior_commercialkitchen","interior_conferenceroom","interior_courtroom","interior_crypt","interior_dancefloor","interior_diningroom","interior_dungeon","interior_elevator","interior_factory","interior_foyer","interior_gym","interior_hallway","interior_helicopter","interior_hospital","interior_kitchen","interior_livingroom","interior_lobby","interior_mall","interior_meditation","interior_nave","interior_office","interior_office_cubicle","interior_office_open","interior_prayer_hall","interior_prison","interior_pulpit","interior_restaurant","interior_spacecraft","interior_stage","interior_stairwell","interior_station_bus","interior_station_fire","interior_station_police","interior_station_subway","interior_station_train","interior_store","interior_store_aisle","interior_store_checkout","interior_study","interior_subway","interior_synagogue","interior_throneroom","interior_train","interior_truck","interior_warehouse","shot_angle_aerial","shot_angle_eyelevel","shot_angle_high","shot_angle_low","shot_focus_deep","shot_focus_out","shot_focus_shallow","shot_framing_closeup","shot_framing_extemelong","shot_framing_extremecloseup","shot_framing_long","shot_framing_medium","shot_level_level","shot_level_tilted","shot_lighting_hard","shot_lighting_key_high","shot_lighting_key_low","shot_lighting_silhouette","shot_lighting_soft","shot_subject_animal","shot_subject_location","shot_subject_object","shot_subject_person","shot_subject_person_body","shot_subject_person_face","shot_subject_person_feet","shot_subject_person_hands","shot_subject_text","shot_timeofday_day","shot_timeofday_night","shot_timeofday_twilight","shot_type_master","shot_type_overtheshoulder","shot_type_portrait","shot_type_twoshot","texture_banded","texture_blotchy","texture_braided","texture_bubbly","texture_bumpy","texture_chequered","texture_cobwebbed","texture_cracked","texture_crosshatched","texture_crystalline","texture_dotted","texture_fibrous","texture_flecked","texture_frilly","texture_gauzy","texture_grid","texture_grooved","texture_honeycombed","texture_interlaced","texture_knitted","texture_lacelike","texture_lined","texture_marbled","texture_matted","texture_meshed","texture_paisley","texture_perforated","texture_pitted","texture_pleated","texture_porous","texture_potholed","texture_scaly","texture_smeared","texture_spiralled","texture_sprinkled","texture_stained","texture_stratified","texture_striped","texture_studded","texture_swirly","texture_veined","texture_waffled","texture_woven","texture_wrinkled","texture_zigzagged"]
LABEL_NAMES = ["color_key_blue","color_key_green","color_key_luma","color_key_matte","color_saturation_desaturated","color_saturation_neutral","color_saturation_pastel","color_saturation_saturated","color_theory_analagous","color_theory_complementary","color_theory_monochrome","color_tones_blackwhite","color_tones_cool","color_tones_warm","exterior","exterior_airplane","exterior_airport","exterior_apartment","exterior_auto_body","exterior_beach","exterior_bicycle","exterior_boat","exterior_bridge","exterior_bus","exterior_bus_stop","exterior_canyon","exterior_car","exterior_castle","exterior_cathedral","exterior_cave","exterior_church","exterior_city","exterior_desert","exterior_farm","exterior_forest","exterior_glacier","exterior_helicopter","exterior_hospital","exterior_house","exterior_industrial","exterior_lake","exterior_library","exterior_mall","exterior_mansion","exterior_monastery","exterior_mosque","exterior_motorcycle","exterior_mountains","exterior_ocean","exterior_office","exterior_palace","exterior_park","exterior_parkinglot","exterior_pier","exterior_plains","exterior_playground","exterior_polar","exterior_port","exterior_restaurant","exterior_river","exterior_road","exterior_ruins","exterior_school","exterior_sidewalk","exterior_sky","exterior_skyscraper","exterior_space","exterior_spacecraft","exterior_stadium","exterior_station_gas","exterior_station_subway","exterior_station_train","exterior_store","exterior_suburb","exterior_synagogue","exterior_temple","exterior_theater","exterior_town","exterior_train","exterior_truck","exterior_tunnel","exterior_warehouse","exterior_wetlands","interior","interior_airplane_cabin","interior_airplane_cockpit","interior_airport","interior_arena","interior_auditorium","interior_auto_repair_shop","interior_bar","interior_barn","interior_bathroom","interior_bedroom","interior_boat","interior_bus","interior_cafe","interior_cafeteria","interior_car","interior_cave","interior_classroom","interior_cloister","interior_closet","interior_command_center","interior_commercialkitchen","interior_conferenceroom","interior_courtroom","interior_crypt","interior_dancefloor","interior_diningroom","interior_dungeon","interior_elevator","interior_factory","interior_foyer","interior_gym","interior_hallway","interior_helicopter","interior_hospital","interior_kitchen","interior_livingroom","interior_lobby","interior_mall","interior_meditation","interior_nave","interior_office","interior_office_cubicle","interior_office_open","interior_prayer_hall","interior_prison","interior_pulpit","interior_restaurant","interior_spacecraft","interior_stage","interior_stairwell","interior_station_bus","interior_station_fire","interior_station_police","interior_station_subway","interior_station_train","interior_store","interior_store_aisle","interior_store_checkout","interior_study","interior_subway","interior_synagogue","interior_throneroom","interior_train","interior_truck","interior_warehouse","shot_angle_aerial","shot_angle_eyelevel","shot_angle_high","shot_angle_low","shot_focus_deep","shot_focus_out","shot_focus_shallow","shot_framing_closeup","shot_framing_extemelong","shot_framing_extremecloseup","shot_framing_long","shot_framing_medium","shot_level_level","shot_level_tilted","shot_lighting_hard","shot_lighting_key_high","shot_lighting_key_low","shot_lighting_silhouette","shot_lighting_soft","shot_subject_animal","shot_subject_location","shot_subject_object","shot_subject_person","shot_subject_person_body","shot_subject_person_face","shot_subject_person_feet","shot_subject_person_hands","shot_subject_text","shot_timeofday_day","shot_timeofday_night","shot_timeofday_twilight","shot_type_master","shot_type_overtheshoulder","shot_type_portrait","shot_type_twoshot","texture_banded","texture_blotchy","texture_braided","texture_bubbly","texture_bumpy","texture_chequered","texture_cobwebbed","texture_cracked","texture_crosshatched","texture_crystalline","texture_dotted","texture_fibrous","texture_flecked","texture_frilly","texture_gauzy","texture_grid","texture_grooved","texture_honeycombed","texture_interlaced","texture_knitted","texture_lacelike","texture_lined","texture_marbled","texture_matted","texture_meshed","texture_paisley","texture_perforated","texture_pitted","texture_pleated","texture_porous","texture_potholed","texture_scaly","texture_smeared","texture_spiralled","texture_sprinkled","texture_stained","texture_stratified","texture_striped","texture_studded","texture_swirly","texture_veined","texture_waffled","texture_woven","texture_wrinkled","texture_zigzagged"]
def get_dataset(file_path, **kwargs):
dataset = tf.data.experimental.make_csv_dataset(
file_path,
batch_size=BATCH_SIZE, # Artificially small to make examples easier to show.
na_value="?",
num_epochs=1,
column_names=COLUMN_NAMES,
ignore_errors=True,
shuffle=True, #TEMPORARY
**kwargs)
return dataset
csv_dataset = get_dataset(csv_file_path)
然后我转换数据集以返回一个由字符串张量和张量组成的元组，其中包含int32的单维向量，每个标签包含我的1、0或-1:
# we need to split out our data set to matching file paths and the labels as a sparse vector for each label,
# containing 1, 0, -1 values for label concept present, label concept not present, dont know if present
# for example, a CSV row like
# FOLD_0/color_key_blue/1-5.jpg,1,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1
# should return:
# * a tensor containing a path like data_root + FOLD_0/color_key_blue/1-5.jpg,
# * a tensor of values [1,0,0,0, -1 etc]
# a function that returns a path, ordered dict of only the
import tensorflow.python.util
def split_csv_to_path_and_labels(csv_row_Dict):
print("Calling split_csv_to_path_and_label")
filepath = csv_row_Dict.pop('filepath')
data_root_tensor = tf.constant(data_root)
filepath = tf.strings.join([data_root_tensor, filepath], separator='')
#make a new tensor with the values of the LABEL_NAMES keys but packed into a 0, len(LABEL_NAMES) array
labels = tf.stack(list(csv_row_Dict.values()), axis=1)
return filepath, labels
#make a new data set from our csv by mapping every value to the above function
split_dataset = csv_dataset.map(split_csv_to_path_and_labels)
最后，我需要创建一个数据集，为批处理中的每个字符串张量加载图像并返回一批图像/标签元组。我定义了一个函数来加载图像，如下所示:
def load_and_preprocess_image(batch_of_paths, batch_of_labels):
# unpack our images from our batch
list_of_paths = tf.unstack(batch_of_paths, num=BATCH_SIZE)
list_of_images = []
for path in list_of_paths:
image = tf.read_file(path)
if image == None:
print("Unable to load image at path:" + path )
image = preprocess_image(image)
list_of_images.append(image)
# repack our now loaded and processed images into a batch
batch_of_images = tf.stack(list_of_images, 0)
return batch_of_images,batch_of_labels
然后通过将数据集映射到此函数来创建一个新的数据集:
image_and_labels_ds = split_dataset.map(load_and_preprocess_image, num_parallel_calls=AUTOTUNE)
由于上面的函数有一个硬编码的BATCH_SIZE，它会一直工作，直到最后一个批处理小于批处理大小，并且 TF 错误导致上述错误。
我该如何解决这个问题!？
我看过Dynamic batch size in tensorflow但我不清楚这是怎么回事
最佳答案
我能够通过使用 tf.map_fn 并映射一个处理单个字符串张量并返回预处理(标准化)图像张量的函数来解决此问题:
主要的tf.map_fn调用:
def load_and_preprocess_image_batch(batch_of_paths, batch_of_labels):
batch_of_images = tf.map_fn(load_and_preprocess_single_image_from_tensor, batch_of_paths, dtype=tf.float32)
return batch_of_images, batch_of_labels
还有 load_and_preprocess_single_image_from_tensor 的声明以及我为关心的人提供的预处理代码:
IMG_SIZE = 224
def preprocess_image(image):
image = tf.image.decode_jpeg(image, channels=3)
image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])
image /= 255.0 # normalize to [0,1] range
return image
def load_and_preprocess_single_image_from_tensor(path):
image = tf.read_file(path)
if image == None:
print("Unable to load image at path:" + path )
return preprocess_image(image)
最后，确保我的数据集使用 load_and_preprocess_image_batch 函数:
image_and_labels_ds = split_dataset.map(load_and_preprocess_image_batch, num_parallel_calls=AUTOTUNE)
关于python - TensorFlow 训练 - "Batch size"和 tf.unpack - 解包非 "batch sized"动态值？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/57719531/