python - 如何在keras中构建嵌入层

Question

我正在尝试按照 Francois Chollet 书中的教程之一在 tensorflow 中构建文本分类模型。我试图首先创建一个嵌入层，但它在这个阶段不断中断。

我的逻辑如下:

• 从文本字符串列表(X)和整数列表(y)开始。

• 对文本数据进行标记、向量化和填充至最长序列长度

• 将每个整数标签转换为一个热编码数组

• 将输入送入嵌入层:
  • input_dim = 唯一标记/单词的总和(在我的例子中为 1499)
  • output_dim = 嵌入向量的维度大小(从 32 开始)
  • input_length = 最大序列的长度，与序列填充到的维度相同(在我的例子中为 295)
• 使用 relu 将嵌入结果传递到 32 个隐藏单元密集层
• 将它们传递到具有 softmax 的 3 个隐藏单元密集层中以预测 3 个类别

有人可以向我解释一下我在这里犯了什么错误吗？我以为我理解了如何实例化嵌入层，但这不是正确的理解吗？

这是我的代码:

# read in raw data
df = pd.read_csv('text_dataset.csv')
samples = df.data.tolist() # list of strings of text
labels = df.sentiment.to_list() # list of integers

# tokenize and vectorize text data to prepare for embedding
tokenizer = Tokenizer()
tokenizer.fit_on_texts(samples)
sequences = tokenizer.texts_to_sequences(samples)
word_index = tokenizer.word_index
print(f'Found {len(word_index)} unique tokens.')

# setting variables
vocab_size = len(word_index) # 1499
# Input_dim: This is the size of the vocabulary in the text data.
input_dim = vocab_size # 1499
# This is the size of the vector space in which words will be embedded.
output_dim = 32 # recommended by tf
# This is the length of input sequences
max_sequence_length = len(max(sequences, key=len)) # 295
# train/test index splice variable
training_samples = round(len(samples)*.8)

# data = pad_sequences(sequences, maxlen=max_sequence_length) # shape (499, 295)
# keras automatically pads to maxlen if left without input
data = pad_sequences(sequences)

# preprocess labels into one hot encoded array of 3 classes ([1., 0., 0.])
labels = to_categorical(labels, num_classes=3, dtype='float32') # shape (499, 3)

# Create test/train data (80% train, 20% test)
x_train = data[:training_samples]
y_train = labels[:training_samples]
x_test = data[training_samples:]
y_test = labels[training_samples:]

model = Sequential()
model.add(Embedding(input_dim, output_dim, input_length=max_sequence_length))
model.add(Dense(32, activation='relu'))
model.add(Dense(3, activation='softmax'))
model.summary()

model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train,
          y_train,
          epochs=10,
          batch_size=32,
          validation_data=(x_test, y_test))

当我运行此程序时，出现此错误:

Found 1499 unique tokens.
Model: "sequential_23"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_21 (Embedding)     (None, 295, 32)           47968     
_________________________________________________________________
dense_6 (Dense)              (None, 295, 32)           1056      
_________________________________________________________________
dense_7 (Dense)              (None, 295, 3)            99        
=================================================================
Total params: 49,123
Trainable params: 49,123
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-144-f29ef892e38d> in <module>()
     51           epochs=10,
     52           batch_size=32,
---> 53           validation_data=(x_test, y_test))

2 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
    129                         ': expected ' + names[i] + ' to have ' +
    130                         str(len(shape)) + ' dimensions, but got array '
--> 131                         'with shape ' + str(data_shape))
    132                 if not check_batch_axis:
    133                     data_shape = data_shape[1:]

ValueError: Error when checking target: expected dense_7 to have 3 dimensions, but got array with shape (399, 3)

为了排除故障，我一直在注释掉各层以尝试查看发生了什么。我发现问题一直持续到第一层，这让我觉得我对 Embedding 层的理解很差。见下文:

model = Sequential()
model.add(Embedding(input_dim, output_dim, input_length=max_sequence_length))
# model.add(Dense(32, activation='relu'))
# model.add(Dense(3, activation='softmax'))
model.summary()

结果是:

Found 1499 unique tokens.
Model: "sequential_24"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_22 (Embedding)     (None, 295, 32)           47968     
=================================================================
Total params: 47,968
Trainable params: 47,968
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-150-63d1b96db467> in <module>()
     51           epochs=10,
     52           batch_size=32,
---> 53           validation_data=(x_test, y_test))

2 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
    129                         ': expected ' + names[i] + ' to have ' +
    130                         str(len(shape)) + ' dimensions, but got array '
--> 131                         'with shape ' + str(data_shape))
    132                 if not check_batch_axis:
    133                     data_shape = data_shape[1:]

ValueError: Error when checking target: expected embedding_22 to have 3 dimensions, but got array with shape (399, 3)

Answer 1

keras 中的密集层预计仅采用二维[BATCH_SIZE, N] 的平面输入。句子嵌入层的输出有 3 个维度:[BS, SEN_LENGTH, EMBEDDING_SIZE]。

有两个选项可以解决这个问题:

1. 在第一个密集层之前压平嵌入层的输出:model.add(Flatten())；
2. 使用卷积层(建议):model.add(Conv1D(filters=32, kernel_size=8,activation='relu'))