python-2.7 - Tensorflow:用于回归的 MLP 显示测试集的相同预测值

Question

这是我第一次使用 Tensorflow。这是用于回归的基本 MLP 的实现。该代码是根据标准 MNIST 分类器修改的:

https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/multilayer_perceptron.py

我只将输入、输出、超参数、成本函数更改为

cost = tf.reduce_mean(tf.square(pred-y))

并将其添加到 out_layer 之后

out = tf.sigmoid(out_layer)

我正在使用 5 个特征的 4440 个输入数据进行训练，并在 2956 个数据上进行测试。在第三个时期之后，训练集的所有值都相同。问题是对于测试集，我得到了相同的预测值。

Training started...

Epoch 1

Loss= 0.001181 , y_pred= 0.485037 , y_actual= 0.450664
Loss= 0.014749 , y_pred= 0.206193 , y_actual= 0.32764
Loss= 0.000000 , y_pred= 0.323003 , y_actual= 0.323016
Loss= 0.028031 , y_pred= 0.276502 , y_actual= 0.109078
Loss= 0.024109 , y_pred= 0.283097 , y_actual= 0.127827
Loss= 0.000688 , y_pred= 0.222412 , y_actual= 0.196174
Loss= 0.022695 , y_pred= 0.285257 , y_actual= 0.13461
Loss= 0.043803 , y_pred= 0.228042 , y_actual= 0.437334
Loss= 0.002999 , y_pred= 0.251055 , y_actual= 0.30582
Epoch 2

Loss= 0.041213 , y_pred= 0.247654 , y_actual= 0.450664
Loss= 0.005612 , y_pred= 0.252729 , y_actual= 0.32764
Loss= 0.001075 , y_pred= 0.29023 , y_actual= 0.323016
Loss= 0.018882 , y_pred= 0.246489 , y_actual= 0.109078
Loss= 0.018060 , y_pred= 0.262215 , y_actual= 0.127827
Loss= 0.001204 , y_pred= 0.23087 , y_actual= 0.196174
Loss= 0.018622 , y_pred= 0.271072 , y_actual= 0.13461
Loss= 0.038593 , y_pred= 0.240883 , y_actual= 0.437334
Loss= 0.002938 , y_pred= 0.251615 , y_actual= 0.30582
Epoch 3

Loss= 0.041822 , y_pred= 0.24616 , y_actual= 0.450664
Loss= 0.005700 , y_pred= 0.252141 , y_actual= 0.32764
Loss= 0.001073 , y_pred= 0.29026 , y_actual= 0.323016
Loss= 0.018882 , y_pred= 0.24649 , y_actual= 0.109078
Loss= 0.018059 , y_pred= 0.26221 , y_actual= 0.127827
Loss= 0.001203 , y_pred= 0.230861 , y_actual= 0.196174
Loss= 0.018622 , y_pred= 0.271074 , y_actual= 0.13461
Loss= 0.038595 , y_pred= 0.240879 , y_actual= 0.437334
Loss= 0.002938 , y_pred= 0.251613 , y_actual= 0.30582
Epoch 4

Loss= 0.041822 , y_pred= 0.24616 , y_actual= 0.450664
Loss= 0.005700 , y_pred= 0.252141 , y_actual= 0.32764
Loss= 0.001073 , y_pred= 0.29026 , y_actual= 0.323016
Loss= 0.018882 , y_pred= 0.24649 , y_actual= 0.109078
Loss= 0.018059 , y_pred= 0.26221 , y_actual= 0.127827
Loss= 0.001203 , y_pred= 0.23086 , y_actual= 0.196174
Loss= 0.018623 , y_pred= 0.271074 , y_actual= 0.13461
Loss= 0.038595 , y_pred= 0.240879 , y_actual= 0.437334
Loss= 0.002938 , y_pred= 0.251613 , y_actual= 0.30582

Training Finished!

Testing started...

Loss= 0.010336 , y_pred= 0.246348 , y_actual= 0.348012
Loss= 0.123387 , y_pred= 0.246348 , y_actual= 0.597613
Loss= 0.005033 , y_pred= 0.246348 , y_actual= 0.175401
Loss= 0.022147 , y_pred= 0.246348 , y_actual= 0.0975305
Loss= 0.004484 , y_pred= 0.246348 , y_actual= 0.313307
Loss= 0.010506 , y_pred= 0.246348 , y_actual= 0.348845
Loss= 0.000052 , y_pred= 0.246348 , y_actual= 0.239131

我已经尝试了描述同一问题的各种帖子提供的所有可能的解决方案。就像数据被打乱和归一化一样，y 和 pred 的维度是相同的。

1) TensorFlow always converging to same output for all items after training

2) MLP in tensorflow for regression... not converging

3) tensorflow deep neural network for regression always predict same results in one batch

这是代码。非常感谢。

# In[67]:

import tensorflow as tf
import numpy as np


# In[68]:

# Parameters
learning_rate = 0.01
epoch = 1
dropout = 0.75
# Network Parameters
n_hidden_1 = 256 # 1st layer number of features
n_hidden_2 = 256 # 2nd layer number of features
n_hidden_3 = 256
n_hidden_4 = 256
n_input = 5
n_val = 1

train_set = 4440

# tf Graph input
x = tf.placeholder("float", [None, n_input], name = "x")
y = tf.placeholder("float", [None, n_val], name = "y")
# keep_prob = tf.placeholder(tf.float32) #dropout (keep probability)


# In[69]:

# Create model
def multilayer_perceptron(x, weights, biases):

    # Hidden layer with RELU activation
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)

    # Hidden layer with RELU activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    layer_2 = tf.nn.relu(layer_2)

    # Hidden layer with RELU activation
    layer_3 = tf.add(tf.matmul(layer_2, weights['h3']), biases['b3'])
    layer_3 = tf.nn.relu(layer_3)

    # Hidden layer with RELU activation
    layer_4 = tf.add(tf.matmul(layer_3, weights['h4']), biases['b4'])
    layer_4 = tf.nn.relu(layer_4)

    # Output layer with linear activation
    out_layer = tf.matmul(layer_4, weights['out']) + biases['out']
    out = tf.sigmoid(out_layer)
    return out


# In[70]:

# Store layers weight & bias
weights = {
    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1], mean=0.0, stddev=0.01 ,dtype=tf.float32, name = "h1")),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2], mean=0.0, stddev=0.01 ,dtype=tf.float32, name = "h2")),
    'h3': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "h3")),
    'h4': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_4], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "h4")),
    'out': tf.Variable(tf.random_normal([n_hidden_4, n_val], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "out"))
}
biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "b1")),
    'b2': tf.Variable(tf.random_normal([n_hidden_2], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "b2")),
    'b3': tf.Variable(tf.random_normal([n_hidden_3], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "b3")),
    'b4': tf.Variable(tf.random_normal([n_hidden_4], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "b4")),
    'out': tf.Variable(tf.random_normal([n_val], mean=0.0, stddev=0.01 ,dtype=tf.float32,name = "out"))
}
# Construct model
pred = multilayer_perceptron(x, weights, biases)
# pred = tf.transpose(pred)

# Define loss and optimizer
cost = tf.reduce_mean(tf.square(pred-y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# Initializing the variables
init = tf.global_variables_initializer()


# In[71]:

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    # Training
    print "Training started...\n"

    for ep in range(1,epoch+1):

        print "Epoch",ep
        print
        num = 0
        with open('norm_rand_feature_y.csv') as f:

            for line in f:

                data = line.split(",")
                x_temp = data[0:5]
                y_temp = data[5]

                x_temp = np.asarray(x_temp)
                x_temp = x_temp.reshape(1,x_temp.shape[0])
                x_temp = x_temp.astype(np.float32)

                y_temp = np.asarray(y_temp)
                y_temp = y_temp.reshape(1,1)
                y_temp = y_temp.astype(np.float32)

                sess.run(optimizer, feed_dict={x: x_temp, y: y_temp})

                loss,y_pre = sess.run([cost,pred], feed_dict={x: x_temp,
                                                  y: y_temp})

#                 print tuple(pred.get_shape().as_list())
#                 print y.shape



                if num%500 == 0:
                    print "Loss= " + "{:.6f}".format(loss),  ", y_pred=",y_pre[0][0],  ", y_actual=",y_temp[0][0]

                num = num+1
                if num == train_set:
                    break

#     variables_names =[v.name for v in tf.trainable_variables()]
#     values = sess.run(variables_names)
#     for k,v in zip(variables_names, values):
#         print(k, v)

#     print sess.run("h1", feed_dict={x: x_temp,y: y_temp, keep_prob:1.0})
    print "Training Finished!\n"

    #Testing
    y_value = list()
    y_actual = list()
    error = 0
    num=0
    print "Testing started...\n"
    with open('norm_rand_feature_y.csv') as f:

            for j in range(train_set):
                f.next()

            for line in f:

                data = line.split(",")
                x_temp = data[0:5]
                y_temp = float(data[5])

                x_temp = np.asarray(x_temp)
                x_temp = x_temp.astype(np.float32)
                x_temp = x_temp.reshape(1,x_temp.shape[0])

                y_temp = np.asarray(y_temp)
                y_temp = y_temp.reshape(1,1)
                y_temp = y_temp.astype(np.float32)

                loss = sess.run(cost, feed_dict={x: x_temp, y:y_temp})


                y_pred = sess.run(pred, feed_dict={x: x_temp})

                print "Loss= " + "{:.6f}".format(loss),  ", y_pred=",y_pre[0][0],  ", y_actual=",y_temp[0][0]

                y_value.append(y_pred[0][0])
                y_actual.append(y_temp)
                error = error + abs(y_pred[0][0] - y_temp)

#                 num = num+1
#                 if num == 100:
#                     break

    print
    print "Testing Finished!\n"
    error = error/(7396-train_set+1)
    print "Total error:",error[0][0]
    y_row = zip(y_value,y_actual)
    np.savetxt("test_y_mlp.csv", y_row, delimiter=",")

Answer 1

我首先会尝试什么:

• 尝试调整学习率。特别是因为您的批量大小为 1，所以它可能太高了。
• 增加批处理大小(根据您的情况，按批处理提供数据，可能从 16 个批处理开始)
• 测试实现是否正确的一个简单方法是尝试过度拟合极少量的数据。取 10 个样本并对其进行 1000 次迭代，您应该能够获得非常低的损失(至少 1E-6)