我试图了解为什么我的本地结果比 gcloud 结果更好。
在本地，我运行这样的工作:
gcloud ml-engine local train --module-name trainer.task --package-path trainer -- --vocabulary-file trainer/data/vocab.txt --class-files $CLASS_FILES --job-dir trainer/lr0001 --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV
对于 gcloud，我运行
gcloud ml-engine jobs submit training $JOBNAME --job-dir gs://.../lr0001 --module-name trainer.task --package-path trainer --region us-west1 --runtime-version 1.10 -- --vocabulary-file gs://.../vocab.txt --class-files $GS_CLASS_FILES --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV
我已经修复了种子，运行了多次，检查了 python 2 与 python 3，但 gcloud 结果仍然比我本地运行的结果差。
我发现的最后一点线索是本地日志如下所示:
INFO:tensorflow:loss = 0.63639945, step = 401 (0.170 sec)
INFO:tensorflow:global_step/sec: 485.821
INFO:tensorflow:loss = 0.61793035, step = 501 (0.206 sec)
INFO:tensorflow:global_step/sec: 490.795
INFO:tensorflow:loss = 0.5869169, step = 601 (0.204 sec)
INFO:tensorflow:global_step/sec: 619.825
INFO:tensorflow:loss = 0.5738391, step = 701 (0.161 sec)
INFO:tensorflow:global_step/sec: 605.698
INFO:tensorflow:loss = 0.51589084, step = 801 (0.165 sec)
而 gcloud 日志看起来像是加倍了之类的
I master-replica-0 loss = 0.40115586, step = 2202 (0.367 sec) master-replica-0
I master-replica-0 global_step/sec: 555.434 master-replica-0
I master-replica-0 global_step/sec: 498.601 master-replica-0
I master-replica-0 loss = 0.4367655, step = 2402 (0.470 sec) master-replica-0
I master-replica-0 global_step/sec: 366.906 master-replica-0
I master-replica-0 global_step/sec: 408.556 master-replica-0
I master-replica-0 loss = 0.41198668, step = 2602 (0.492 sec) master-replica-0
I master-replica-0 global_step/sec: 388.73 master-replica-0
I master-replica-0 global_step/sec: 380.982 master-replica-0
I master-replica-0 loss = 0.35386887, step = 2802 (0.522 sec) master-replica-0
I master-replica-0 global_step/sec: 401.002 master-replica-0
I master-replica-0 global_step/sec: 465.647 master-replica-0
I master-replica-0 loss = 0.4420835, step = 3002 (0.417 sec) master-replica-0
如有任何指点，我们将不胜感激!
到目前为止，我在互联网上找到的唯一的就是这些没有答案的问题:
Results of training a Keras model different on Google Cloud
Differents outputs from predictions using Tensorflow from same data?
最佳答案
根据日志，我猜测正在发生的情况是 gcloud on cloud 正在获取检查点并恢复训练(请参阅步骤数)。您能否进行以下测试:从头开始在本地运行并确保不存在输出模型文件夹，然后在云上重复相同的设置并进行比较。
关于tensorflow - 与本地训练相比，云训练的结果更差，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/53699535/