python - Python Lambda打包和部署Bash

Question

从我的buildspec.yml（在CodeBuild中）执行一个bash文件（下面的代码），以复制，重命名，压缩，移动，然后将168个Python lambda上载到S3。 Lambda zip的平均大小为381KB。

在本地，运行Mojave 10.14的Mac，bash每次都能按预期成功执行。

在构建容器Linux（Ubuntu 18.04-aws / codebuild / standard：2.0）中运行时，脚本永远不会完全成功，但也不会引发错误。最好是将5-9个zip文件上载到S3中，即使该脚本还可以做到。

我已经尝试了许多纠正措施，从优化bash（例如不复制到_staging目录和不移动zip）到在S3上传中使用复制而不是同步，到将环境计算能力提高到最大。

关于此问题，我唯一的“线索”是在未预期到时开始记录输出。
例如

Copying Lambdas to S3 Lambda Repository    
adding: psycopg2/psycopg1.py (deflated 56%) 
adding: psycopg2/__init__.py (deflated 65%)

代替

Copying Lambdas to S3 Lambda Repository 
upload: _deploy/beta-forgot-password.zip to s3://beta-com-lambdas/20190731142007/beta-forgot-password.zip
upload: _deploy/beta-change-password.zip to s3://beta-com-lambdas/20190731142007/beta-change-password.zip

任何想法表示赞赏。提前致谢。

#!/bin/bash
BUILD_ENVIRONMENT=$1
UNIQUE_KEY=$2
echo 'Executing Lambda Package and Deployment for: '$BUILD_ENVIRONMENT' environment';
echo 'Unique Lambda S3 Key: ' $UNIQUE_KEY
echo 'Creating Lambda Staging Directory';
mkdir ./_staging ;
echo 'Copying Lambdas to Staging Directory' ;
cp -r ./lambdas/* ./_staging;
echo 'Renaming Lambdas in Staging Directory' ;
cd _staging
for i in */; do mv "$i" "$BUILD_ENVIRONMENT-${i%/}" ; done
echo 'Zipping Lambdas in Staging Directory' ;
for i in */; do (cd "$i"; zip -r "../${i%/}.zip" .) &> /dev/null ;
echo "Zipping: ${i%/}.zip" ; done
echo 'Creating Lambda Deploy Directory' ;
mkdir ./_deploy ;
echo 'Moving Lambda Deployment Packages' ;
find . -name "*.zip" -exec mv {} ./_deploy \;
echo "Sync Lambdas to S3 Lambda Repository "
aws sync ./_deploy s3://$BUILD_ENVIRONMENT-smc-com-lambdas/$UNIQUE_KEY/
echo "Lambda Package and Deployment Completed"

Answer 1

插入“ set -x”以调试构建容器中的脚本，并且在梳理CloudWatch日志后，我能够确定zip执行尚未完全完成，然后再移至下一个执行。我插入了一个Linux运算符“ &&”，并将两个命令链接在一起-zip和echo。

“ &&”运算符的插入可确保zip命令在echo命令执行之前已完全完成。

我还重构了脚本，以移动生成的zip文件，将文件移动到S3，而不是复制它们，并删除了所有流程构件，以减少buildspec.yml发送到S3的整体构建构件输出中的混乱情况

问题解决了。

#!/bin/bash
set -x
# set build environment from injected parameter
BUILD_ENVIRONMENT=$1
UNIQUE_KEY=$2
echo 'Executing Lambda Package and Deployment for: '$BUILD_ENVIRONMENT' environment';
echo 'Unique Lambda S3 Key: ' $UNIQUE_KEY
echo 'Creating Lambda Staging Directory';
mkdir ./_staging ;
echo 'Copying Lambdas to Staging Directory' ;
cp -r ./lambdas/* ./_staging;
echo 'Renaming Lambdas in Staging Directory' ;
cd _staging
for i in */; do mv "$i" "$BUILD_ENVIRONMENT-${i%/}" ; done
mkdir ./_deploy ;
echo 'Zipping Lambdas in Staging Directory & Moving to Deploy Directory' ;
for i in */; do (cd "$i"; zip -r "../_deploy/${i%/}.zip" . && echo "Zipped & Moved: ${i%/}.zip" ) ; done
echo 'Moving Lambda Packages to S3 Lambda Repository'
aws s3 mv --recursive ./_deploy s3://$BUILD_ENVIRONMENT-smc-com-lambdas/$UNIQUE_KEY/
echo 'Lambda Package and Deployment Completed'
echo 'Removing Lambda P & D Artifacts - Staging directory'
cd ..
rm -r _staging
echo 'Lambda Bash Complete'