amazon-web-services - 上传到S3的批处理文件

Question

我需要开发一个批处理文件，它将多个小文本文件 (4KB – 100KB) 发送到 S3 存储桶中的一个文件夹。灵感来自 http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash 的 bash 脚本，我开发了下面列出的批处理脚本。此脚本使用以下依赖项:

• openssl.exe(以及两个库 libeay32.dll 和 ssleay32.dll)

• b64.exe ( https://sourceforge.net/projects/base64/)

• curl.exe

• grep.exe

效果不错；然而，有时一个文件或多个文件无法上传，我会收到“HTTP/1.1 403 Forbidden/SignatureDoesNotMatch”。这似乎是一个签名问题，可能是在创建签名时。这种行为似乎没有规律可循。为了测试，我使用 8 个文本文件上传。我可以多次运行它，导致每次都上传所有文件，而下一次脚本将只上传 8 个文件中的 5、6 或 7 个。我有点不知所措，为什么会这样。如有任何建议，我们将不胜感激。

@echo off
setlocal enableDelayedExpansion

for /f %%f in ('dir /b *.txt') do (

REM Set date/time variables
for /f "tokens=1-4 delims=/ " %%a in ('date /t') do set dow=%%a&&set day=%%c&& set month=%%b&&set year=%%d
for /f "tokens=1-4 delims=:,. " %%h in ('echo %time%') do set hour=%%h&set min=%%i&set sec1=%%j&set sec2=%%k

REM Obtain three letter month value for DateValue HTTP Header
IF !month! == 01 set mname=Jan
IF !month! == 02 set mname=Feb
IF !month! == 03 set mname=Mar
IF !month! == 04 set mname=Apr
IF !month! == 05 set mname=May
IF !month! == 06 set mname=Jun
IF !month! == 07 set mname=Jul
IF !month! == 08 set mname=Aug
IF !month! == 09 set mname=Sep
IF !month! == 10 set mname=Oct
IF !month! == 11 set mname=Nov
IF !month! == 12 set mname=Dec

REM Set time as a 4 digit value if leading zero does not exist
for /f "tokens=1" %%u in ('echo %time%') do set t=%%u
IF "!hour:~1,1!"==":" set t=0!hour!

REM Obtain the ActiveBias value and convert to decimal 
for /f "tokens=3" %%a in ('reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v ActiveTimeBias ^| grep -i "ActiveTimeBias"') do set /a abias=%%a

REM Set the + or - sign variable to reflect the timezone offset
IF "!abias:~0,1!"=="-" (set si=+) ELSE (set si=-)
for /f "tokens=1 delims=-" %%t in ('echo !abias!') do set tzc=%%t

REM Calculate to obtain floating points (decimal values)
set /a tzd=100*!tzc!/60

REM Calculate the active bias to obtain the hour
set /a tze=!tzc!/60

REM Set the minutes based on the result of the floating point calculation
IF "!tzd!"=="0" (set en=00 && set si=)
IF "!tzd:~1!"=="00" (set en=00) ELSE IF "!tzd:~2!"=="00" (set en=00 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="50" (set en=30) ELSE IF "!tzd:~2!"=="50" (set en=30 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="75" (set en=45) ELSE IF "!tzd:~2!"=="75" (set en=45 && set tz=!tzd:~0,2!) 

REM Adding a 0 to the beginning of a single digit hour value
IF !tze! LSS 10 (set tz=0!tze!) 

REM Set the date/timestamp to meet required format
set dateValue=!dow!, !day! !mname! !year! !hour!:!min!:!sec1! !si!!tz!!en!

REM Preparing the HTTP header field
set file=%%f
set bucket=yourbucketname
set resource=/!bucket!/upload/!file!
set contentType=text/plain

REM You MUST have two returns after set NL=^
setlocal enableDelayedExpansion
set NL=^


set stringToSign=PUT!NL!!NL!!contentType!!NL!!dateValue!!NL!!resource!
<nul set /p ".=!stringToSign!" > put.tmp

set S3KEY="Your S3 Key Here"
set S3SECRET="Your S3 Secrect Key Here"

for /f "tokens=*" %%a in ('type put.tmp ^| openssl sha1 -hmac !S3SECRET! -binary ^| b64 -e') do set signature=%%a

REM Sending the data
curl -vvv --no-alpn --http2 -1 -S -X PUT -T "!file!" -H "Host: !bucket!.s3.amazonaws.com" -H "Date: !dateValue!" -H "Content-Type: !contentType!" -H "Authorization: AWS !S3KEY!:!signature!" "https://!bucket!.s3.amazonaws.com/upload/!file!"

REM Reset Variables
set "day="
set "month="
set "year="
set "retry="
set "dow="
set "hour="
set "min="
set "sec1="
set "sec2="
set "dateValue="
set "file="
set "mname="
set "bucket="
set "resource="
set "contentType="
set "stringToSign="
set "S3KEY="
set "S3SECRET="
set "t="

del put.tmp

)

Answer 1

为什么不使用具有显式 S3 支持的 Windows 命令行客户端？无论如何，您的批处理文件离不开第 3 方依赖项。

亚马逊有自己的aws-cli .

---

如果您需要更轻量级的解决方案，请使用 WinSCP:

winscp.com /log=s3.log /command ^
    "open s3://%S3KEY%:%S3SECRET%@s3.amazonaws.com/" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

您需要URL-encode special characters in the credentials . WinSCP GUI 可以 generate an S3 script template ，就像上面那个，给你的。

或者，从 WinSCP 5.19 开始，您可以使用 -username和 -password不需要任何编码的开关:

winscp.com /log=s3.log /command ^
    "open s3://s3.amazonaws.com/ -username=%S3KEY% -password=%S3SECRET%" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

(我是 WinSCP 的作者)