Cannot Copy Large (5 Gb) Files With Awscli 1.5.4

I have problem with aws-cli, I did a yum update, it updated awscli (among other things) and now awscli fails on large files (e.g. 5.1 Gb) with SignatureDoesNotMatch. The exact same

Solution 1:

The aws-cli can certainly handle uploads of 5GB and higher.

I launched an EC2 instance and was able to complete it successfully:

$ /usr/bin/aws --version
aws-cli/1.3.6 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64

$ ddif=/dev/zero of=small bs=1K count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000388469 s, 2.6 MB/s

$ ddif=/dev/zero of=big bs=5M count=1000
1000+0 records in
1000+0 records out
5242880000 bytes (5.2 GB) copied, 73.733 s, 71.1 MB/s

$ aws s3 mb s3://target77
make_bucket: s3://target77/

$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small

$ aws s3 cp big s3://target77/big
upload: ./big to s3://target77/big       

$ aws s3 ls s3://target77/
2014-11-17 09:44:42 5242880000 big
2014-11-17 09:44:29       1024 small

So far, so good!

I then updated to a newer version of the CLI and tried again:

$ sudo yum update
...

$ /usr/bin/aws --version
aws-cli/1.5.5 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64

$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small

$ aws s3 cp big s3://target77/big
upload failed: ./big to s3://target77/bigs) remaining
A client error (SignatureDoesNotMatch) occurred when calling the UploadPart operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I repeated the experiment with the EC2 instance and S3 bucket in the same region, and in different regions. Both situations eventually gave the error, but also sometimes worked.

So, it does seem to be an issue with the updated aws-cli. Plus, the error occurred after a random number of 'parts' when I tried re-executing the copy command.

The SignatureDoesNotMatch error is related to security credentials rather than the Amazon S3 service. It sounds similar to issues that people have reported on GitHub: Recurring SignatureDoesNotMatch client error #995

So, I'd recommend using an earlier version of aws-cli until the issue is resolved.

Solution 2:

It wouldn't surprise me if it's a bug in aws-cli. I'd recommend using s3cmd instead for copying files to/from S3 and doing all your other S3-based interactions.