Amazon claims that the largest file you can store on S3 is 5 GB. In reality, uploading a file that large to Amazon S3 is nearly impossible.

While we were testing TyphoonDrive, we noticed that very large files would sometimes fail on upload. The same files would then work when TyphoonDrive tried again. And sometimes, the files went through the first time without any problems.

After a great deal of testing, and research, we determined that it was either a problem with Amazon S3 timing out, or simply the nature of trying to upload such large files through the Internet.

I think it’s a little bit of both.

The rememedy that most people and organizations seem to recommend, is to break your files up into smaller pieces before uploading.  In otherwords, ‘Hey doctor, it hurts when I do this…”,  Doctor says, “Well then don’t do that!”

In our testing, we found that when an upload was over 250 MB, we started seeing failures.

A smoking gun appeared recently, when Amazon upgraded the AWS console to allow management of S3 buckets. If you try uploading a really big file, here is what it says:

“The AWS Management Console currently supports uploading files of up to 300 mb in size.”

300 MB doesn’t mean anything, there is no technical reason for that size. So let me point out the white elephant in the room, and say that Amazon knows that trying to upload more than 300 MB at once is error prone.

Interestingly, that size limit does coincide with our testing. Obviously, the answer here is for Amazon to introduce some way of re-continuing a failed upload.

But until that happens, you can either ship your files off to Amazon, and have them physically load them onto S3 for you, or you will need to break big files into easier to digest bite size pieces.

Bon Appétit!