6

We are using Amazon S3 for storing up to 500K data. We have a .NET 4.0 web service installed on EC2 instance, which makes the PutObject call with 500K data.

The problem is that when we make more than 100 simultaneous calls (with unique S3 keys) to this service, respectively to the S3, and the EC2 instance CPU is hitting 100%. We made some profiling on the web service and it shows that 99% of the processing time is taken by AmazonS3Client.PutObject method.

We've tried configuring the S3 client to use HTTP (instead of default HTTPS) , and also played a little with the S3 keys generation scheme, but nothing helped. This article Amazon S3 PutObject is very slow didn't help either.

Our S3 key schema is: "111_[ incrementing ID ].txt"

This extensive CPU usage does not happen if we use a shorter data - like less than 1K.

Can you give us some guidance what could be done to improve CPU performance or where to look?

And here's the source code for that call:

string fileName = "111_" + Id + ".txt";
using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(RegionEndpoint.XXXX))
        {
            try
            {
PutObjectRequest request = new PutObjectRequest();
request.WithContentBody(dataRequest.Base64Data)
                 .WithBucketName(bucketName)
                 .WithKey(fileName);

S3Response response = client.PutObject(request);
response.Dispose();
            }
            catch (AmazonS3Exception amazonS3Exception)
            {
              //Handle exceptiom...
            }
        }

Thanks!

Alex from Jitbit
  • 39,345
  • 13
  • 111
  • 109
Doncho P
  • 61
  • 2
  • You may find the answers to [this question](https://forums.aws.amazon.com/thread.jspa?messageID=400164&tstart=0) helpful. – David Schwartz Nov 14 '12 at 14:00
  • I have no .net experience but reading this article http://improve.dk/archive/2011/11/07/pushing-the-limits-of-amazon-s3-upload-performance.aspx I have a hunch that the base64 call might be causing the CPU load. Can you try with a 500 kbyte string that is not generated on the fly? – tix3 Dec 24 '12 at 20:21
  • If you are using the SDK take a look at the source to figure out how the multipart upload is being handled. It almost seems like a case of chunk sizes causing issues with performance. Also another note about naming objects which may not be entirely related to this question but useful to follow: http://aws.typepad.com/aws/2012/03/amazon-s3-performance-tips-tricks-seattle-hiring-event.html – Keshi Jul 23 '13 at 18:19

1 Answers1

0

I would give a try to the TransferUtility higher level class, rather than low level PutObject calls. http://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/T_Amazon_S3_Transfer_TransferUtility.htm

Sébastien Stormacq
  • 12,326
  • 2
  • 31
  • 55