Aws S3 File Upload Your Post Request Fields Preceeding the Upload File Was Too Large

When information technology comes to file uploads performed by customer apps, "traditionally," in a "serverful" world, we might use the post-obit approach:

1. on the client side, the user submits a form and the upload begins
2. once the upload has been completed, we practice all of the necessary work on the server, such equally bank check the file type and size, sanitize the needed data, perchance do image optimizations, and then, finally, move the file to a preferred location, be it another storage server or maybe S3.

Although this is pretty straight frontward, at that place are a few downsides:

1. Uploading files to a server tin can negatively impact its system resources (RAM and CPU), especially when dealing with larger files or epitome processing.
2. If you are storing files on a separate storage server, yous also don't have unlimited disk infinite, which ways that, as the file base grows, you lot'll need to do upgrades.
3. Oh, yeah, and did I mention backups?
4. Security — there are never plenty preventive steps that you tin implement in this section.
5. Nosotros constantly demand to monitor these servers in gild to avoid downtime and provide the best possible user experience.

Woah! 😰

But, luckily, there's an easier and meliorate style to perform file uploads! By using pre-signed Mail service information, rather than our own servers, S3 enables united states to perform uploads directly to it, in a controlled, performant, and very safety mode. 🚀

You might be asking yourself: "What is pre-signed Postal service data and how does it all piece of work together." Well, sit back and relax, because, in this short post, nosotros'll cover everything you need to know to get y'all started.

For demonstration purposes, nosotros'll also create a simple app for which we'll use a trivial bit of React on the frontend and a simple Lambda office (in conjunction with API gateway) on the backend.

Let'south go!

How does information technology work?

On a high level, it is basically a ii-step process:

1. The customer app makes an HTTP request to an API endpoint of your choice (1), which responds (2) with an upload URL and pre-signed Postal service data (more than information most this soon). Note that this asking does not contain the actual file that needs to be uploaded, but it can contain boosted information if needed. For case, yous might want to include the file name if for some reason you need information technology on the backend side. You are gratis to send annihilation you demand, but this is certainly non a requirement. For the API endpoint, as mentioned, we're going to utilize a simple Lambda part.
2. Once it receives the response, the client app makes a multipart/form-data POST request (iii), this time straight to S3. This i contains received pre-signed Postal service information, along with the file that is to be uploaded. Finally, S3 responds with the 204 OK response code if the upload was successful or with an appropriate mistake response code if something went incorrect.

Alright, at present that we've gotten that out of the way, you might still be thinking what pre-signed Mail service data is and what information it contains.

Information technology is basically a ready of fields and values, which, first of all, contains information about the actual file that's to be uploaded, such as the S3 key and destination saucepan. Although not required, it'south also possible to set additional fields that farther describe the file, for example, its content type or allowed file size.

It likewise contains data about the file upload asking itself, for example, security token, policy, and a signature (hence the name "pre-signed"). With these values, the S3 determines if the received file upload request is valid and, fifty-fifty more importantly, allowed. Otherwise, anybody could just upload any file to information technology as they liked. These values are generated for yous by the AWS SDK.

To cheque it out, let's have a wait at a sample result of thecreatePresignedPost method call, which is part of the Node.js AWS SDK and which we'll later use in the implementation section of this postal service. The pre-signed Mail information is contained in the "fields" key:

            {     "url": "https://s3.united states-east-ii.amazonaws.com/webiny-cloud-z1",     "fields": {         "key": "uploads/1jt1ya02x_sample.jpeg",         "bucket": "webiny-cloud-z1",         "X-Amz-Algorithm": "AWS4-HMAC-SHA256",         "X-Amz-Credential": "A..../u.s.a.-eastward-2/s3/aws4_request",         "X-Amz-Engagement": "20190309T203725Z",         "X-Amz-Security-Token": "FQoGZXIvYXdzEMb//////////...i9kOQF",         "Policy": "eyJleHBpcmF0a...UYifV19",         "X-Amz-Signature": "05ed426704d359c1c68b1....6caf2f3492e"     } }          

As developers, we don't really need to concern ourselves too much with the values of some of these fields (once we're sure the user is actually authorized to request this information). It's of import to note that all of the fields and values must be included when doing the actual upload, otherwise the S3 volition respond with an fault.

At present that we know the basics, we're ready to move onto the bodily implementation. Nosotros'll get-go with the client side, after which we'll prepare our S3 bucket and finally create our Lambda office.

Client

As nosotros've mentioned at the kickoff of this post, we're going to apply React on the customer side, then what we have here is a elementary React component that renders a button, which enables the user to select any type of file from his local arrangement. Once selected, we immediately commencement the file upload procedure.

Let's take a look:

	import React from "react" ;
	import Files from "react-butterfiles" ;
	/**
	* Recall pre-signed Mail information from a dedicated API endpoint.
	* @param selectedFile
	* @returns {Promise<any>}
	*/
	const getPresignedPostData = selectedFile => {
	render new Promise ( resolve => {
	const xhr = new XMLHttpRequest ( ) ;
	// Set the proper URL here.
	const url = "https://mysite.com/api/files" ;
	xhr . open up ( "Post" , url , true ) ;
	xhr . setRequestHeader ( "Content-Type" , "awarding/json" ) ;
	xhr . send (
	JSON . stringify ( {
	proper noun: selectedFile . name ,
	type: selectedFile . blazon
	} )
	) ;
	xhr . onload = office ( ) {
	resolve ( JSON . parse ( this . responseText ) ) ;
	} ;
	} ) ;
	} ;
	/**
	* Upload file to S3 with previously received pre-signed Postal service information.
	* @param presignedPostData
	* @param file
	* @returns {Hope<any>}
	*/
	const uploadFileToS3 = ( presignedPostData , file ) => {
	render new Promise ( ( resolve , reject ) => {
	const formData = new FormData ( ) ;
	Object . keys ( presignedPostData . fields ) . forEach ( key => {
	formData . append ( fundamental , presignedPostData . fields [ key ] ) ;
	} ) ;
	// Actual file has to be appended last.
	formData . append ( "file" , file ) ;
	const xhr = new XMLHttpRequest ( ) ;
	xhr . open ( "Postal service" , presignedPostData . url , true ) ;
	xhr . ship ( formData ) ;
	xhr . onload = function ( ) {
	this . status === 204 ? resolve ( ) : pass up ( this . responseText ) ;
	} ;
	} ) ;
	} ;
	/**
	* Component renders a unproblematic "Select file..." button which opens a file browser.
	* One time a valid file has been selected, the upload procedure will starting time.
	* @returns {*}
	* @constructor
	*/
	const FileUploadButton = ( ) => (
	< Files
	onSuccess = { async ( [ selectedFile ] ) => {
	// Pace 1 - become pre-signed POST information.
	const { data: presignedPostData } = await getPresignedPostData ( selectedFile ) ;
	// Footstep 2 - upload the file to S3.
	attempt {
	const { file } = selectedFile . src ;
	await uploadFileToS3 ( presignedPostData , file ) ;
	panel . log ( "File was successfully uploaded!" ) ;
	} catch ( e ) {
	panel . log ( "An fault occurred!" , e . message ) ;
	}
	} }
	>
	{ ( { browseFiles } ) => < push onClick = { browseFiles } >Select file...< / button > }
	< / Files >
	) ;

For an easier file selection and cleaner code, we've utilized a small packet chosen react-butterfiles. The author of the package is actually me, then if you have whatsoever questions or suggestions, feel free to allow me know! 😉

Other than that, there aren't whatsoever boosted dependencies in the lawmaking. We didn't even bother to use a 3rd party HTTP client (for case axios) since we were able to achieve everything with the congenital-in XMLHttpRequest API.

Notation that nosotros've used FormData for assembling the asking torso of the 2nd S3 asking. Besides appending all of the fields contained in the pre-signed Mail information, also brand sure that the actual file is appended as the last field. If you lot do that before, S3 volition return an mistake, so watch for that 1.

S3 bucket

Let's create an S3 bucket, which will store all of our files. In case you don't know how to create information technology, the simplest fashion to exercise this would exist via the S3 Management Console.

Once created, nosotros must suit the CORS configuration for the saucepan. By default, every bucket accepts only GET requests from some other domain, which means our file upload attempts (POST requests) would be declined:

            Access to XMLHttpRequest at 'https://s3.amazonaws.com/presigned-post-examination' from origin 'http://localhost:3001' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is nowadays on the requested resource.          

To fix that, simply open your bucket in the S3 Management Console and select the "Permissions" tab, where you lot should be able to meet the "CORS configuration" button.

Looking at the default policy in the in a higher place screenshot, we only demand to append the post-obit dominion:

            <AllowedMethod>POST</AllowedMethod>          

The complete policy would then be the following:

            <CORSConfiguration>     <CORSRule>         <AllowedOrigin>*</AllowedOrigin>         <AllowedMethod>GET</AllowedMethod>         <AllowedMethod>POST</AllowedMethod>         <MaxAgeSeconds>3000</MaxAgeSeconds>         <AllowedHeader>Authorization</AllowedHeader>     </CORSRule> </CORSConfiguration>          

Alright, let's move to the last slice of the puzzle and that's the Lambda part.

Lambda

Since it is a bit out of the scope of this post, I'll assume you already know how to deploy a Lambda function and expose it via the API gateway, using the Serverless framework. The serverless.yaml file I used for this trivial project can be plant hither.

To generate pre-signed Mail data, we volition utilize the AWS SDK, which is by default bachelor in every Lambda part. This is corking, only we must be aware that it can only execute actions that were allowed past the function that is currently assigned to the Lambda role. This is important because, in our case, if the role didn't have the permission for creating objects in our S3 bucket, upon uploading the file from the customer, S3 would respond with the Access Denied fault:

            <?xml version="1.0" encoding="UTF-viii"?> <Error><Code>AccessDenied</Lawmaking><Message>Access Denied</Message><RequestId>DA6A3371B16D0E39</RequestId><HostId>DMetGYguMQ+east+HXmNShxcG0/lMg8keg4kj/YqnGOi3Ax60=</HostId></Fault>          

So, before continuing, make sure your Lambda function has an acceptable role. For this, nosotros tin can create a new function, and attach the following policy to it:

            {     "Version": "2012-10-17",     "Statement": [         {             "Sid": "VisualEditor0",             "Effect": "Allow",             "Activity": "s3:PutObject",             "Resource": "arn:aws:s3:::presigned-postal service-data/*"         }     ] }          

A quick tip here: for security reasons, when creating roles and defining permissions, make certain to follow the principle of to the lowest degree privilege, or in other words, assign but permissions that are actually needed by the function. No more than, no less. In our case, we specifically allowed s3:PutObject action on the presigned-post-data bucket. Avert assigning default AmazonS3FullAccess at all costs.

Alright, if your role is gear up, allow'due south accept a wait at our Lambda function:

	const S3 = crave ( "aws-sdk/clients/s3" ) ;
	const uniqid = require ( "uniqid" ) ;
	const mime = require ( "mime" ) ;
	/**
	* Use AWS SDK to create pre-signed POST information.
	* Nosotros besides put a file size limit (100B - 10MB).
	* @param key
	* @param contentType
	* @returns {Promise<object>}
	*/
	const createPresignedPost = ( { cardinal, contentType } ) => {
	const s3 = new S3 ( ) ;
	const params = {
	Expires: 60 ,
	Bucket: "presigned-post-data" ,
	Conditions: [ [ "content-length-range" , 100 , 10000000 ] ] , // 100Byte - 10MB
	Fields: {
	"Content-Type": contentType ,
	key
	}
	} ;
	return new Promise ( async ( resolve , reject ) => {
	s3 . createPresignedPost ( params , ( err , data ) => {
	if ( err ) {
	pass up ( err ) ;
	return ;
	}
	resolve ( data ) ;
	} ) ;
	} ) ;
	} ;
	/**
	* Nosotros need to answer with adequate CORS headers.
	* @blazon {{"Access-Control-Let-Origin": string, "Access-Control-Allow-Credentials": boolean}}
	*/
	const headers = {
	"Access-Control-Let-Origin": "*" ,
	"Access-Control-Permit-Credentials": true
	} ;
	module . exports . getPresignedPostData = async ( { body } ) => {
	attempt {
	const { name } = JSON . parse ( body ) ;
	const presignedPostData = wait createPresignedPost ( {
	key: ` ${ uniqid ( ) } _ ${ proper noun } ` ,
	contentType: mime . getType ( name )
	} ) ;
	return {
	statusCode: 200 ,
	headers,
	trunk: JSON . stringify ( {
	error: fake ,
	data: presignedPostData ,
	bulletin: null
	} )
	} ;
	} catch ( due east ) {
	render {
	statusCode: 500 ,
	headers,
	trunk: JSON . stringify ( {
	fault: true ,
	data: null ,
	message: east . message
	} )
	} ;
	}
	} ;

Besides passing the basicprimal and Content-Type fields (line xviii), nosotros as well appended the content-length-range condition (line 17), which limits the file size to a value from 100B to 10MB. This is very important, because without the condition, users would basically exist able to upload a 1TB file if they decided to do information technology.

The provided values for the condition are in bytes. Also note that in that location are other file conditions you can utilize if needed.

I terminal note regarding the "naive" ContentType detection y'all might've noticed (line 49). Because the HTTP request that will trigger this Lambda part doesn't comprise the actual file, it's impossible to check if the detected content type is actually valid. Although this will suffice for this post, in a existent-world awarding y'all would do additional checks in one case the file has been uploaded. This can be washed either via an additional Lambda function that gets triggered once the file has been uploaded, or yous could design custom file URLs, which point to a Lambda function and not to the actual file. This manner, y'all tin can brand necessary inspections (ideally just once is enough) before sending the file back to the client.

Let'southward try it out!

If you've managed to execute all of the steps correctly, everything should be working fine. To try it out, permit's first try to upload files that don't comply with the file size status. Then, if the file is smaller than 100B, nosotros should receive the following mistake message:

            Post https://s3.us-due east-two.amazonaws.com/webiny-cloud-z1 400 (Bad Request)  Uncaught (in promise) <?xml version="one.0" encoding="UTF-8"?> <Error><Code>EntityTooSmall</Code><Bulletin>Your proposed upload is smaller than the minimum allowed size</Message><ProposedSize>19449</ProposedSize><MinSizeAllowed>100000</MinSizeAllowed><RequestId>AB7CE8CC00BAA851</RequestId><HostId>mua824oABTuCfxYr04fintcP2zN7Bsw1V+jgdc8Y5ZESYN9/QL8454lm4++C/gYqzS3iN/ZTGBE=</HostId></Error>          

On the other manus, if it'due south larger than 10MB, we should likewise receive the following:

            Postal service https://s3.us-east-2.amazonaws.com/webiny-cloud-z1 400 (Bad Asking)  Uncaught (in promise) <?xml version="1.0" encoding="UTF-8"?> <Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum immune size</Message><ProposedSize>10003917</ProposedSize><MaxSizeAllowed>10000000</MaxSizeAllowed><RequestId>50BB30B533520F40</RequestId><HostId>j7BSBJ8Egt6G4ifqUZXeOG4AmLYN1xWkM4/YGwzurL4ENIkyuU5Ql4FbIkDtsgzcXkRciVMhA64=</HostId></Error>          

Finally, if nosotros tried to upload a file that's in the immune range, we should receive the 204 No content HTTP response and we should be able to see the file in our S3 saucepan.

Other approaches to uploading files

This method of uploading files is certainly not the only or the "right" one. S3 actually offers a few means to accomplish the same thing. You choose the one that mostly aligns with your needs and environment.

For case, AWS Amplify client framework might be a good solution for yous, merely if you're not utilizing other AWS services like Cognito or AppSync, you don't really demand to utilize it. The method we've shown here, on the client side, consists of two simple HTTP POST requests for which we certainly didn't demand to utilise the whole framework, nor whatsoever other bundle for that matter. Always strive to make your customer app build as light as possible.

You might've also heard about the pre-signed URL arroyo. If you were wondering what is the difference between the two, on a high level, it is like to the pre-signed Mail service information arroyo, but it is less customizable:

Note: Not all operation parameters are supported when using pre-signed URLs. Sure parameters, such as SSECustomerKey, ACL, Expires, ContentLength, or Tagging must be provided as headers when sending a asking. If you are using pre-signed URLs to upload from a browser and demand to use these fields, see createPresignedPost().

One notable characteristic that it lacks is specifying the minimum and maximum file size, which in this post we've achieved with the content-length-rangecondition. Since this is a must-have if you ask me, the approach we've covered in this post would definitely exist my go-to option.

Additional steps

Although the solution we've built does the job pretty well, there is always room for improvement. Once you hit production, you will certainly want to add the CloudFront CDN layer, so that your files are distributed faster all over the world.

If you'll be working with epitome or video files, you will too want to optimize them, because it tin salvage you a lot of bytes (and money of course), thus making your app piece of work much faster.

Conclusion

Serverless is a really hot topic these days and it's not surprising since then much work is abstracted abroad from us, making our lives easier as software developers. When comparing to "traditional serverful" architectures, both S3 and Lambda that we've used in this postal service basically require no or very little system maintenance and monitoring. This gives us more time to focus on what really matters, and ultimately that is the bodily production we're creating.

Thanks for sticking until the very cease of this article. Experience complimentary to let me know if you have any questions or corrections, I would exist glad to bank check them out!

---

Cheers for reading! My name is Adrian and I work as a total stack developer at Webiny. In my spare time, I similar to write about my experiences with some of the modern frontend and backend web development tools, hoping it might assistance other developers. If you lot accept any questions, comments or just wanna say hi, feel costless to attain out to me via Twitter.

farrfreand.blogspot.com

Source: https://www.webiny.com/blog/upload-files-to-aws-s3-using-pre-signed-post-data-and-a-lambda-function-7a9fb06d56c1/

Share this post