Challenge
Large files become difficult for the client’s application to import straight into the database. The current server-based API resulted in:
- Overload on the server when processing files
- Slow request processing that results in timeouts and delays
- A bad user experience brought on by system lag
Solutions:
We used AWS Lambda and Amazon SQS to process the file import procedure to a server less architecture:
- File Upload: The frontend uploads them to AWS Lambda rather than uploading files to the server.
- Processing & Chunking: Lambda divides files into smaller pieces To make processing easier .
- Queue Management: To avoid system overload, chunks are routed to Amazon SQS.
- Database Entry: In parallel, another Lambda function pulls data from SQS and enters it into Amazon RDS.
Outcomes:
- File processing is 80% faster without server saturation.
- Scalable solution that can handle any big file.
- Saving money by using Lambda on-demand
- Enhanced usability with seamless, non-blocking imports
Important AWS services utilized
- Lambda on AWS
- SQS on Amazon
- RDS on Amazon