We modernized TekCor4's Data Infrastructure with Scalable AWS Services, Streamlining File Processing and Seamless Migration from 20-Year-Old System

CHALLENGE

• High-cost architecture with a substantial investment required in infrastructure, technology, or development resources
• Difficulty in onboarding new customers/modify the file profile of customers
• No user control at key stages of file processing & reports are more prone to error

They wanted to build a system that can effectively handle a massive influx of data from over 5,000 customers. This involves processing 200,000 files of various sizes every month, storing over 20 million documents in the database, and retaining data for 3 years.

Why were we brought in?

Ganit has made a significant dent in various industries using data science and analysis. Ganit partners with clients to translate their data into a tangible, insightful plan of action that delivers on a measurable impact to the clients’ topline & bottom-line growth.

Solution

Ganit modernized their entire infrastructure to micro-services driven application with a data platform on AWS that can process large influx of data at scale and facilitate consumption for analytics and visualization.

An event driven architecture with the following components was built:

• AWS Transfer Family was used to pick up files uploaded by users and transfer it to Raw S3 zone on Amazon S3
• ECS cluster was spun up upon receiving a file in the Raw S3 Zone that reads the file type and data present in it to perform necessary transformation using Amazon EMR with automated scaling configured. The transformed data is stored in Curated Zone in Amazon S3
• Data from the curated zone was then loaded in Oracle hosted on AWS
• Application UI facilitates seamless onboarding of new customers and file profiles, providing access to features and processed files, while also enabling users to view processed data effortlessly

Features of the tool

• Reliable system to monitor and report any system issues or failures to maintain accurate inventory records through timely analysis and reporting of problems
• Autoscaling application to meet the spike in demand ensuring smooth operation
• Entirely templatized IaaC architecture