Success Story
Automating infrastructure provisioning for Drewry; leading to a significantly reduced cloud cost
Services Used:
Drewry is a leading provider of research and consulting services to the maritime & shipping industry. With over 3,000 clients across 100+ countries, it’s trusted for impartial market insights, industry analysis, and advice. Drewry serves its clients through its diverse business units which include maritime research, maritime advisory, supply chain advisory, and maritime financial research.
- 2X faster decision making
- 30% reduction in data redundancy
- 20x faster freight costing calculations
Business Situation
Enterprises in the maritime sector possess large volumes of data on freight rates but lack the necessary insights to sense market changes and drive decisions in a timely manner. To help them ensure competitive freight rates, Drewry developed a Freight Intelligence Portal (FIP) that could accumulate freight rate data to draw comparisons.
How Daffodil helped Drewry to modernize the Freight Intelligence Portal. Read the full case study here.
Drewry’s portal uses ETL pipelines for data cleansing, data manipulation, and benchmarking. Amongst the various ETL processes, benchmarking requires high memory and CPU resources.
The core of the application was executed in the EKS-based Kubernetes cluster. Manual provisioning of EKS node groups of large instance sizes was done before the ETL pipelines run. This approach of infrastructure provisioning resulted in under-utilized or idle cloud resources, which ultimately leads to cloud waste. Since it was a static load, a solution was required to create an on-demand infrastructure for ETL pipelines.
The Solution
The DevOps engineers at Daffodil analyzed the shortcomings of the existing resource provisioning system and ensured that infrastructure is created on-demand whenever there is a need to run processes with high resource utilization. To automate infrastructure provisioning, AWS CloudFormation was utilized.
AWS CloudFormation is an Infrastructure as Code (IaC) service that allows automating and managing resources across all AWS accounts & regions through a single operation.
Here are a few highlights of CloudFormation helped Drewry in automating infrastructure provisioning:
- CloudFormation templates were created to launch various sizes of node groups according to the requirement.
- CloudFormation templates were stored in S3 for version control and to download at the time of creating the CloudFormation stack.
- AWS SDK for Python- Boto3 was utilized to integrate the Airflow DAGS. Two types of DAGs were created with the purpose of creating and deleting the CloudFormation stack having EKS Node Groups on which the ETL pipeline runs.
- The users at the maritime firm were able to create and delete infrastructure with a clickable button which they run before starting the high resource utilization processes and after completing it.
The Impact
With the new system in place, the infrastructure can be created on demand resulting in reduced cloud costs. The Node groups running in multiple zones improve high availability for the Airflow workloads. Automation with CloudFormation has resulted in increased deployment speed and reduced manual efforts. Infrastructure scaling has become simpler than before. With infrastructure configurations, it has now become easier to scale up or scale down the instance size.
Read Related Case Studies
Get in Touch
Sign up for a 30 min no-obligation strategic session with us
Let us understand your business objectives, set up initial milestones, and plan your software project.
At the end of this 30 min session, walk out with:
- Validation of your project idea/ scope of your project
- Actionable insights on which technology would suit your requirements
- Industry specific best practices that can be applied to your project
- Implementation and engagement plan of action
- Ballpark estimate and time-frame for development
Awards & Accolades
-
Discover Daffodill
-
Industries
-
What We Do
-
Domain Expertisee
A CMMI level 3 company