In the past, TES hadn’t maximised the power of their data, so decided to invest in building an ETL/Big Data platform to better understand their audience, as well as keep track of business KPIs using dashboards.
TES undertook a full digital transformation with the help of the One Beyond team.
One Beyond created an ETL platform (extract, transform & load) with the aim of collecting data from multiple sources. They normalised and transformed the data to store it in a data warehouse and other databases, in order to make it available to applications, APIs and data scientists. To achieve this, the team created more than twenty microservices, a Hadoop-based transformation layer and an integration layer to move data around.
This was just the tip of the iceberg as we also used the Hadoop ecosystem to aggregate and process data, way richer than the original subsets. This drew a line between what is data, analytics and insight. When you have insight, you have power, and only then can you make thoughtful decisions. If you don’t have insights extracted from your data, all you have is an opinion.
All data and processes are monitored using Datadog dashboards, and alerts are flagged when something abnormal happened. More than 20GBs of data are ingested on a daily basis, just from user interactions.
- Implemented our microservices in NodeJS, hosted in AWS using a CI/CD pipeline in Jenkins
- For the data warehouse the team used Amazon Redshift. AWS was also first choice for the Hadoop ecosystem: EMR gave access to starting Hive/Spark clusters for batch processing overnight
- Some of the databases used to digest and store data are Postgres, Redis, MySql and Mongo
- Finally, for the integration layer they used RabbitMQ and Amazon Kinesis