Consumerizing Data at Scale with Data Mesh

Data Architecture: Why is it crucial to know?

In the age of self-service business intelligence, becoming a data-driven organization remains the chief strategic goal of every company. However, few companies tend to their data architecture with the level of democratization and scalability it deserves.

Both the analytics and technology industries are now in a state of transition. In fact, we rarely use the phrase “Big Data” anymore; instead, we talk about “digital transformation” or “data-driven organizations”. The industry, largely, has realized that data is not the new oil, because, unlike oil, the same data can be repurposed for several initiatives.

Much in the same way that software engineering teams transitioned from monolithic applications to microservice architectures, data mesh is, in many ways, emerging as the data platform version of microservices.

This new kind of data architecture will empower faster innovation cycles and lower costs of operations, with evidence from early adopters of this approach validating potential large-scale benefits. [1] [2] [3]

Traditional Data Lake Architecture: An E-commerce Case Study

A typical data lake architecture for an e-commerce business mainly constitutes the following:

  • Customer domain
  • Order domain
  • Invoice domain
  • Inventory domain

For each domain, a team of data engineers inputs all the data, via ETL tools or streaming solutions, on a central platform (Data Lake). Although each team may possess expertise about their specific domain, a knowledge gap among different teams and their data sets may persist.