Kafka Summit 2021 Keynote Emphasizes Principles for Data Mesh Success
Big Compass recently attended the Kafka Summit 2021 and is encouraged by what we heard. This article captures a few of the ideas and principles of Jay Kerp's keynote and describes how those align with Big Compass's focus on data agility.
Note: This article isn't a comprehensive review of the entire event – if you are interested in that level of detail, please check out this overview from Tim Berglund. Tim did an excellent job emceeing the Summit and highlighting the lighthearted, fun culture of Confluent and the Confluent community
The key idea from Jay's keynote that resonated with us was "Companies are becoming software." Traditionally we think of companies utilizing software to manage a domain (customer) or operation (production). But as companies and their customers become more sophisticated, they have evolved into organisms that thrive when their software infrastructure and data are optimized to deliver value outside the software system's confines. The CRM system isn't just software; it's the source of value, especially when data is designed to flow to and from other software systems. The "company" is the collective value generated and delivered to customers by their connected "software" sources.
The "companies are software" idea was the introduction to the Data Mess to Data Mesh keynote. The mess was illustrated with the image representing the spaghetti integration lines between all the software systems being directly connected to all the other software systems. Customer expectations (and the pandemic) have driven the focus on delivering value and how to build a "company in software." The emphasis is on answering how all the applications come together, how they communicate, and how they will grow and scale to meet future needs.
The Big Compass vision of a successful, modern enterprise is one that has broken the data silos and allowed free (with governance) flow of information between software applications and thus allows them to run as one cohesive unit i.e., a giant mesh of software applications that work together efficiently.
-Kapil Kumar, Co-Founder & Co-CEO of Big Compass
Events (a notification with data attached) and Event Streams are the answer, and two main movements support this answer and move the solution towards decentralization: microservices and data mesh. The use of microservices is common among modern enterprise IT. The idea of a data mesh is gaining momentum and helps fulfill the requirement to decentralize the data architecture necessary to realize the
While the idea of data mesh seems logical, the reality is event (data) streams will be necessary to stand the mesh up. As Jay said in this presentation, "this creates kind of central nervous system that allows the different parts of a company to connect and come together."
Using the internet meme of drawing an owl, the presentation moved into how you actually get the implementation of data mesh accomplished. Four principles were identified and divided into two categories; How Data is
Produced and How Data is Consumed.
How Data is Produced Principles
- Domain Ownership – People with domain ownership are responsible for
producing the data for that domain.
- Data as a Product – The domain owners need to consider data consumers as
their customers and treat data production as a product.
Kafka works well in the production area, creating event streams and connecting the decentralized items.
How Data is Consumed Principles
- Data is available via self-service – Access to the entire environment should be open, and the cloud is the implementation to achieve this.
- Governance is required – With guidelines, discoverability, and the ability to quickly access data, the mesh delivers on the "companies are software" idea.Confluent announced a new feature, Stream Governance, (URL) which will help implement best practices for decentralized governance to facilitate this principle.
Concluding the presentation, Jay stressed this point: "I think these four principles of data meshes are actually a few key things to get right as you think about how you democratize data and how you start to decentralize it across the organization."
We're excited to see how Data Mesh gains momentum and helps organizations decentralize data to create value and serve their customers. It might seem odd for Big Compass, an integration consultancy, to be excited about data. But it's only the natural evolution of our expertise. In the future, you will see us continue to focus on data, specifically data agility – which we define as the ability to move and combine data for insights and operations.