Creating a Data Mesh – First Steps and Delivery Roadmap
Read all blog posts in the Data Mesh vs Azure series via this link.
Background
A common question I get asked a lot when creating a data mesh architecture is where to start? The consultant in me defaults the answer to ‘it depends’, of course 🙂
However, in this blog post I want to give a better answer based on my experience of working with various customers so far. As always, the usual caveats apply, I’m happy to go first when trying to define a starting point for our data mesh delivery and fully accept that parts of this are probably wrong. This is also founded in the knowledge that every customer I’ve worked with is different, with different priorities and very subjective views on why they even need a data mesh architecture. Not to mention various levels of data platform maturity.
Another important thing to call out, is that to deliver a data mesh must to think about more than just the technology. I say this as a red-blooded technologist myself and one that likes nothing more than to grab a whiteboard and start solutioning with design patterns etc. For data mesh, this is not the starting point and will not help. We need to front load a whole bunch of other non-technical stuff first before we can even think about writing some code.
Data Mesh Phase 1 Roadmap
Given this context. Have a look at my first attempt to create a data mesh delivery roadmap, below. One that I think could become common in supporting customers with a starting point. Or offer a simple check list of tasks already completed.
I used the blue dotted lines to inform relationships between tasks across the swim lanes as well as support a bit of a dependency chain when moving from top left to bottom right.
A PDF version of the same is also available for download here.
As you will see, to break apart this thinking I’ve given the roadmap three key swim lanes:
- Business Definition
- People & Process
- Technical Delivery
In a backlog we could easily take these headings as epics, with (what I have called tasks in the visual) as our features. There is so much more that could then be added in terms of work items and tasks, but it was already getting difficult to read so I decided to pause and publish given the demand for an answer to the ‘mesh starting point’ question.
To be clear, this takes us from nothing to the build of our first data product. So, to reiterate, a starting point. This isn’t a finished data mesh roadmap, maybe we call it phase 1 of moving towards a minimum viable data mesh. After all a single data product is not a mesh.
In phase 2, we might naturally onboard more data products and start creating our mesh supervision plane.
Beyond Phase 1 – Team Profile
Another perspective I wanted to visualise when delivering a data mesh is the potential team profile. Also, a common ask when starting out with a data mesh delivery. Everyone still seems hungry for data engineers, which is fine, but let’s consider what a data mesh could give us in terms of a reduction in that specialised engineering skillset. Especially if we move beyond the phase 1 outline I’ve created above. Note: the colour palette here is does not relate to the first visual.
Once we have a our minimum viable mesh and we can start to scale the architecture, aligned to the de-centralised principals of the mesh. To that end, we want all the technical engineering effort to empower the less technical business users. With a consistent splash of architecture technical leadership along the way (AKA me!).
For those moving beyond the theory of a data mesh architecture and now putting the principals into practice I hope you found my above thinking usual. Please let me know your thoughts either way.
Many thanks for reading.
Source link