The data team is the most interdisciplinary team in any organization.
Data Scientists, Analysts, Engineers, Business Users - diverse people, with diverse tools, skillsets and DNA.
All doing diverse things. Sometimes they're asking open ended questions to get to the bottom of “why”, just like a scientist in a research lab. Sometimes they're working on scaling petabyte-scale data processing systems, like a software engineer.
Add to all this the living and breathing thing that is... data. It's constantly changing (unlike code or design).
There's no easy answer.
We started as a data team ourselves, on a quest to make ourselves as agile as we could. We borrowed the principles of Agile from product teams, DevOps from engineering teams, and Lean Manufacturing from supply chain teams. We then experimented for two years, across 200 data projects, to create our own idea of what makes data teams successful. These principles are the foundation of everything we build at Atlan.
Data teams will always have a variety of roles, each with their own skills, favorite tools and DNA. Embrace the diversity, and create mechanisms for effective collaboration.
All data assets, from data to dashboards, are assets, and they should be treated like assets.
Assets should be easily discoverable.
Assets should be maintained.
Assets should be easily reusable.
In today’s world, as business needs evolve rapidly, data teams need to be a step ahead, not deluged with three months of backlog.
Constantly measure your team’s velocity, and invest in foundational initiatives to improve cycle times.
Reduce dependencies between business, analysts and engineers.
Enable a documentation-first culture.
Automate whatever is repetitive.
With the inherent diversity of data teams, it's all too easy to misunderstand other team members' roles. But that creates trust deficiencies — especially when things go wrong! Intentionally create systems of trust in your team.
Make everyone’s work accessible and discoverable to break down ‘tool’ silos.
Create transparency in data pipelines and lineage so everyone can see and troubleshoot issues.
Set up monitoring and alerting systems to proactively know when things break.
The data ecosystem will rapidly evolve. The tools, technology and infrastructure you use today will (and should) be different from the tools you use two years later.
Your data stack should allow your team to experiment and innovate as technology evolves, without creating lock-ins.
Embrace tools that are open and extensible.
Leverage a strong metadata layer to tie diverse tooling together.
Teams at Airbnb famously said, "Designing the interface and user experience of a data tool should not be an afterthought." Without good user experience, the best tools or most thoughtful processes won't be adopted in your team.
Invest in user experience, even for internal tools. It will define adoption velocity!
Invest in simple and intuitive tools.
Software shouldn't need training programs.