Data Career — Needs, Skillset, Growth, Responsibilities & Roadmap
Data Engineer/Analyst/Scientist/ML
The Data Hierarchy of Needs:
Serves as a guide that captures the complete picture of an organization’s tooling at different levels in their data product platform — from logging and user-generated content at the bottom to AI and deep learning at the very top.This helps understand data processing steps and find the best solution to a given problem. While building and supporting a data product, before going to advanced data modeling (top of the pyramid), data teams need to fill huge holes they frequently have in the pyramid’s base, lacking reliable, complete data flow.
Each hierarchy level represents a different way that delivers value — not better or worse, just different. The hierarchy reflects trying out simple solutions before jumping to more advanced ones, thereby enabling data teams to make an impact faster and then iterate with more complex solutions if necessary. The pinnacle of a data platform is not machine learning or AI, but it is just the impact data teams are trying to achieve, no matter the technology they use.
These stages of growth are based upon a hierarchy of data-driven needs. Essentially, we can’t get to the next stage of organizational transformation until we have sufficiently satisfied our lower (more primal) needs. If that foundation is weak, the higher stages may crumble and leak even if they initially work.
Skillsets & Interaction: There are only a handful of “unicorn” data engineers and scientists engineers on the planet, who have superpowers in maths/stats, a variety of programming languages, AI/machine learning, an even wider variety of tools and techniques, and of course are great in understanding business problems and articulating complex models in business-speak.
One can only strive to become a unicorn — with experience and exposure in each & every data practice mentioned. Here is a selection of core skills & interactions that are needed in your data team depending on a specific role. Ultimately, some of these may overlap, and you may not need one of each. It depends on what your team wants to achieve.
Career Growth & Responsibilities: Where am I? Many engineers from data domain find themselves become stalled in their data journey at some point in time. Sometimes its because they haven’t built the necessary foundation to transition to the next level. Other times, it’s because they don’t know where to go next. The below hierarchy would help them to reevaluate their data journey. Throughout your data career journey, while it’s easier to manage your career development, growth is a little trickier because it can be affected by your environment and available opportunities.
At the same time, this gives you the chance to define your career growth on your own terms. There’s nothing wrong with working to become a CEO, but it’s important to remember that growth isn’t always vertical. Consider your career values and how your work aligns with them, and you’ll be able to make a career growth plan that fulfills you personally and professionally.
Below are the diagrams that gives you a directional idea on the data career you choose and might also be useful to start prepping for your next role early.
Career Roadmap: Design and develop your data career journey by pinpointing milestones with required technical skills.
You don’t want to jump straight into the deep end when it comes to a data career. Start small by outlining exactly what your goals are with respect to data. Knowing what you plan to do with the data you are going to collect can help you to, keep only the ideas that are relevant to your goal, ensuring both your brain and data management software don’t get overcrowded and unorganized. Below are the high-level responsibilities for any data role to keep in mind all the time.
Problem Formulation — Analyzes the business problem within one’s discipline and questions assumptions to help the business identify the root cause.
Data Strategy — Understands, articulates, interprets, and applies the principles to business problems.
Data Source Identification — Defines and identifies the most suitable sources for required data that is fit for purpose.
Data Transformation & Integration Requirements — Builds the infrastructure required for optimal transformation and integration from a wide variety of data sources using appropriate data integration technologies.
Code Development and Testing — Reviews the solution and application design to ensure it meets business, technical, and data requirements.
Data Cataloging, Governance & Observability— Data cataloging is the process of making an organized inventory of your data. Data governance is the process of managing the availability, usability, integrity and security of the data in enterprise systems.Data observability is the blanket term for understanding the health and the state of data in your system.
Hope this post helps you identify and work on the skills you need to succeed in the data career. All the best and stay tuned to my future posts regarding the tech stack, and tutorials in the above-mentioned roles.
Follow me:
https://shivaga9esh.medium.com/
https://www.linkedin.com/in/shivaga9esh/
Let’s connect on discord for group learning, daily questions & clarifications: https://discord.gg/wePCJpbrVy
Working hard to keep it simple and keeping it real :)
Related posts:
