The software and services are going to be getting advanced enough to just eliminate the need for a dedicated team to build ETL. People with relevant domain knowledge will have an easier time to deliver their work product, without the overhead of building phase.
To get a reasonably good data platform - point-and-click ETL service, SAAS offering and the likes of Metabase - are already good enough for medium enterprises... and beat Databricks offerings for speed(setup, delivery and operation) in reporting and operational data access.
I am absolutely sure that there will be a massive contraction in the DS, DE and ML opportunity market in the next few years. The major companies will consolidate and jobs in those domains will only be available at only a handful of companies... or extremely specialized startups.(much like chip design is now consolidated)
Long story short, for companies - you probably don't need DS, ML and DE departments.
My experience is that many established companies are still struggling to get adequate operational reporting. Data engineers are still helpful to move the data necessary to make that happen. DS and ML become useful later once there's a more mature data culture and infrastructure. Otherwise you have analysts spending most of their time doing data engineering so they have something to analyze.
I think we are aligned I am just much more sceptical towards no code solutions. So I think these roles will shrink massively and the little code you have to still do will then become just another thing to integrate for your swe role (this is what is happening in my industry at least)
The software and services are going to be getting advanced enough to just eliminate the need for a dedicated team to build ETL. People with relevant domain knowledge will have an easier time to deliver their work product, without the overhead of building phase.
To get a reasonably good data platform - point-and-click ETL service, SAAS offering and the likes of Metabase - are already good enough for medium enterprises... and beat Databricks offerings for speed(setup, delivery and operation) in reporting and operational data access.
I am absolutely sure that there will be a massive contraction in the DS, DE and ML opportunity market in the next few years. The major companies will consolidate and jobs in those domains will only be available at only a handful of companies... or extremely specialized startups.(much like chip design is now consolidated)
Long story short, for companies - you probably don't need DS, ML and DE departments.