Data Mesh, o what art thou?🤔🎭

I'm still trying to figure out what it is, and if it has any practical use

Author

Gladstone Benjamin
January 13, 2025

What’s in today’s newsletter

  • AWS simplifies AI and data processes with SageMaker 🌟

  • Amey Banarse (Yugabyte) explains cloud technology's transformative potential 🌥️

  • AWS's new instances are 50% Graviton-powered 🚀

  • Data engineering trends: automation, governance, innovation ahead! 🚀

  • Choosing between XTable and Delta Lake impacts efficiency 📊

Check out the and up to the minute “New Database Job Opportunities” sections (lots of current jobs this week), as well as the weekly Deep Dive - This week we take a look at Data Mesh. Is it just another fad or is it useful? You can read what I think.

AWS

TL;DR: AWS enhances SageMaker with new features that automate model training and simplify data processes, enabling broader AI adoption and fostering innovation across various sectors with reduced barriers.

  • Amazon Web Services (AWS) enhances cloud computing with SageMaker, simplifying AI and data processes for users.  

  • New SageMaker features automate model training and improve data preparation workflows, boosting efficiency and development speed.  

  • Built-in algorithms and a low-code interface make AI model creation accessible to users with varying technical skills.  

  • The evolution of SageMaker lowers entry barriers, empowering businesses to adopt AI solutions and innovate effectively.  

Why this matters: By simplifying AI development, AWS's upgraded SageMaker empowers more businesses to embrace AI technology, democratizing access and fostering innovation. This transformation could significantly enhance productivity and decision-making across industries, marking a critical shift in who can leverage AI's potential for competitive advantage. 

TL;DR: Amey Banarse's keynote at AWS re:Invent 2024 focused on cloud technology advancements, democratizing AI access, and enhancements in AWS infrastructure to improve businesses' agility and decision-making capabilities.

  • Amey Banarse's keynote at AWS re:Invent 2024 showcased advancements in cloud technology's transformative potential for industries.  

  • During his presentation, he emphasized the importance of democratizing AI access for improved decision-making across organizations.  

  • Banarse highlighted AWS’s infrastructure enhancements, including reduced latency and increased scalability for diverse customer needs.  

  • His insights suggest that cloud solutions can significantly enhance businesses' operational agility and customer engagement capabilities.   

Why this matters: Amplifying AI access through AWS’s cloud infrastructure democratization signals a pivotal shift toward more equitable tech landscapes. It forms a bedrock for innovation across industries, bolstering business agility and engagement. The enhancements in AWS infrastructure ensure enterprises are equipped to tackle the accelerating demands of a digital-first economy. 

TL;DR: AWS reports that 50% of new instances now run on Graviton processors, enhancing efficiency and performance, setting new standards in cloud computing, and influencing industry innovation.

  • AWS has achieved a milestone with 50% of new instances powered by its Graviton custom silicon processors.  

  • The Graviton series, launched in 2018, enhances efficiency and cost-effectiveness in cloud computing services.  

  • Graviton3 instances show significant performance improvements over previous versions, solidifying AWS's leadership in cloud infrastructure.  

  • The rise of custom silicon could reshape cloud computing, pushing traditional chip manufacturers to innovate.  

Why this matters: The milestone of 50% of new AWS instances using Graviton processors highlights the tech shift toward custom silicon, redefining efficiency in cloud computing. This reshapes industry dynamics by challenging traditional chipmakers and influencing customer preferences, ultimately driving a wave of innovation across global data centers.  

DATA ENGINEERING

TL;DR: The article forecasts key data engineering trends for 2025, focusing on automation via AI, enhanced analytics, and stronger data governance to improve organizational efficiency and decision-making.

  • Organizations are increasingly leveraging data engineering to enhance decision-making and operational efficiency in the digital age.  

  • The automation of data engineering processes using machine learning and AI is expected to improve productivity significantly.  

  • Enhanced focus on data governance and security is critical due to the handling of large volumes of sensitive information.  

  • Embracing automation and advanced analytics will enable businesses to unlock efficiencies and drive greater innovation. 

  • Why this matters:  As we enter into 2025, these trends indicate that organizations stand to gain competitive advantages by streamlining operations and safeguarding information, fueling more informed decisions and fostering innovation. The integration of automation and analytics in data engineering signals a paradigm shift toward data-driven, efficient, and secure business practices. 

 

DATA STORAGE & ENGINES

TL;DR: The article examines Apache XTable and Delta Lake Uniform interoperability, highlighting their distinct features and use cases to guide organizations in optimizing data lakehouse strategies for enhanced efficiency and analytics.

  • Researchers analyze interoperability of Apache XTable and Delta Lake Uniform for effective data lakehouse architecture.  

  • Apache XTable focuses on performance and simplicity, making it suitable for lightweight data storage applications.  

  • Delta Lake Uniform offers richer features like ACID transactions, catering to enterprise needs for data integrity.  

  • Choosing between XTable and Delta Lake can enhance operational efficiency and improve decision-making in data strategy.

  • Why this matters: As organizations increasingly adopt lakehouse architectures, choosing between Apache XTable and Delta Lake Uniform impacts how effectively they can manage, analyze, and utilize their data. Making strategic decisions on data storage and management tools influences operational efficiency, data integrity, and ultimately, business success through insightful and timely data-driven decisions. 

 

NEW DATABASE JOB OPPORTUNITIES

DEEP DIVE

I still don’t understand the need for Data Mesh

From my perspective, Data Mesh has not really been that earth shattering. I remember two years ago, when I started my current position, there was a buzz about Data Mesh. I even bought the book about it. I still remember that it cost me $103.39 Canadian dollars at the time. It now costs $99.99 in my neck of the woods. That’s neither here nor there.

I understand the novel idea of domains for data and data as a product. My thoughts are that to implement such a paradigm, one would have to move proverbial mountains to get Data Mesh implemented fully in an organization. You would have to have a very strong and influential individual or team to get Data Mesh put into place fully.

Much to the consternation of many people in my life, I deal with facts and empirical evidence in almost everything I do. But in this case, my gut tells me that it is a novel idea. But my practical and realistic side of my brain tells me that the future of data processing is the Data Lake, and all of the ancillary technologies that run on top of Data Lakes (Snowflake, Trino, Dremio, AWS Athena, Apache Hive, Apache Spark, Databricks Delta Lake). You get the idea, right?

You might be a snarky smarty pants know-it-all and say “Gladstone, Data Lakes and Data Mesh are two different paradigms”. I know that. As a matter of fact, you can read my report here. What I am stressing here is that from what I have seen, and this is just my informed opinion, you are going to face a struggle if you are dealing with a mid sized to large organization if you want to implement Data Mesh to its fullest extent.

Just talking to some of my esteemed colleagues, Data Mesh to them is not more than just interesting idea that is hard to implement. In fact, one of my team members that I highly respect, even left his previous position at a very prestigious firm because of the complete and utter fiasco in trying to implement Data Mesh (I might be adding a bit of hyperbole for effect, but it was not far off the mark).

As much as we may research and work with the latest and greatest technologies, you always have to remember that we are dealing with human beings with their foibles, corporate fiefdoms, office politics, and executives who hear a buzzword on some Gartner webinar, and run with it.

As always, I like to dig a little more into stuff I discuss here in this newsletter, so for those who want to get the facts about Data Mesh, and not the feelings of some battle hardened IT guy who has seen everything and then some, read about Data Mesh on my blog, Big Data Clouds.

Gladstone