EKS and Redshift news; GCP & Oracle; Azure data migration tools

AWS
Amazon EKS support in Amazon SageMaker HyperPod to scale foundation model development

The update introduces Amazon EKS and SageMaker HyperPod, which significantly enhance the development of foundation models at scale. HyperPod is designed to streamline distributed training and inference for large-scale machine learning models, integrating seamlessly with Kubernetes on Amazon EKS.

It reduces overhead and accelerates the deployment of high-performance, resource-intensive models.

Amazon Redshift now supports altering sort keys on tables in zero-ETL integration

The update highlights two key features in Amazon Redshift: the ability to alter sort keys on tables and a zero-ETL integration with Amazon Aurora and Amazon RDS.

These enhancements allow for more flexible performance optimization and simplify the data integration process, reducing the need for complex ETL workflows.

GCP
Amit Zavery is the VP of Google Cloud Platform. I found this interview of him talking about the partnership between Google Cloud and Oracle.

I think the time is approaching I add a section for OCI in this newsletter.

AZURE
Microsoft is getting rid of some database migration tools by the end of the year.

This is directly from the Microsoft Blog. The blog post discusses Microsoft's consolidation of Azure Database Migration tools, streamlining database migration processes for users.

It highlights the retirement of older migration tools and the shift toward more unified, modern solutions to improve efficiency and ease for migrating to Azure.

The update emphasizes Microsoft's focus on simplifying the transition to cloud-native databases, offering support for various platforms.

DATABRICKS
More on the “Data Table Format Wars“ in which Databricks is one the main combatants.

The article from HackerNoon explains Apache Polaris, an open-source catalog for managing Apache Iceberg tables. Polaris facilitates multi-engine compatibility, centralized metadata management, and secure access to Iceberg tables via a REST API.

It supports multiple cloud storage configurations and enforces role-based access control (RBAC). Polaris enhances interoperability by enabling various query engines like Spark and Flink to work on the same dataset, minimizing data duplication and simplifying storage management.

SNOWFLAKE
This one is close to home.

You can find out how the Snowflake Toronto office is expanding and the types of positions it is looking for.

They are hiring for about a dozen positions. Snowflake has a lot off cool technology, but there seems to be at the moment, a lot of emphasis on the stock in the news lately.

This news here is encouraging.

GRAPH DATABASES
A real example of rolling your own…

The article discusses how Glean raised $260 million by leveraging GraphRAG, a combination of knowledge graphs and retrieval-augmented generation (RAG).

This tool enhances AI models by improving accuracy and context-awareness, addressing the challenges of managing large enterprise data sets.

The integration of knowledge graphs with large language models (LLMs) provides higher accuracy, security, and explainability in AI outputs. It also touches on how organizations can implement this technology efficiently despite budget constraints.

VECTOR DATABASES
PostgreSQL is now getting into the realm of vector databases.

The post mentions how to create a local retrieval-augmented generation (RAG) application using Postgres with the pgvector extension, Ollama for model hosting, and the Llama 3 large language model.

DEEP DIVE
I just want to make a quick note on how all of this stuff I talk about works together. As I always infer in my deep dive, keep on learning, explore things that are outside of your comfort zone, and talk to people about technology.

It makes you a well rounded person where you are well-versed on what is out there in the tech world. Your eyes will not glaze over in a meeting when something is brought up that you may not necessarily work on, and it helps you think.

I had a discussion about some newer technologies yesterday with some folks, that we will eventually have to be well versed in. Thankfully many of the certifications I have taken in the last 3 years helped contribute to the discussion, and I was able to ask intelligent questions. due to the nature of my work, I can’t get into specifics but you get the idea.

Just remember, instead of doom scrolling on the couch after dinner (I am guilty of that too), click on some of the links in the newsletter, and learn about something new, or outside of the scope of your daily tasks. I read the articles I write about.

Have a great weekend, and I will see you on Monday.

p.s. I am taking some immediate steps to vastly improve the newsletter. More to come in the next few weeks.

Gladstone