Redshift Boost; QuickSight Direct Query; Medallion Data Lake; Reltio & Databricks Pipeline; Vector Indexes; SQL Server's Decline

AWS
Amazon Redshift Serverless has increased its maximum capacity to 1,024 Redshift Processing Units (RPUs), allowing customers to handle more demanding workloads without managing the underlying infrastructure.

This enhancement enables greater scalability and performance for data analytics tasks.

GCP
Amazon QuickSight now supports direct query functionality for the Google BigQuery connector.

This allows users to connect to and analyze data in Google BigQuery directly from QuickSight without the need to import the data, enabling more efficient and real-time data analysis.

AZURE
The Fabric Community Call for September 2024 focused on the "Medallion Data Lake" concept in action.

This session likely covered strategies for implementing the medallion architecture within Microsoft Fabric, providing insights into data lake design patterns, and how to manage data flow and quality across different stages (bronze, silver, and gold layers) in a data lake.

DATABRICKS
Reltio and Databricks have launched a new data pipeline solution that aims to streamline data integration and management processes.

This collaboration combines Reltio's data management capabilities with Databricks' data and AI platform to help businesses accelerate data processing, ensure data quality, and facilitate more effective analytics and insights.

SNOWFLAKE
Finally some real news about Snowflake other than their stock price…

Snowflake has identified traditional large enterprises as its top customers but is not currently investing more in GPUs.

This suggests that while Snowflake remains focused on serving the data needs of established companies, it may not be prioritizing GPU-based workloads, which are typically associated with AI and machine learning tasks.

VECTOR DATABASES
This is a post that is part of a series. "Vector Databases Part 4: Vector Indexes" from Oralytics, delves into the concept of vector indexes in the context of vector databases.

It likely explains how vector indexes are used to efficiently search and retrieve high-dimensional data, which is essential for applications like machine learning, recommendation systems, and natural language processing.

Here are some more articles:

• Vector Database – Part 1 – An introduction and overview

• Vector Databases – Part 2 – Core Terms explained

• Vector Databases – Part 3 – Vector Search

• Vector Databases – Part 4 – Vector Indexes

• Using Cohere to Generate Vector Embeddings

• Using OpenAI to Generate Vector Embeddings

  
  
  

DEEP DIVE
Just some quick thoughts of my own not based on on any empirical evidence or data. Very much not like me.

I was lamenting with a former co-worker of mine about the role of SQL Server in the database world right now. I used to visit the page Microsoft SQL Server Versions List at least once a week. At one point in time in the not so distant past, I had to administer over 50 SQL servers by myself.

SQL Server was a flagship product for Microsoft, in my opinion. In modern times, I really don’t hear much about it. It seems it is now just another database service in Azure. Not much focus or even promotion of it anymore. in my opinion.

I think Microsoft have a strategy of elevating this database and pushing it as a starting point for data professionals and software development. I always found it easier to use than Oracle.

Nothing too crazy today. See you on Wednesday.

Gladstone