Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Start Data Engineering

MAY 21, 2024

Enabling Stakeholder data access with RAGs 3.1. Loading: Read raw data and convert them into LlamaIndex data structures 3.2.1. Read data from structured and unstructured sources 3.2.2. Transform data into LlamaIndex data structures 3.3. Indexing: Generate & store numerical representation of your data 3.

Accessible 213

Accessible Accessibility SQL Raw Data 213

KDnuggets

APRIL 8, 2024

Are you looking to make a career in data science? Start by learning SQL with these free courses.

SQL 138

SQL Data Science Data 138

dbt Developer Hub

JANUARY 22, 2025

The main thing I knew going in was "SDF understands SQL". For the next era of Analytics Engineering to be as transformative as the last, dbt needs to move beyond being a string preprocessor and into fully comprehending SQL. Today we're going to dig into what SQL comprehension actually means, since it's so critical to what comes next.

SQL 78

SQL Database Coding Technology 78

KDnuggets

OCTOBER 12, 2023

SQL is the essential data science language due to its universal database accessibility, efficient data cleaning capabilities, seamless integration with other languages, and requirement for most data science jobs.

Data Science 154

Data Science SQL Database Data 154

Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. HNY 2025 ( credits ) Happy new year ✨ I wish you the best for 2025. Not really digest.

Data 130

Data Data Warehouse Coding Programming Language 130

KDnuggets

JUNE 12, 2024

A simple tutorial on how to connect to databases, execute SQL queries, and analyze and visualize data.

SQL 157

SQL Python Database Data 157

KDnuggets

JULY 31, 2024

If you work in data, you’ll write SQL queries all the time. So how do you write efficient SQL queries that are optimized for performance? This tutorial will help you with just that.

SQL 140

SQL Data 140

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

KDnuggets

NOVEMBER 12, 2024

In this tutorial, we’ll explore when and how SQL functionality can be integrated within the Pandas framework, as well as its limitations.

SQL 142

SQL Data Analysis Data IT 142

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Meanwhile, business data stewards dont have the skills to program or even tweak these tests.

SQL 73

SQL Python Government Data Engineering 73

KDnuggets

MARCH 6, 2024

Want to become an SQL expert? Check out these free courses to learn and hone your SQL skills for data science.

SQL 138

SQL Data Science Data 138

KDnuggets

MARCH 13, 2024

If you are a data scientist who works with large amounts of data and hasn’t learned SQL yet - now might be the time.

SQL 148

SQL Data Science Certification Data 148

KDnuggets

NOVEMBER 20, 2024

Can SQL be used for advanced data manipulation in data science? It sure can with these seven techniques.

Data Science 128

Data Science SQL Data IT 128

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake. Striim automates the rest.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Start Data Engineering

FEBRUARY 3, 2025

Split your SQL into smaller parts 2.1. Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago. Introduction 2. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Conclusion 4.

SQL 147

SQL Data Pipeline Data 147

Start Data Engineering

OCTOBER 17, 2024

Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. STRUCT data types are sorted based on their keys from left to right 1.4. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

KDnuggets

JUNE 26, 2024

Learning SQL in 2024 is essential as it remains the most in-demand skill for data professionals, enabling efficient management and analysis of large datasets.

SQL 134

SQL Datasets Management IT 134

databricks

OCTOBER 14, 2024

Over the last few years, we've seen tremendous growth and adoption of Databricks SQL , our intelligent data warehouse purpose-built on the Data.

SQL 131

SQL Data Warehouse Data 131

KDnuggets

JANUARY 14, 2025

What are essential commands for data analysis in SQL? This article will answer this question by mentioning 10 SQL commands. As a bonus, well add some other SQL commands that are not that common but will certainly add flexibility to your analyses.

Data Analysis 114

Data Analysis SQL Data 114

KDnuggets

JANUARY 5, 2024

Want to clean your messy data so you can start analyzing it with SQL? Learn how to handle missing values, duplicate records, outliers, and much more.

SQL 133

SQL Data IT 133

KDnuggets

SEPTEMBER 9, 2023

This comprehensive SQL tutorial covers everything from setting up your SQL environment to mastering advanced concepts like joins, subqueries, and optimizing query performance. With step-by-step examples, this guide is perfect for beginners looking to enhance their data management skills.

SQL 140

SQL Data Management Management Data 140

KDnuggets

MARCH 5, 2024

Looking to learn SQL and databases to level up your data science skills? Learn SQL, database internals, and much more with these free university courses.

SQL 146

SQL Database Data Science Data 146

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on.

Metadata 272

Metadata Data Datasets Data Warehouse 272

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineering Data Engineer 135

KDnuggets

JANUARY 15, 2024

Learn the generic scenarios and techniques of grouping and aggregating data, partitioning and ranking data in SQL, which will be very helpful in reporting requirements.

SQL 140

SQL Aggregated Data Data Science Data 140

databricks

JANUARY 31, 2024

Welcome to the blog series covering product advancements in 2023 for Databricks SQL, the serverless data warehouse from Databricks. This is part 2.

SQL 130

SQL Programming Data Warehouse Data 130

Snowflake

MARCH 26, 2025

In this edition, we talk to Richard Meng, co-founder and CEO of ROE AI , a startup that empowers data teams to extract insights from unstructured, multimodal data including documents, images and web pages using familiar SQL queries. I experienced the thrilling pace of AI data innovation firsthand.

Unstructured Data 91

Unstructured Data SQL Data Data Workflow 91

Sync Computing

FEBRUARY 4, 2025

As data engineers, understanding the intricacies of your Databricks environment is important. But raw system data can be tricky to navigate, and sometimes you just need a quick answer to that burning question. Wow the team with insights in your Jobs, SQL warehouses, APC clusters, and DLT usage.

SQL 52

SQL Datasets Utilities Accessible 52

Waitingforcode

JANUARY 23, 2025

Even though Apache Spark SQL provides an API for structured data, the framework sometimes behaves unexpectedly. It's the case of an insertInto operation that can even lead to some data quality issues. Let's try to understand in this short article.

SQL 130

SQL Structured Data IT Data 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. Whats more, fixing the data quality issues this way often leads to new problems. Create a custom dashboard for your specific data qualityproblem.

PostgreSQL 75

PostgreSQL Data Python SQL 75

KDnuggets

NOVEMBER 13, 2023

This week, we delve into the vital world of Databases, SQL, Data Management, and Statistical Concepts in Data Science. Welcome back to Week 2 of KDnuggets’ "Back to Basics" series.

Database 135

Database SQL Data Management Management 135

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. With Materialize, you can!

BI 208

BI SQL Machine Learning Data 208

Waitingforcode

FEBRUARY 26, 2025

Is there an easier way to address the insertInto position-based data writing in Apache Spark SQL? Totally, if you use a column-based method such as saveAsTable with append mode.

SQL 130

SQL Data 130

databricks

DECEMBER 4, 2024

Databricks launches two new self-paced trainings to enhance SQL and AI-powered analytics skills The "Get Started with SQL analytics and BI" course covers how to use Databricks SQL for data analysis and Databricks AI/BI Dashboards and Genie spaces Additional courses being developed include "Databricks AI/BI for self-service analytics" and a deep dive (..)

BI 59

BI SQL Data Analysis Building 59