Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 172

SQL MySQL Programming Language Relational Database 172

Netflix Tech

DECEMBER 17, 2024

This article is the first in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. Subsequent posts will detail examples of exciting analytic engineering domain applications and aspects of the technical craft.

Engineering 85

Engineering Entertainment Amazon Web Services Utilities 85

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

dbt Developer Hub

JANUARY 22, 2025

The main thing I knew going in was "SDF understands SQL". For the next era of Analytics Engineering to be as transformative as the last, dbt needs to move beyond being a string preprocessor and into fully comprehending SQL. What is SQL comprehension? Check whether the SQL is valid.

SQL 78

SQL Database Coding Technology 78

Data Engineering Weekly

NOVEMBER 24, 2024

What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. All credit goes to the incredible data engineering community, where people are constantly writing and sharing their knowledge with the community. The blog narrates a few examples of Pipe Syntax in comparison with the SQL queries.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. appeared first on Confessions of a Data Guy.

SQL 182

SQL Python Coding IT 182

KDnuggets

DECEMBER 6, 2022

SQL is the essential language for developers, engineers, and data professionals. Intermediate knowledge in SQL gives you an edge in your data science career.

SQL 154

SQL Data Science Data Engineering 154

Start Data Engineering

OCTOBER 17, 2024

Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3. STRUCT data types are sorted based on their keys from left to right 1.4.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Data Engineering Weekly

MAY 4, 2025

Get started → Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engines, and more. Register Now Kirill Bobrov: Data Engineering: Now with 30% More B t So we beat on, boats against the current, borne back ceaselessly into the past.

Data Engineering 67

Data Engineering Data Engineer Engineering Java 67

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.

Data Engineering 248

Data Engineering Data Engineer Engineering Data 248

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake. Establish a secure connection (e.g.,

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Data Engineering Weekly

APRIL 27, 2025

Get started → Chip Huyen: Exploring three strategies - functional correctness, AI-as-a-judge, and comparative evaluation As AI development becomes mainstream, so does the need to adopt all the best practices in software engineering. The author emphasises the need for evaluation-driven development, inspired by test-driven development.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Engineering at Meta

JANUARY 22, 2025

We then built an intuitive UX into our tooling that enables developers to effectively consume all of this lineage data in a systematic way, saving significant engineering time for building privacy controls. The involved SQL queries are logged for data processing activities by the Presto and Spark compute engines (among others).

Data Warehouse 118

Data Warehouse SQL Programming Language Data 118

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. ) In a simple words dbt sits on top of your raw data to organise all your SQL queries that are defining your data assets.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Even if data engineers had the resources, they lacked the full context of data use.

SQL 73

SQL Python Government Data Engineering 73

Data Engineering Weekly

APRIL 20, 2025

[link] Jing Ge: Context Matters — The Vision of Data Analytics and Data Science Leveraging MCP and A2A All aspects of software engineering are rapidly being automated with various coding AI tools, as seen in the AI technology radar. Data engineering is one aspect where I see a few startups starting to disrupt.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Weekly

APRIL 13, 2025

link] Sponsored: The Ultimate Guide to Apache Airflow® DAGs Download this free 130+ page eBook for everything a data engineer needs to know to take their DAG writing skills to the next level (+ plenty of example code). link] All rights reserved, ProtoGrowth Inc.,

Data Engineering 62

Data Engineering Data Engineer Engineering Datasets 62

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. These two tools are hot. appeared first on Confessions of a Data Guy.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Sync Computing

FEBRUARY 4, 2025

As data engineers, understanding the intricacies of your Databricks environment is important. Thats why weve created a Databricks Health SQL Toolkit a free set of queries you can copy, paste, and run to get instant insights into your environment. Wow the team with insights in your Jobs, SQL warehouses, APC clusters, and DLT usage.

SQL 52

SQL Datasets Utilities Accessible 52

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineering 147

Software Engineering Software Engineer Engineering Data Lake 147

Data Engineering Weekly

JANUARY 15, 2025

The Critical Role of AI Data Engineers in a Data-Driven World How does a chatbot seamlessly interpret your questions? To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. How does a self-driving car understand a chaotic street scene?

Data Engineering 62

Data Engineering Data Engineer Unstructured Data Engineering 62

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Start Data Engineering

MAY 21, 2024

Introduction 2. Enabling Stakeholder data access with RAGs 3.1. Set up 3.1.1. Pre-requisite 3.1.2. Demo 3.1.3. Key terminology 3.2. Loading: Read raw data and convert them into LlamaIndex data structures 3.2.1. Read data from structured and unstructured sources 3.2.2. Transform data into LlamaIndex data structures 3.3.

Accessibility 213

Accessibility Accessible SQL Raw Data 213

Data Engineering Weekly

FEBRUARY 9, 2025

The blog post highlights the industry trend of search engines transitioning towards embedding-based systems, moving beyond traditional IDF models. link] Fernando Borretti: Composable SQL One of the biggest challenges in SQL is the unit testing. The author highlights three key challenges in SQL.

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

databricks

APRIL 30, 2025

SQL has been the lingua franca for structured data analysis for multiple decades, and we have done a lot of work in the last few

SQL 112

SQL Structured Data Data Analysis Data 112

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). From March 26th-28th 2024, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Weekly

MARCH 30, 2025

Save Your Spot → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. As a special perk for Data Engineering Weekly subscribers, you can use the code dataeng20 for an exclusive 20% discount on tickets!

Data Engineering 57

Data Engineering Data Engineer Engineering Pipeline-centric 57

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Look no further than Materialize, the streaming database you already know how to use.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Start Data Engineering

FEBRUARY 3, 2025

Split your SQL into smaller parts 2.1. Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago. Introduction 2. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Conclusion 4.

SQL 147

SQL Data Pipeline Data 147

Christophe Blefari

OCTOBER 19, 2023

With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries. Fast forward to 2 years later, you find yourself with hundreds or thousands of SQL queries.

Project 264

Project Finance SQL Government 264

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineering Data Engineer 135

Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? AI data engineers play a critical role in developing and managing AI-powered data systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52