Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 209

Database SQL Coding Accessibility 209

Data Engineering Weekly

APRIL 13, 2025

link] Sponsored: The Ultimate Guide to Apache Airflow® DAGs Download this free 130+ page eBook for everything a data engineer needs to know to take their DAG writing skills to the next level (+ plenty of example code). link] All rights reserved, ProtoGrowth Inc.,

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 172

SQL MySQL Programming Language Relational Database 172

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 129

SQL Data Engineering Data Engineer Engineering 129

Start Data Engineering

OCTOBER 17, 2024

Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. STRUCT data types are sorted based on their keys from left to right 1.4. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Data Engineering Weekly

APRIL 20, 2025

[link] Jing Ge: Context Matters — The Vision of Data Analytics and Data Science Leveraging MCP and A2A All aspects of software engineering are rapidly being automated with various coding AI tools, as seen in the AI technology radar. Data engineering is one aspect where I see a few startups starting to disrupt.

Data Engineering 68

Data Engineering Data Engineer Engineering Kafka 68

Data Engineering Weekly

APRIL 27, 2025

Before Hoptimator, Pinot ingestion often required data producers to create and manage separate, Pinot-specific preprocessing jobs to optimize data, such as re-keying, filtering, and pre-aggregating. reducing user friction, operator toil, and resource consumption on Pinot servers, while automating pipeline management.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is.

SQL 182

SQL Python Coding IT 182

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Data Engineering Weekly

JANUARY 15, 2025

The Critical Role of AI Data Engineers in a Data-Driven World How does a chatbot seamlessly interpret your questions? The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. How does a self-driving car understand a chaotic street scene?

Data Engineering 63

Data Engineering Data Engineer Unstructured Data Engineering 63

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 118

Data Engineering Data Engineer Engineering Generalist 118

Data Engineering Weekly

MARCH 30, 2025

Save Your Spot → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. link] BVP: Roadmap: Data 3.0

Data Engineering 58

Data Engineering Data Engineer Engineering Pipeline-centric 58

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Even if data engineers had the resources, they lacked the full context of data use.

SQL 73

SQL Python Government Data Engineering 73

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Weekly

JULY 28, 2024

[link] Instacart: Data Science Spotlight - Cracking the SQL Interview at Instacart (LLM Edition) Instacart writes about integrating LLM in their interview process and how it helps them identify the right candidates. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 110

Data Engineering Data Engineer Engineering Database-centric 110

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Sync Computing

FEBRUARY 4, 2025

As data engineers, understanding the intricacies of your Databricks environment is important. But raw system data can be tricky to navigate, and sometimes you just need a quick answer to that burning question. Wow the team with insights in your Jobs, SQL warehouses, APC clusters, and DLT usage.

SQL 52

SQL Datasets Utilities Accessibility 52

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Weekly

FEBRUARY 9, 2025

GetInData writes an excellent summary of adding data quality checks in a Flink streaming pipeline. link] Fernando Borretti: Composable SQL One of the biggest challenges in SQL is the unit testing. The author highlights three key challenges in SQL.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K

SQL 100

SQL Data Engineering Data Engineer Engineering 100

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineering 106

Data Engineering Data Engineer Engineering Metadata 106

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0

SQL 126

SQL Python Data Engineering Data Engineer 126

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. In the ELT, the load is done before the transform part without any alteration of the data leaving the raw data ready to be transformed in the data warehouse. dbt Labs also popularised the analytics engineer role.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Confessions of a Data Guy

MARCH 4, 2025

Context and Motivation dbt (Data Build Tool): A popular open-source framework that organizes SQL transformations in a modular, version-controlled, and testable way. Databricks: A platform that unifies data engineering and data science pipelines, typically with Spark (PySpark, Scala) or SparkSQL.

Scala 100

Scala Data Science SQL Data Engineering 100

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineering Data Engineer 135

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. appeared first on Confessions of a Data Guy.

ETL Tools 113

ETL Tools SQL Data Engineering Data Engineer 113

Christophe Blefari

OCTOBER 19, 2023

With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries. Fast forward to 2 years later, you find yourself with hundreds or thousands of SQL queries.

Project 264

Project Finance SQL Government 264

Start Data Engineering

FEBRUARY 16, 2025

SQL Tools 5. Data Wrangler 7. Introduction 2. Python environment setup 3. VSCode Primer 4. Extensions overview 1. Python test & debug 3. autoDocstring 8. Rainbow csv 9. DBT power user 5. Privacy, Performance, and Cognitive Overload 6. Conclusion 7. Recommended reading 1.

Coding 130

Coding Data Engineering Data Engineer Engineering 130

Monte Carlo

JANUARY 23, 2024

In the world of data engineering, Maxime Beauchemin is someone who needs no introduction. Currently, Maxime is CEO and co-founder of Preset , a fast-growing startup that’s paving the way forward for AI-enabled data visualization for modern companies. Enter, the data engineer. What is a data engineer today?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

KDnuggets

JANUARY 19, 2023

In this article, we will discuss use cases and methods for using ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes along with SQL to integrate data from various sources.

SQL 129

SQL Data Integration Data Process 129