Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 204

Database SQL Coding Accessible 204

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 137

SQL Data Engineering Data Engineer Engineering 137

Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 168

SQL MySQL Programming Language Relational Database 168

Start Data Engineering

OCTOBER 17, 2024

Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. STRUCT data types are sorted based on their keys from left to right 1.4. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is.

SQL 182

SQL Python Coding IT 182

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 119

Data Engineering Data Engineer Engineering Generalist 119

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Weekly

JULY 28, 2024

[link] Instacart: Data Science Spotlight - Cracking the SQL Interview at Instacart (LLM Edition) Instacart writes about integrating LLM in their interview process and how it helps them identify the right candidates. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 111

Data Engineering Data Engineer Engineering Database-centric 111

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K

SQL 100

SQL Data Engineering Data Engineer Engineering 100

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution.

Data Pipeline 113

Data Pipeline Python Data Engineering Data Engineer 113

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0

SQL 134

SQL Python Data Engineering Data Engineer 134

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineering 106

Data Engineering Data Engineer Engineering Metadata 106

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. In the ELT, the load is done before the transform part without any alteration of the data leaving the raw data ready to be transformed in the data warehouse. dbt Labs also popularised the analytics engineer role.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

KDnuggets

JANUARY 19, 2023

In this article, we will discuss use cases and methods for using ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes along with SQL to integrate data from various sources.

SQL 137

SQL Data Integration Data Process 137

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineering Data Engineer 135

Christophe Blefari

OCTOBER 19, 2023

With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries. Fast forward to 2 years later, you find yourself with hundreds or thousands of SQL queries.

Project 264

Project Finance SQL Government 264

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. appeared first on Confessions of a Data Guy.

ETL Tools 113

ETL Tools SQL Data Engineering Data Engineer 113

Confessions of a Data Guy

DECEMBER 29, 2023

The post SQL Bad, Reddit Mad appeared first on Confessions of a Data Guy.

SQL 100

SQL Data Data Engineering Data Engineer 100

Data Engineering Podcast

SEPTEMBER 10, 2023

In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 208

BI SQL Machine Learning Data 208

Seattle Data Guy

FEBRUARY 26, 2024

SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services.

SQL 130

SQL Project Data Consulting 130

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

KDnuggets

DECEMBER 12, 2022

4 Useful Intermediate SQL Queries for Data Science • How to Select Rows and Columns in Pandas Using [ ],loc, iloc,at and.iat • 3 Free Machine Learning Courses for Beginners • 7 Essential Cheat Sheets for Data Engineering • 7 Techniques to Handle Imbalanced Data.

Data Science 116

Data Science SQL Machine Learning Data Engineering 116

Data Engineering Podcast

OCTOBER 29, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. With Materialize, you can! Hex brings everything together.

Building 189

Building SQL BI Python 189

Data Engineering Weekly

NOVEMBER 24, 2024

Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. The blog narrates a few examples of Pipe Syntax in comparison with the SQL queries.

Data Engineering 59

Data Engineering Data Engineer Engineering Insurance 59

databricks

NOVEMBER 13, 2023

Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways.

SQL 105

SQL Data Engineering Data Engineer Engineering 105

KDnuggets

FEBRUARY 13, 2023

SQL and Python Interview Questions for Data Analysts • Learn Machine Learning From These GitHub Repositories • Learn Data Engineering From These GitHub Repositories • The ChatGPT Cheat Sheet • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2

SQL 108

SQL Python Machine Learning Data Engineering 108

Data Engineering Weekly

JULY 7, 2024

The authors implement this using Airflow's dynamic task mapping to automatically clone tagged models to each contributor's development dataset, allowing seamless dbt builds with production-like data. The blog explains how we can programmatically measure the real cost of Azure databricks SQL warehouse instances.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Data Engineering Weekly

MAY 26, 2024

[link] Ryan Eakman: SQLFrame - Turning PySpark into a Universal DataFrame API SQL or DataFrames, two programming models often used interchangeability in engineering data. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineering 78

Data Engineering Data Engineer Engineering Google Cloud 78