Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K

SQL 100

SQL Data Engineering Data Engineer Engineering 100

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 129

SQL Data Engineering Data Engineer Engineering 129

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 168

SQL MySQL Programming Language Relational Database 168

Analytics Vidhya

MARCH 14, 2023

This results in the generation of so much data daily. This generated data is stored in the database and will maintain it. SQL is a structured query language used to read and write these databases.

SQL 202

SQL Database Technology Data 202

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Analytics Vidhya

FEBRUARY 27, 2023

So, we are […] The post How to Normalize Relational Databases With SQL Code? If a corrupted, unorganized, or redundant database is used, the results of the analysis may become inconsistent and highly misleading. appeared first on Analytics Vidhya.

Relational Database 255

Relational Database Database SQL Coding 255

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Structured Query Language is a powerful language to manage and manipulate data stored in databases. SQL is widely used in the field of data science and is considered an essential skill to have if you work with data.

SQL 223

SQL Relational Database Data Science Database 223

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 204

Database SQL Coding Accessible 204

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineering 240

Data Engineering Data Engineer Engineering Data 240

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is.

SQL 182

SQL Python Coding IT 182

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

OCTOBER 6, 2024

link] Ravi Vedula: Leveraging AI for next-level data productivity in IDEAS In the new era of LLM, data engineers strive to improve our interaction with the vast data storage in the data warehouse. “text-to-SQL” and “text-to-insight.” Sonnet, GPT-4o, Mistral Large 2, Llama 3.1

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution.

Data Pipeline 109

Data Pipeline Python Data Engineering Data Engineer 109

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. appeared first on Confessions of a Data Guy.

ETL Tools 113

ETL Tools SQL Data Engineering Data Engineer 113

Data Engineering Weekly

MAY 26, 2024

[link] Ryan Eakman: SQLFrame - Turning PySpark into a Universal DataFrame API SQL or DataFrames, two programming models often used interchangeability in engineering data. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineering 76

Data Engineering Data Engineer Engineering Google Cloud 76

Data Engineering Weekly

APRIL 7, 2024

VCs have moved on from data catalogs, yet practitioners told me they look forward to solving data discovery. link] Pinterest: How we built Text-to-SQL at Pinterest Last week Intuit shared its key learning building Text 2 SQL , and Pinterest publishes the tech deep dive on how its internal Text2SQL work.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Data Engineering Weekly

OCTOBER 13, 2024

Genie addresses these challenges by providing quick and accurate answers to questions, retrieving relevant information from internal knowledge bases, and reducing the need for constant interaction with on-call engineers.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Data Engineering Weekly

SEPTEMBER 29, 2024

link] Google: SQL Has Problems - We Can Fix Them - Pipe Syntax In SQL It was a good weekend read about the proposed pipe syntax in SQL, which is more similar to Unix pipes in terms of its core concept—sequential data flow and transformation. Unix pipes typically represent a physical flow of data between processes.

Data Engineering 62

Data Engineering Data Engineer Engineering SQL 62

Data Engineering Weekly

SEPTEMBER 15, 2024

During his keynote, he’ll share his insights into how data and pattern recognition shape the world, build trust, and—sometimes—even help us predict the future. The blog narrates how the lack of neutral guidance or standards leads to ambiguity and the formation of a central team to bring trust in data.

Data Engineering 67

Data Engineering Data Engineer Engineering Data Lake 67

Data Engineering Weekly

JULY 7, 2024

The authors implement this using Airflow's dynamic task mapping to automatically clone tagged models to each contributor's development dataset, allowing seamless dbt builds with production-like data. The blog explains how we can programmatically measure the real cost of Azure databricks SQL warehouse instances.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Weekly

SEPTEMBER 22, 2024

Uber: QueryGPT – Natural Language to SQL Using Generative AI Companies are increasingly adopting the natural language interface to interact with the enterprise data. link] Sponsored: IMPACT - Have you seen the agenda for IMPACT: The Data Observability Summit yet?

Data Engineering 70

Data Engineering Data Engineer Engineering Data Lake 70

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineering 80

Data Engineering Data Engineer Engineering SQL 80

Data Engineering Weekly

JULY 14, 2024

[link] Wasteman.codes: Engineering Principles for Building Financial Systems Any system that involves moving money or counting money is always complex to get it correct. I used to joke the entire Walstreet runs on unknown SQL codes that no one understands.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Team Data Science

DECEMBER 17, 2020

Even then, developing a team to successfully manage AI tasks is essential to tackle any challenges faced by organizations as regard data. While studies over the last few years have shown that data engineering, a vital phase forward to Data Science and AI, it is still a phase that many businesses completely skip, misunderstand, or neglect.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Seattle Data Guy

FEBRUARY 26, 2024

SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services.

SQL 130

SQL Project Data Consulting 130

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 131

SQL Data Data Engineering Data Engineer 131

Confessions of a Data Guy

DECEMBER 29, 2023

The post SQL Bad, Reddit Mad appeared first on Confessions of a Data Guy.

SQL 100

SQL Data Data Engineering Data Engineer 100

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.

Data Engineering 96

Data Engineering Data Engineer Engineering Software Engineer 96

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Cleanse 98

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Missing data? The data you’re looking for is already in your data warehouse and BI tools. No more scripts, just SQL.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100