Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineer 147

Data Engineer Data Engineering Engineering SQL 147

KDnuggets

FEBRUARY 24, 2023

This article will discuss SQL visualization, its role in augmenting the modern-day data engineer, and five categories of SQL visualization tools.

SQL 102

SQL Data Engineer Data Engineering Engineering 102

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 129

SQL Data Engineer Data Engineering Engineering 129

Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K

SQL 100

SQL Data Engineer Data Engineering Engineering 100

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 240

Data Engineer Data Engineering Engineering Data 240

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is feature engineering is and why/to whom it matters?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Data Engineering Weekly

OCTOBER 6, 2024

Minimize the data used in pipelines, aka do incremental data pipeline design. Filter data effectively to make sure the query uses partition pruning. Don’t query two years of data. Data quality rules, that is. “text-to-SQL” and “text-to-insight.” Optimize pipeline schedules.

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. This allows your applications to handle large data sets and complex workflows efficiently.

Data Pipeline 111

Data Pipeline Python Data Engineer Data Engineering 111

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

Towards Data Science

NOVEMBER 20, 2024

Data is booming. Is it worth switching to a data career now? It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc).

Data Engineer 61

Data Engineer Data Engineering Engineering Electronics 61

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake. Striim automates the rest.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Data Engineering Weekly

MAY 26, 2024

link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. seconds, enhancing real-time sports data analytics efficiency! The author highlights some of the key missing features from S3.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Analytics Vidhya

MARCH 14, 2023

This results in the generation of so much data daily. This generated data is stored in the database and will maintain it. SQL is a structured query language used to read and write these databases. Introduction In today’s world, technology has increased tremendously, and many people are using the internet.

SQL 202

SQL Database Technology Data 202

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 168

SQL MySQL Programming Language Relational Database 168

Data Engineering Weekly

OCTOBER 13, 2024

Key reasons for them are, LLM relies on probabilistic pattern matching; hence, instead of understanding the underlying mathematical concepts, LLMs might simply replicate patterns they observed in their training data. A key highlight for this is 👇🏼 Data’s still a mess. Most data initiatives fail.

Data Engineer 52

Data Engineer Data Engineering Engineering PostgreSQL 52

Analytics Vidhya

FEBRUARY 27, 2023

Introduction Data is the new oil in this century. The database is the major element of a data science project. So, we are […] The post How to Normalize Relational Databases With SQL Code? So, we are […] The post How to Normalize Relational Databases With SQL Code? appeared first on Analytics Vidhya.

Relational Database 255

Relational Database Database SQL Coding 255

Data Engineering Weekly

SEPTEMBER 29, 2024

Airbnb: Sandcastle - data/AI apps for everyone Product ideas powered by data and AI must go through rapid iteration on shareable, lightweight live prototypes instead of static proposals. link] Grab: Enabling conversational data discovery with LLMs at Grab.

Data Engineer 62

Data Engineer Data Engineering Engineering SQL 62

Data Engineering Weekly

SEPTEMBER 22, 2024

Editor’s Note: Coming Next on Comparison Matrix Series - Data LakeHouse Our mission is to empower data professionals and organizations to make informed, data-driven decisions by providing a comprehensive buyer's guide and comparison matrix for selecting the best data tools.

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. Poor data quality and unlcear data ownership remains the top challenges for the data teams. Data Mesh continuously gaining popularity among the enterprises.

Data Engineer 70

Data Engineer Data Engineering Engineering Unstructured Data 70

Data Engineering Weekly

SEPTEMBER 15, 2024

Uber: DataMesh - How Uber laid the foundations for the Data Lake Cloud migration Many companies are slowly adopting DataMesh, and Uber writes about adopting the data mesh principle. Whether or not Data Mesh is a separate product is debatable, but it is certainly an impactful framework for scaling data platforms.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Structured Query Language is a powerful language to manage and manipulate data stored in databases. SQL is widely used in the field of data science and is considered an essential skill to have if you work with data.

SQL 223

SQL Relational Database Data Science Database 223

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Data Engineering Weekly

JULY 14, 2024

link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. Techniques for turning text data and documents into vector embeddings and structured data.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 204

Database SQL Coding Accessibility 204

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is.

SQL 182

SQL Python Coding IT 182

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. appeared first on Confessions of a Data Guy.

ETL Tools 113

ETL Tools SQL Data Engineer Data Engineering 113

Team Data Science

DECEMBER 17, 2020

Organizations in several industries such as banking, healthcare, and automobiles are now acknowledging the value of data science in their mode of operation. Thus, an ideal and efficacious data science team are therefore expected to manage numerous volume of tasks.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Start Data Engineering

MAY 21, 2024

Enabling Stakeholder data access with RAGs 3.1. Loading: Read raw data and convert them into LlamaIndex data structures 3.2.1. Read data from structured and unstructured sources 3.2.2. Transform data into LlamaIndex data structures 3.3. Indexing: Generate & store numerical representation of your data 3.

Accessibility 214

Accessibility Accessible SQL Raw Data 214

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Start Data Engineering

OCTOBER 14, 2024

Code & Data 3. Using nested data types effectively 3.1. Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Nested data types can be sorted 3.3.3. Introduction 2. Use STRUCT for one-to-one & hierarchical relationships 3.2.

SQL 130

SQL Data Schemas Data Coding 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 78

Data Engineer Data Engineering Engineering SQL 78

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. Let’s introduce some chaos.”

Data Engineer 95

Data Engineer Data Engineering Engineering Software Engineer 95