Start Data Engineering

MAY 28, 2024

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline 219

Data Pipeline Python Building Data 219

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessible 239

Accessible Accessibility Python Database 239

Data Engineering Weekly

NOVEMBER 24, 2024

What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. All credit goes to the incredible data engineering community, where people are constantly writing and sharing their knowledge with the community. We are planning many exciting product lines to trial and launch in 2025.

Data Engineering 58

Data Engineering Data Engineer Engineering Insurance 58

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Recognizing this shift, Snowflake is taking a Python-first approach to bridge the gap and help users leverage the power of both worlds.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

Data Engineering Weekly

APRIL 13, 2025

link] Sponsored: The Ultimate Guide to Apache Airflow® DAGs Download this free 130+ page eBook for everything a data engineer needs to know to take their DAG writing skills to the next level (+ plenty of example code). link] All rights reserved, ProtoGrowth Inc.,

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Data Engineering Weekly

JANUARY 15, 2025

The Critical Role of AI Data Engineers in a Data-Driven World How does a chatbot seamlessly interpret your questions? To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. How does a self-driving car understand a chaotic street scene?

Data Engineering 63

Data Engineering Data Engineer Unstructured Data Engineering 63

Christophe Blefari

JANUARY 11, 2025

Agents write python code to call tools and orchestrate other agents. A few things to notice: Interest in AI grew by 190%, Prompt Engineering by 456%. Python and Java still leads the programming language interest, but with a decrease in interest (-5% and -13%) while Rust gaining traction (+13%), not sure it's related, tho.

Data 130

Data Data Warehouse Coding Programming Language 130

Analytics Vidhya

JUNE 20, 2023

Traditional data engineering solutions, such as Apache Airflow, have played an important role in orchestrating and controlling data operations in order to tackle these difficulties.

Data Process 224

Data Process Data Engineering Data Engineer Process 224

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Even if data engineers had the resources, they lacked the full context of data use.

SQL 73

SQL Python Government Data Engineering 73

The Pragmatic Engineer

OCTOBER 17, 2024

Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Python, Angular, SSR, SQLite, DuckDB, Cockroach DB, and many others. Tech stack. Benchmarking tools.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

KDnuggets

NOVEMBER 5, 2024

Enhance your data science portfolio with these seven engaging Python projects that demonstrate essential programming and software engineering skills.

Portfolio 131

Portfolio Data Science Python Project 131

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python 141

Python Cloud Data Science Machine Learning 141

Confessions of a Data Guy

JUNE 3, 2024

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python 130

Python Programming IT Data 130

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python 97

Python Utilities Coding Data Engineering 97

KDnuggets

MAY 16, 2024

This guide introduces some key techniques in the feature engineering process and provides practical examples in Python.

Engineering 122

Engineering Python Process Machine Learning 122

Edureka

JANUARY 21, 2025

One of the most popular choices among developers is Flask, a Python framework that is both lightweight and flexible. This blog will explain a core web framework, go over the basics of Python and Flask, discuss its uses, show how popular it is, compare it to Django, and give you a general idea of the pros and cons of using Flask.

Python 52

Python Building Certification Database 52

Confessions of a Data Guy

JANUARY 19, 2023

For those using Python, it’s probably one of the […] The post Replacing Pandas with Polars. I haven’t used Pandas in many a year, decades, or whatever. We’ve all been there, done that. Pandas I mean. I would dare say it’s a rite of passage for most data folk. A Practical Guide.

Python 361

Python IT Data Big Data 361

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Analytics Vidhya

FEBRUARY 20, 2023

This blog is a tutorial for building intuitive frontend interfaces for Machine Learning models using two popular open-source libraries […] The post Streamlit vs Gradio – A Guide to Building Dashboards in Python appeared first on Analytics Vidhya.

Python 152

Python Building Machine Learning Business Intelligence 152

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. __init__ covers the Python language, its community, and the innovative ways it is being used. In that time there have been a number of generational shifts in how data engineering is done. Closing Announcements Thank you for listening!

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

KDnuggets

AUGUST 16, 2024

Explore essential tools and skills for AI engineers: Python, R, big data frameworks, and cloud services essential for building and optimizing AI systems.

Engineering 129

Engineering Big Data Python Cloud 129

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? AI data engineers play a critical role in developing and managing AI-powered data systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Snowflake

SEPTEMBER 18, 2023

Enter Streamlit Streamlit is an open-source library that turns Python scripts into shareable web apps in minutes. No front-end experience is needed and apps are written in pure Python. Install additional Python packages from the Snowflake Anaconda Channel to utilize the full power of the Python ecosystem in your Streamlit app.

Python 143

Python Building Cloud Amazon Web Services 143

KDnuggets

OCTOBER 24, 2022

Preprocessing data for machine learning models is a core general skill for any Data Scientist or Machine Learning Engineer. Follow this guide using Pandas and Scikit-learn to improve your techniques and make sure your data leads to the best possible outcome.

Python 160

Python Machine Learning Data Engineering 160

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!

Data Engineering 118

Data Engineering Data Engineer Engineering Generalist 118

Start Data Engineering

JANUARY 12, 2023

Singleton, & Object pool patterns Python helpers 1. Introduction Sample project Code design patterns 1. Functional design 2. Factory pattern 3. Strategy pattern 4. Dataclass 3. Context Managers 4. Testing with pytest 5.

Designing 147

Designing Coding Python Data Pipeline 147

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. Python, Java, C++, etc.) With Materialize, you can! How does that translate into computer languages?

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Weekly

JANUARY 12, 2025

TypeScript, Python, and Rust are the fastest-growing programming languages, whereas others hold their position as it is. This blog captures the current state of Agent adoption, emerging software engineering roles, and the use case category. I always like a good overview of the conference's learning.

Pipeline-centric 58

Pipeline-centric Data Engineer Data Engineering Engineering 58

Analytics Vidhya

APRIL 17, 2023

Introduction Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science code. It uses best practices of software engineering to build production-ready data science pipelines. This article will give you a glimpse of Kedro framework using news classification tasks.

Data Science 226

Data Science Software Engineer Software Engineering Machine Learning 226

Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. The blog Prompt Engineering for a Better SQL Code Generation With LLMs is a pretty good guide on applying prompt engineering to improve productivity.

Data Engineering 110

Data Engineering Data Engineer Engineering Database-centric 110

Towards Data Science

DECEMBER 10, 2024

However, engineers who only use this method of data quality are potentially missing out on some big wins for their organization. It can also reduce anxiety for the data engineer/s responsible for the pipeline. However, engineering is all about weighing the pros and cons of different solutions. Created by the author using draw.io

PostgreSQL 75

PostgreSQL Data Python SQL 75

Edureka

JANUARY 21, 2025

Python is a sophisticated predictive analytics platform that uses libraries such as Pandas, NumPy, and Scikit-learn for data manipulation, analysis, and modeling. Python’s machine learning techniques can use past data to guess what will happen in the future. How do you build a predictive model in Python?

Python 52

Python Datasets Machine Learning Certification 52

Start Data Engineering

FEBRUARY 16, 2025

Python environment setup 3. Python test & debug 3. Introduction 2. VSCode Primer 4. Extensions overview 1. SQL Tools 5. Data Wrangler 7. autoDocstring 8. Rainbow csv 9. DBT power user 5. Privacy, Performance, and Cognitive Overload 6. Conclusion 7. Recommended reading 1.

Coding 130

Coding Data Engineer Data Engineering Engineering 130

Data Engineering Podcast

APRIL 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. __init__ covers the Python language, its community, and the innovative ways it is being used. Support Data Engineering Podcast

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277