Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineer 117

Data Engineer Data Engineering Engineering Generalist 117

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineer 130

Data Engineer Data Engineering Engineering Programming Language 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineer 130

Data Engineer Data Engineering Engineering Programming Language 130

Data Engineering Weekly

JULY 28, 2024

Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.

Data Engineer 109

Data Engineer Data Engineering Engineering Database-centric 109

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineer Data Engineering Process 217

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python 96

Python Utilities Coding Data Engineer 96

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessibility 234

Accessibility Accessible Python Database 234

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.

Scala 130

Scala Java Data Engineer Data Engineering 130

Towards Data Science

JUNE 12, 2024

Python tricks and techniques for data ingestion, validation, processing, and testing: a practical walkthrough Continue reading on Towards Data Science »

Python 52

Python Data Engineer Data Engineering Coding 52

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is feature engineering is and why/to whom it matters?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Confessions of a Data Guy

APRIL 16, 2023

You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.

Data Engineer 147

Data Engineer Data Engineering Engineering SQL 147

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

Data Engineering Weekly

OCTOBER 13, 2024

Key reasons for them are, LLM relies on probabilistic pattern matching; hence, instead of understanding the underlying mathematical concepts, LLMs might simply replicate patterns they observed in their training data. A key highlight for this is 👇🏼 Data’s still a mess. Most data initiatives fail.

Data Engineer 52

Data Engineer Data Engineering Engineering PostgreSQL 52

Confessions of a Data Guy

JUNE 3, 2024

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python 130

Python Programming IT Data 130

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python 218

Python Project Management Data Engineering 218

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. As the next edition of the evaluation series, we are looking to take Data Observability. The blog makes an excellent point on what makes data illiterate. Stay Tuned.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Towards Data Science

OCTOBER 21, 2023

Advanced ETL techniques for beginners Continue reading on Towards Data Science »

Data Engineer 83

Data Engineer Data Engineering Python Engineering 83

Confessions of a Data Guy

OCTOBER 6, 2023

I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. Running Rust inside Apache Airflow.

Data Engineer 147

Data Engineer Data Engineering Engineering Data 147

Towards Data Science

MAY 22, 2023

Solving data preparation tasks with ChatGPT Photo by Ricardo Gomez Angel on Unsplash Data engineering makes up a large part of the data science process. In CRISP-DM this process stage is called “data preparation”. It comprises tasks such as data ingestion, data transformation and data quality assurance.

Data Engineer 90

Data Engineer Data Engineering Engineering Data Science 90

Data Engineering Weekly

JULY 14, 2024

link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. Techniques for turning text data and documents into vector embeddings and structured data.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python 141

Python Cloud Data Science Machine Learning 141

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . CDP data lifecycle integration and SDX security and governance. Easy job deployment.

Data Engineer 126

Data Engineer Data Engineering Engineering Pipeline-centric 126

Data Engineering Weekly

MAY 26, 2024

link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. seconds, enhancing real-time sports data analytics efficiency! The author highlights some of the key missing features from S3.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It discusses the significance of data governance, sharing history, and generative AI's impact on data economy standards.

Data Engineer 70

Data Engineer Data Engineering Engineering Kafka 70

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Confessions of a Data Guy

JUNE 11, 2024

One of the things I love about Python is its flexibility and huge community, a community that puts out a never-ending stream of useful packages for the average Software Engineer.

Python 100

Python Building Software Engineering Software Engineer 100

Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. The only thing worse than having bad data is not knowing that you have it.

Data Engineer 100

Data Engineer Data Engineering Engineering Python 100

Data Engineering Weekly

FEBRUARY 18, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Our hope is only with the amazing community of data practitioners who constantly support us. We are so over the Big Data Era to Modern Data Stack.

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 79

Data Engineer Data Engineering Engineering SQL 79

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses.

Data Engineer 58

Data Engineer Data Engineering Engineering Software Engineering 58

Snowflake

JUNE 20, 2024

In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. The tool serves two primary functions: assessment and conversion.

Data Engineer 92

Data Engineer Data Engineering Scala Engineering 92

Confessions of a Data Guy

AUGUST 1, 2024

The post Daft: Distributed Dataframes with Python. appeared first on Confessions of a Data Guy.

Python 100

Python Data Big Data Data Engineering 100