Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. link] Jason Liu & Eugene Yan: 10 Ways to Be Data Illiterate (and How to Avoid Them) If you ask any executives in an organization, they will say they are data-driven.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Towards Data Science

MARCH 3, 2024

Data engineering techniques for robust and sustainable ETL Continue reading on Towards Data Science »

Data Pipeline 72

Data Pipeline Building Data Science Data 72

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance.

Data Pipeline 104

Data Pipeline Data Engineer Data Engineering Engineering 104

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Cleanse 52

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Towards Data Science

MARCH 23, 2024

Data pipelines that would turn you into a decorated data professional Continue reading on Towards Data Science »

Data Engineering 92

Data Engineering Data Engineer Project Data Pipeline 92

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 130

Data Engineer Data Engineering Engineering Project 130

Data Engineering Weekly

MAY 26, 2024

seconds, enhancing real-time sports data analytics efficiency! link] Zillow: Building a strong foundation to accelerate StreetEasy’s data science efforts There is a huge difference between data and easy-to-use data. Query times have been reduced from over 90 seconds to less than 0.5

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Silectis

FEBRUARY 17, 2021

In this blog post, we’ll walk you through how data science and data engineering are complementary disciplines. We’ll also delineate a third category: data analysis. We’ll explore how both data engineering and data science should be marshaled to make better decisions.

Data Science 52

Data Science Data Engineer Data Engineering Engineering 52

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Knowledge Hut

SEPTEMBER 6, 2023

Nowadays, I often hear people saying they aspire to become data scientists or they want to work with data, but they don’t know the path to do so. I myself have faced this problem and data science certifications come as a rescue for this problem. What is Data Science Certification?

Data Science 97

Data Science Certification Business Analyst Machine Learning 97

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. I bet you are, this article is for you!

Data Engineer 82

Data Engineer Data Engineering Engineering Data Lake 82

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Cleanse 98

Data Engineering Weekly

DECEMBER 3, 2023

link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. Lackluster AI/ML results often stem from poor data quality.

Data Engineer 70

Data Engineer Data Engineering Engineering Bytes 70

Knowledge Hut

DECEMBER 26, 2023

Of course, handling such huge amounts of data and using them to extract data-driven insights for any business is not an easy task; and this is where Data Science comes into the picture. You can execute this by learning data science with python and working on real projects.

Data Science 97

Data Science BI Business Intelligence Machine Learning 97

Towards Data Science

DECEMBER 11, 2023

Streaming data pipelines and real-time analytics Continue reading on Towards Data Science »

Data Engineering 73

Data Engineering Data Engineer Engineering Data Pipeline 73

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

Knowledge Hut

JANUARY 18, 2024

I’ve often noticed that people use terms like Data Science and Artificial Intelligence ( AI ) interchangeably. The key connection between Data Science and AI is data. Understanding  Data Science course eligibility  can help you understand more about Data Science. What is Data Science?

Data Science 98

Data Science Deep Learning Business Analyst Data Mining 98

Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. The only thing worse than having bad data is not knowing that you have it.

Data Engineer 100

Data Engineer Data Engineering Engineering Python 100

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Via the UI or via an API.

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Towards Data Science

JULY 27, 2024

Top trends to help your data pipelines scale with ease Continue reading on Towards Data Science »

Data Engineering 53

Data Engineering Data Engineer Engineering Data Pipeline 53

Towards Data Science

SEPTEMBER 10, 2023

Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale Image by Cosmin Paduraru As data teams scale up on the cloud, data platform teams need to ensure the workloads they are responsible for are meeting business objectives. Okay, so what’s the problem?

Data Pipeline 85

Data Pipeline Data Coding Utilities 85

KDnuggets

MARCH 17, 2020

As the role of the data engineer continues to grow in the field of data science, so are the many tools being developed to support wrangling all that data. Five of these tools are reviewed here (along with a few bonus tools) that you should pay attention to for your data pipeline work.

Data Engineer 130

Data Engineer Data Engineering Engineering Project 130

Knowledge Hut

DECEMBER 29, 2023

The market for analytics is flourishing, as is the usage of the phrase Data Science. Professionals from a variety of disciplines use data in their day-to-day operations and feel the need to understand cutting-edge technology to get maximum insights from the data, therefore contributing to the growth of the organization.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 79

Data Engineer Data Engineering Engineering SQL 79

Towards Data Science

JULY 24, 2023

Build scalable and fast data pipelines with Polars Continue reading on Towards Data Science »

Data Pipeline 76

Data Pipeline Data Science Data Building 76

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification.

Data Engineer 70

Data Engineer Data Engineering Engineering Kafka 70

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.

Data Engineer 96

Data Engineer Data Engineering Engineering Software Engineer 96

Towards Data Science

SEPTEMBER 5, 2023

Establishing data pipelines towards automated spatial data science Continue reading on Towards Data Science »

Data Engineering 67

Data Engineering Data Engineer Engineering Data Science 67

Knowledge Hut

MAY 9, 2024

Data Science, with its interdisciplinary approach, combines statistics, computer science, and domain knowledge and has opened up a world of exciting and lucrative career opportunities for professionals with the right skills and expertise. The market is flooding with the highest paying data science jobs.

Data Science 52

Data Science Data Architect Data Mining Programming Language 52

Towards Data Science

AUGUST 22, 2023

As a founder of a VC funded startup in the data / technology space, I’ve seen a lot of things and have spoken to a lot of other founders. A lot of readers are probably familiar with working at a “big company” as a data engineer. You want to understand how critical the data pipelines are for the business.

Data Engineer 88

Data Engineer Data Engineering Engineering Data Pipeline 88

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Snowflake

AUGUST 24, 2023

The family of companies, a part of Radian Group, relies on data science and machine learning to build and deploy models using a variety of data from different sources to bring technology to all parts of the home-buyer experience. For homegenius’ data science programs, Snowflake has significantly sped up their work.

Data Pipeline 52

Data Pipeline Python Programming Language Data Validation 52

Towards Data Science

FEBRUARY 12, 2024

In this first article, we’re exploring Apache Beam, from a simple pipeline to a more complicated one, using GCP Dataflow. Let’s learn what… Continue reading on Towards Data Science »

Data Pipeline 92

Data Pipeline Data Process Process Data Science 92

Towards Data Science

JULY 6, 2023

Maximize the quality and productivity of your data pipelines Continue reading on Towards Data Science »

Data Pipeline 65

Data Pipeline Data Science Data Data Engineering 65

Data Engineering Weekly

SEPTEMBER 8, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. How We Created Data Science Personas for Spotify’s Analytics Platform It takes a village to engineer data.

Data Engineer 52

Data Engineer Data Engineering Engineering Kafka 52