ProjectPro

JUNE 6, 2025

Building data pipelines is a core skill for data engineers and data scientists as it helps them transform raw data into actionable insights. In this blog, you’ll build a complete ETL pipeline in Python to perform data extraction from the Spotify API, followed by data manipulation and transformation for analysis.

Python 40

Python Building PostgreSQL Raw Data 40

Teradata

MAY 30, 2025

The scripts demonstrate how to easily extract data from a source into Vantage with Airbyte, perform necessary transformations using dbt, and seamlessly orchestrate the entire pipeline with Dagster. These are the core of your project, where you write SQL to transform raw data into an analytics-friendly format.

Data Integration 52

Data Integration Raw Data Metadata Data Pipeline 52

ProjectPro

JUNE 6, 2025

Table of Contents 20 Open Source Big Data Projects To Contribute How to Contribute to Open Source Big Data Projects? 20 Open Source Big Data Projects To Contribute There are thousands of open-source projects in action today. This blog will walk through the most popular and fascinating open source big data projects.

Big Data 49

Big Data Project Metadata Programming Language 49

ProjectPro

JUNE 6, 2025

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data.

Big Data 40

Big Data Data Process Process Kafka 40

ProjectPro

JUNE 6, 2025

Learning Snowflake data Warehouse is like gaining a superpower for handling and analyzing data in the cloud. This blog is a definitive guide to mastering how to learn Snowflake data warehouse for all aspiring data engineers. That's exactly what Snowflake Data Warehouse enables you to do!

Data Warehouse 40

Data Warehouse SQL AWS Big Data 40

ProjectPro

JUNE 6, 2025

Become an ETL wizard and demystify the world of data transformation with our detailed blog on How to Learn ETL. So, if you are willing to build a successful big data career, this is the perfect ETL tutorial for you! Imagine an organization gathering heaps of data daily, like sales figures, customer data, and product inventory.

ETL Tools 45

ETL Tools AWS Big Data Data Validation 45

Lyft Engineering

APRIL 21, 2023

In a previous blog post , we explored the architecture and challenges of the platform. In our previous blog , we discussed the various challenges we faced in model monitoring and our strategy to address some of these issues. The profiles are very compact and efficiently describe the dataset with high fidelity.

Systems 112

Systems Building Machine Learning Raw Data 112

Cloudera

MARCH 29, 2021

It also provides an advanced materialized view engine to enable live aggregated datasets to be accessible by other applications via a simple REST API. Data decays. Yes, data has a shelf life. This allows users to run continuous queries on data streams over specific time windows. Register NOW!

SQL 116

SQL Scala Manufacturing Kafka 116

ProjectPro

JUNE 6, 2025

However, it might not be ideal for time series data because it requires importing all helper classes for the year, month, week, and day formatters. It's also inconvenient when dealing with several datasets, but converting a dataset into a long format and plotting it is simple. total size of data’).

Python 40

Python Datasets Programming Language Data Science 40

ProjectPro

JUNE 6, 2025

This blog answers all your questions about how to learn AIOps- the latest marvel in the tech world that empowers organizations to thrive in an increasingly dynamic and competitive landscape of AI. AIOps analyzes extensive IT data in real time, offering actionable insights to detect and address issues before they escalate.

Machine Learning 40

Machine Learning Algorithm Big Data Aggregated Data 40

ProjectPro

JUNE 6, 2025

Are you struggling to adapt data analysis techniques? Look no further than Pandas Functions to streamline your efforts and advance your skills in data manipulation. This blog is a guided tour through the must-know Pandas functions that will empower you to manipulate, transform, and extract insights from your data like never before.

Python 40

Python Datasets Data Analysis Data Science 40

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 99

Bytes Datasets Metadata Data 99

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 91

Process Data Pipeline Datasets Aggregated Data 91

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Data Engineering to ingest bulk data and data from mainframes.

Database 83

Database Machine Learning Kafka Data Lake 83

ProjectPro

JUNE 6, 2025

Are you passionate about data but lack experience as a data analyst? This comprehensive blog will show you how to become a data analyst with no experience, breaking down the process into simple steps and providing you with resources and tools to help you along the way. Filter, sort, and aggregate data with ease.

Portfolio 40

Portfolio Programming Language Consulting Hadoop 40

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? It turns out that Apache Impala scales down with data just as well as it scales up. The entire collection is available here.

Metadata 143

Metadata SQL Coding Database 143

ProjectPro

JUNE 6, 2025

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. A pipeline may include filtering, normalizing, and data consolidation to provide desired data.

Data Pipeline 40

Data Pipeline Architecture Kafka Data Lake 40

Workfall

SEPTEMBER 18, 2023

Pair this with Snowflake , the cloud data warehouse that acts as a vault for your insights, and you have a recipe for data-driven success. Get ready to explore the realm where data dreams become reality! In this blog, we will cover: What is Airbyte? With Airbyte and Snowflake, data integration is now a breeze.

Raw Data 93

Raw Data Data Pipeline Data Schemas Healthcare 93

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 109

Kafka Data Ingestion Architecture Datasets 109

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

ProjectPro

JUNE 6, 2025

It is also essential to practice with simple datasets initially, gradually advancing to more complex ones to create various visualizations, dashboards, and analyzes to hone your data skills. This integration enhances the depth and flexibility of data exploration, making complex data easier to understand and interpret.

Data Science 40

Data Science Datasets Telecommunication Portfolio 40

LinkedIn Engineering

DECEMBER 20, 2023

In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. At the core of inference at scale lies the fusion of ML with a wealth of data.

Building 91

Building Kafka Algorithm Machine Learning 91

Netflix Tech

APRIL 26, 2022

The Iceberg table created by Keystone contains large blobs of unstructured data. These large unstructured blogs are not efficient for querying, so we need to transform and store this data in a different format to allow efficient queries. As our label dataset was also random, presorting facts data also did not help.

Metadata 81

Metadata Datasets Machine Learning Designing 81

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

ProjectPro

JUNE 6, 2025

SQL provides a unified language for efficient interaction where data sources are diverse and complex. Despite the rise of NoSQL, SQL remains crucial for querying relational databases, data transformations, and data-driven decision-making.

SQL 40

SQL Relational Database Business Analyst Database 40

Knowledge Hut

JANUARY 25, 2024

It doesn't matter if you're a data expert or just starting out; knowing how to clean your data is a must-have skill. The future is all about big data. This blog is here to help you understand not only the basics but also the cool new ways and tools to make your data squeaky clean. What is Data Cleaning?

Data Cleanse 52

Data Cleanse Data Preparation Datasets Aggregated Data 52

ProjectPro

JUNE 6, 2025

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40

DoorDash Engineering

APRIL 12, 2023

Challenges of ad-hoc SQLs Our initial goal with Curie was to standardize the analysis methodologies and simplify the experiment analysis process for data scientists. After considering the aforementioned factors and studying other existing metric frameworks, we decided to adopt standard BI data models.

SQL 83

SQL Metadata Raw Data Government 83

Knowledge Hut

OCTOBER 13, 2023

As per Microsoft, “A Power BI report is a multi-perspective view of a dataset, with visuals representing different findings and insights from that dataset. ” Reports and dashboards are the two vital components of the Power BI platform, which are used to analyze and visualize data. Read Power BI blogs and articles.

BI 52

BI Business Analyst Certification Raw Data 52

ProjectPro

JANUARY 24, 2023

One of the core features of ADF is the ability to preview your data while creating your data flows efficiently and to evaluate the outcome against a sample of data before completing and implementing your pipelines. Such features make Azure data flow a highly popular tool among data engineers.

Retail 52

Retail Big Data Data Warehouse Media 52

Airbnb Tech

APRIL 30, 2021

While we have previously shared how we ingest data into our data warehouse and how to enable users to conduct their own analyses with contextual data , we have not yet discussed the middle layer: how to properly model and transform data into accurate, analysis-ready datasets. Our work hardly stopped there, however.

Data Warehouse 52

Data Warehouse Finance Metadata Aggregated Data 52

ProjectPro

JUNE 6, 2025

In this blog, we’ll explain how to use AI in project management more efficiently, the benefits it brings to the table, the top AI project management tools making waves, and some real-world examples on how AI is transforming project management. This leads to better planning, resource allocation, and risk management.

Project 40

Project Management Machine Learning Utilities 40

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. In this blog, we distilled their migration journeys into 5 steps. We often see ingest queries aggregate data by time.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

AltexSoft

JANUARY 26, 2022

For more detailed information on data science team roles, check our video. An analytics engineer is a modern data team member that is responsible for modeling data to provide clean, accurate datasets so that different users within the company can work with them. Data modeling. What is an analytics engineer?

Engineering 59

Engineering Software Engineering Software Engineer Data Warehouse 59

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! The Data Lake acts as the central repository for aggregating data from diverse sources in its raw format.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data.

Big Data 52

Big Data Data Process Process Kafka 52

DoorDash Engineering

OCTOBER 17, 2023

Using weights in regression allows efficient scaling of the algorithm, even when interacting with large datasets. With this approach, we don’t just perform the regression computation more efficiently, we also minimize any network transfer costs and latencies and can perform much of the aggregation to get the inputs on the data warehouse.

Education 87

Education Kafka Algorithm Data Warehouse 87