DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional responsible for designing, implementing, and managing data solutions using Microsoft's Azure cloud platform. They work with various Azure services and tools to build scalable, efficient, and reliable data pipelines, data storage solutions, and data processing systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Databand.ai

JULY 19, 2023

This capability is particularly useful in complex data landscapes, where data may pass through multiple systems and transformations before reaching its final destination Impact analysis: When changes are made to data sources or data processing systems, it’s critical to understand the potential impact on downstream processes and reports.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Ripple Engineering

JANUARY 29, 2024

The key aspect of any business-centric team in delivering products and features is to make critical decisions on ensuring low latency, high throughput, cost-effective storage, and highly efficient infrastructure. Multiple data processing systems also make building detailed dashboards and monitoring very difficult.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Monte Carlo

NOVEMBER 14, 2023

This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. According to one Redditor on r/dataengineering, “Seems like 99/100 data engineering jobs mention Airflow.”

Pipeline-centric 40

Pipeline-centric Google Cloud Python Data Workflow 40

Snowflake

JUNE 28, 2023

Snowpark is our secure deployment and processing of non-SQL code, consisting of two layers: Familiar Client Side Libraries – Snowpark brings deeply integrated, DataFrame-style programming and OSS compatible APIs to the languages data practitioners like to use. Previously, tasks could be executed as quickly as 1-minute.

Python 52

Python Accessibility Accessible Pipeline-centric 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. Open question: how to seed data in a staging environment? Test system with A/A test. Be adaptable.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Dataquest

JANUARY 25, 2017

A data scientist is only as good as the data they have access to. Most companies store their data in variety of formats across databases and text files. This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Knowledge Hut

JULY 3, 2023

Real-time Data ingestion performs the utilization of data from various origins, does the data cleaning, validation, and preprocessing operations and at the end store it in the required format, either structured or unstructured. As real-time insights gain popularity, real-time data ingestion remains vital for companies worldwide.

Data Ingestion 52

Data Ingestion Google Cloud Pipeline-centric Media 52

Knowledge Hut

SEPTEMBER 26, 2023

Organisations are constantly looking for robust and effective platforms to manage and derive value from their data in the constantly changing landscape of data analytics and processing. These platforms provide strong capabilities for data processing, storage, and analytics, enabling companies to fully use their data assets.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Ascend.io

SEPTEMBER 14, 2023

Data engineers can find one for almost any need, from data extraction to complex transformations, ensuring that they’re not reinventing the wheel by writing code that’s already been written. PySpark, for instance, optimizes distributed data operations across clusters, ensuring faster data processing.

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Confluent

SEPTEMBER 25, 2019

Of course, this is not to imply that companies will become only software (there are still plenty of people in even the most software-centric companies), just that the full scope of the business is captured in an integrated software defined process. Here, the bank loan business division has essentially become software.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is the role of a Data Engineer? They are required to have deep knowledge of distributed systems and computer science.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Edureka

JULY 3, 2024

ADF connects to various data sources, including on-premises systems, cloud services, and SaaS applications. It then gathers and relocates information to a centralized hub in the cloud using the Copy Activity within data pipelines. Transform and Enhance the Data: Once centralized, data undergoes transformation and enrichment.

Pipeline-centric 52

Pipeline-centric Data Lake Database-centric Data Pipeline 52

Edureka

JULY 4, 2024

Databricks runs on an optimized Spark version and gives you the option to select GPU-enabled clusters, making it more suitable for complex data processing. The platform’s massive parallel processing (MPP) architecture empowers you with high-performance querying of even massive datasets. But it doesn’t stop there.

Data Lake 40

Data Lake Pipeline-centric Data Warehouse ETL Tools 40

phData: Data Engineering

JULY 18, 2023

Slow Response to New Information: Legacy data systems often lack the computation power necessary to run efficiently and can be cost-inefficient to scale. This typically results in long-running ETL pipelines that cause decisions to be made on stale or old data.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Databand.ai

DECEMBER 20, 2022

Follow Eric on LinkedIn 10) Brian Femiano Senior Data Engineer at Apple Brian is a senior data engineer with nearly two decades of experience at companies like Booz Allen Hamilton, Magnetic, Pandora, and, most recently, Apple. He is also a member of The Apache Software Foundation.

Data Analytics 52

Data Analytics Google Cloud Data Science Data Mining 52