Cloudera

MARCH 2, 2021

This data engineering skillset typically consists of Java or Scala programming skills mated with deep DevOps acumen. The result is that streaming data tends to be “locked away” from everyone but a small few, and the data engineering team is highly overworked and backlogged. A rare breed.

SQL 112

SQL Java Data Lake Scala 112

Monte Carlo

OCTOBER 31, 2024

Both traditional and AI data engineers should be fluent in SQL for managing structured data, but AI data engineers should be proficient in NoSQL databases as well for unstructured data management. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Christophe Blefari

SEPTEMBER 15, 2023

How to reduce warehouse costs? — Hugo propose 7 hacks to optimise data warehouse cost. Scrape & analyse football data — Benoit nicely put in perspective how to use Kestra, Malloy and DuckDB to analyse data. teej/titan — Titan is a Python library to manage data warehouse infrastructure.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Data Engineering Weekly

OCTOBER 11, 2024

Agent systems powered by LLMs are already transforming how we code and interact with data. I converted a Java streaming platform into Rust, completing the task faster and gaining valuable insights into Rust's intricacies. These systems provided centralized data storage and processing at the cost of agility.

Manufacturing 110

Manufacturing Transportation Data Warehouse Unstructured Data 110

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. My advice on this point is to learn from others.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Cloudera

OCTOBER 4, 2023

What we need is: An openness to support a wide range in streaming ingest sources, including NiFi, Spark Streaming, Flink, as well as APIs for languages like C++, Java, and Python. The ability to support not just “insert” type data changes, but Insert+update patterns as well, to accommodate both new data, and changing data.

Data Warehouse 89

Data Warehouse Telecommunication Java Manufacturing 89

Data Engineering Podcast

FEBRUARY 22, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Data Engineering Podcast

NOVEMBER 6, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Data Engineering Podcast

NOVEMBER 20, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Data Engineering Podcast

MARCH 11, 2018

Links Database Refactoring Website Book Thoughtworks Martin Fowler Agile Software Development XP (Extreme Programming) Continuous Integration The Book Wikipedia Test First Development DDL (Data Definition Language) DML (Data Modification Language) DevOps Flyway Liquibase DBMaintain Hibernate SQLAlchemy ORM (Object Relational Mapper) ODM (Object Document (..)

Database 100

Database MongoDB NoSQL Database Design 100

Picnic Engineering

FEBRUARY 13, 2025

In the early days, data was the foundation to support basic operations and learn how to achieve operational excellence. Over time, data became the driver for strategic decision-making and innovation. Our journey began by building a strong Master Data Foundation , which laid the groundwork for our first generation of systems.

Engineering 52

Engineering Database-centric Generalist Java 52

Data Engineering Podcast

OCTOBER 30, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Engineering 100

Engineering MongoDB MySQL Scala 100

Cloudera

JULY 13, 2023

The Apache Iceberg project continues developing an implementation of Iceberg specification in the form of Java Library. Several compute engines such as Impala, Hive, Spark, and Trino have supported querying data in Iceberg table format by adopting this Java Library provided by the Apache Iceberg project.

Java 70

Java Metadata PostgreSQL Data Warehouse 70

Data Engineering Podcast

AUGUST 18, 2021

What is a typical workflow for someone using Compilerworks to manage their data lineage? How does Compilerworks simplify the process of migrating between data warehouses/processing platforms? What is a typical workflow for someone using Compilerworks to manage their data lineage?

SQL 100

SQL Programming Language Java Metadata 100

Data Engineering Podcast

DECEMBER 16, 2019

You work hard to make sure that your data is clean, reliable, and reproducible throughout the ingestion pipeline, but what happens when it gets to the data warehouse? Dataform picks up where your ETL jobs leave off, turning raw data into reliable analytics.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Cloudera

MAY 3, 2021

Contrast this with the skills honed over decades for gaining access, building data warehouses, performing ETL, creating reports and/or applications using structured query language (SQL). Benefits of Streaming Data for Business Owners. A rare breed. What does all this mean for those in business leadership roles? .

Scala 103

Scala Retail Java SQL 103

Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Christophe Blefari

JULY 26, 2024

How Mixpanel delivers funnels up to 7x faster than the data warehouse — Mixpanel team is proud to say that they have better performance than Snowflake. It's written in Java and it does what other orchestrator are already doing. I think it will unlock a lot of use-cases in BigQuery. Curious to see if it will pick up.

MySQL 130

MySQL Data PostgreSQL Datasets 130

DataKitchen

AUGUST 3, 2021

The past decades of enterprise data platform architectures can be summarized in 69 words. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Data professionals are not perfectly interchangeable.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

AltexSoft

OCTOBER 30, 2021

Data engineer’s integral task is building and maintaining data infrastructure — the system managing the flow of data from its source to destination. This typically includes setting up two processes: an ETL pipeline , which moves data, and a data storage (typically, a data warehouse ), where it’s kept.

Data Engineer 98

Data Engineer Data Engineering Engineering Machine Learning 98

Data Engineering Podcast

MARCH 2, 2020

Snowplow takes care of everything from installing your pipeline in a couple of hours to upgrading and autoscaling so you can focus on your exciting data projects. Your team will get the most complete, accurate and ready-to-use behavioral web and mobile data, delivered into your data warehouse, data lake and real-time streams.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

APRIL 24, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. How do you maintain feature parity between the Python and Java integrations? How do you maintain feature parity between the Python and Java integrations?

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Podcast

DECEMBER 22, 2019

Data warehouses are optimized for batched writes and complex analytical queries. Between those use cases there are varying levels of support for fast reads on quickly changing data. Data warehouses are optimized for batched writes and complex analytical queries.

SQL 100

SQL Engineering Building Java 100

Data Engineering Podcast

JUNE 5, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Data Engineering Podcast

SEPTEMBER 25, 2022

The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Lake 100

Data Lake Food MongoDB MySQL 100

Data Engineering Podcast

NOVEMBER 13, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

MongoDB 100

MongoDB MySQL Google Cloud Scala 100

Knowledge Hut

APRIL 25, 2024

Spark provides an interactive shell that can be used for ad-hoc data analysis, as well as APIs for programming in Java, Python, and Scala. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data. The two most popular data warehouse systems are Teradata and Oracle Exadata.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Netflix Tech

JULY 26, 2021

The CDC events are passed on to the Data Mesh enrichment processor, which issues GraphQL queries to Studio Edge to enrich the data. Once the data has landed in the Iceberg tables in Netflix Data Warehouse, they could be used for ad-hoc or scheduled querying and reporting. Currently Iceberg sink is appended only.

Data 102

Data MySQL Data Pipeline Data Warehouse 102

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection.

Data Pipeline 94

Data Pipeline Building Kafka Java 94

Knowledge Hut

MAY 3, 2023

A degree in computer science, software engineering, or a similar subject is often required of data engineers. They have extensive knowledge of databases, data warehousing, and computer languages like Python or Java. Also, data engineers are well-versed in distributed systems, cloud computing, and data modeling.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Cleanse 98

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. For analytical systems there are decades of investment in data warehouses and various modeling techniques.

Machine Learning 100

Machine Learning Database MySQL MongoDB 100

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

OCTOBER 2, 2022

The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

IT 100

IT Food MongoDB PostgreSQL 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

MongoDB 100

MongoDB MySQL Scala Machine Learning 100