DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix. Azure DevOps.

Consulting 120

Consulting Machine Learning Data Science Data Pipeline 120

Monte Carlo

FEBRUARY 25, 2025

Data engineering design patterns are repeatable solutions that help you structure, optimize, and scale data processing, storage, and movement. They make data workflows more resilient and easier to manage when things inevitably go sideways. Common solutions include AWS S3 , Azure Data Lake , and Google Cloud Storage.

Designing 52

Designing Data Engineer Data Engineering Engineering 52

Ascend.io

DECEMBER 21, 2022

By harnessing the power of Ascend and Snowflake, data teams can now ingest any data from any location, and in just minutes begin releasing entirely new data products. “We’re here to help our customers focus on the data transformation that produces real value, not just busywork.

Data Ingestion 52

Data Ingestion Google Cloud Data Lake Cloud 52

Monte Carlo

NOVEMBER 14, 2023

Let’s take a look at 11 other data orchestration tools challenging Airflow’s seat at the table. But even as the modern data stack continues to evolve, Airflow maintains its title as a perennial data orchestration favorite—and for good reason. First things first—what’s Airflow?

Pipeline-centric 40

Pipeline-centric Google Cloud Python Data Workflow 40

RandomTrees

SEPTEMBER 17, 2024

This integration ensures that data governance is cohesive and consistent across all aspects of the data workflow. Unity Catalog vs. Other Data Catalog Tools: A Simple Comparison 1. Integration: Unity Catalog works easily with Databricks and ideal for those using Databricks for data & AI.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineer Data Engineering Engineering 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. This left Dylan’s team with a gap to fill.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Monte Carlo

JUNE 14, 2023

Apache Spark – Labeled as a unified analytics engine for large scale data processing, many leverage this open source solution for streaming use cases, often in conjunction with Databricks. Data orchestration Airflow : Airflow is the most common data orchestrator used by data teams.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

ProjectPro

DECEMBER 21, 2021

Machine Learning engineers are often required to collaborate with data engineers to build data workflows. As a Google research scientist, one must work on cutting-edge machine intelligence and machine learning systems and generate solutions for real-world, large-scale challenges.

Machine Learning 52

Machine Learning Engineering Algorithm Data Science 52

Knowledge Hut

MARCH 28, 2024

Strong understanding of cloud computing principles, data warehousing concepts, and best practices. If you’re on the lookout for Azure data engineer job options, you are pacing in the right direction.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Databand.ai

DECEMBER 29, 2022

In his current role as Senior Director of Product Management at Google, he focuses on BigQuery, Cloud Dataflow, Cloud DataProc, Cloud DataPrep, Cloud PubSub, and Cloud Composer.

BI 52

BI Consulting Data Science Data Governance 52

AltexSoft

SEPTEMBER 21, 2023

Accessible via a unified API, these new features enhance search relevance and are available on Elastic Cloud. The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization.

Engineering 52

Engineering NoSQL Programming Language Java 52

phData: Data Engineering

JANUARY 3, 2022

This is a config driven tool that is made by HashiCorp and is supported by over 1000+ providers such as: AWS Azure Google Cloud Oracle Alibaba Okta Kubernetes As you can see, there’s support for all the major cloud providers and various other auxiliary tooling that enterprises frequently leverage.

IT 52

IT AWS Software Engineer Software Engineering 52

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. When Airflow won’t work.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52