Snowflake

JUNE 4, 2024

Snowflake’s new Python API (GA soon) simplifies data pipelines and is readily available through pip install snowflake. Additionally, Dynamic Tables are a new table type that you can use at every stage of your processing pipeline. Interact with Snowflake objects directly in Python. Automate or code, the choice is yours.

Data Pipeline 122

Data Pipeline Python SQL Database 122

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. We’re excited to share more innovations soon, making data even more accessible for all.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

Cloudera

AUGUST 17, 2021

We did this because we wanted to give users the greatest flexibility to define their data pipelines, that go beyond a single spark job and that can have complex sequencing logic with dependencies and triggers. With Airflow based pipelines in DE, customers can now specify their data pipeline using a simple python configuration file.

Data Pipeline 115

Data Pipeline Management BI Python 115

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 258

Data Engineering Data Engineer Engineering Data 258

Towards Data Science

MAY 2, 2023

I know the manual work you did last summer Photo by EJ Yao on Unsplash Introduction A few weeks ago, I wrote a post about developing a data pipeline using both on-premise and AWS tools. This post is part of my recent effort in bringing more cloud-oriented data engineering posts.

Data Pipeline 98

Data Pipeline Management AWS Data 98

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Precisely

DECEMBER 28, 2023

Enterprise technology is having a watershed moment; no longer do we access information once a week, or even once a day. Business success is based on how we use continuously changing data. That’s where streaming data pipelines come into play. What is a streaming data pipeline? Now, information is dynamic.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Hevo

JANUARY 26, 2025

Is your business incapacitated due to slow and unreliable data pipelines in today’s hyper-competitive environment? Data pipelines are the backbone that guarantees real-time access to critical information for informed and quicker decisions. The data pipeline market is set to grow from USD 6.81

Data Pipeline 40

Data Pipeline Data Accessibility Accessible 40

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Cloudera

AUGUST 13, 2021

Rather than collecting every single event and analyzing later, it would make sense to identify the important data as it is being collected. Let’s transform the first mile of the data pipeline. They also reduced terabytes of data ingestion, which significantly brought down the infrastructure and licensing costs by 30%.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Cloudera

DECEMBER 4, 2024

Our customers rely on NiFi as well as the associated sub-projects (Apache MiNiFi and Registry) to connect to structured, unstructured, and multi-modal data from a variety of data sources – from edge devices to SaaS tools to server logs and change data capture streams. and its potential to revolutionize data flow management.

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

Marc Lamberti

SEPTEMBER 22, 2023

Let’s imagine you have the following data pipeline: In a nutshell, this data pipeline trains different machine learning models based on a dataset and the last task selects the model with the highest accuracy. To access XComs, go to the user interface, then Admin and XComs. How to use XCom in Airflow? Yes, there is!

MySQL 246

MySQL Data Pipeline Database Python 246

Towards Data Science

APRIL 6, 2023

Today’s post follows the same philosophy: fitting local and cloud pieces together to build a data pipeline. And, when it comes to data engineering solutions, it’s no different: They have databases, ETL tools, streaming platforms, and so on — a set of tools that makes our life easier (as long as you pay for them). not sponsored.

Data Pipeline 98

Data Pipeline AWS Amazon Web Services Python 98

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are the most interesting, innovative, or unexpected ways that you have seen Trino lakehouses used?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

MARCH 6, 2023

On-premise and cloud working together to deliver a data product Photo by Toro Tseleng on Unsplash Developing a data pipeline is somewhat similar to playing with lego, you mentalize what needs to be achieved (the data requirements), choose the pieces (software, tools, platforms), and fit them together. data/ mkdir -p. .

Google Cloud 98

Google Cloud Cloud Storage Data Pipeline Cloud 98

Snowflake

JANUARY 29, 2025

As part of the private preview, we will focus on providing access inline with our product principles of ease, efficiency and trust. To request access during preview please reach out to your sales team. We do not share data with the model provider. Governance controls can be implemented consistently across data and AI.

Unstructured Data 97

Unstructured Data SQL Python Government 97

Data Engineering Podcast

APRIL 23, 2023

Applications powered by real-time data were the exclusive domain of large and/or sophisticated tech companies for several years due to the inherent complexities involved. What are the shifts that have made them more accessible to a wider variety of teams?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Towards Data Science

JANUARY 7, 2024

Streamline Data Pipelines: How to Use WhyLogs with PySpark for Effective Data Profiling and Validation Photo by Evan Dennis on Unsplash Data pipelines, made by data engineers or machine learning engineers, do more than just prepare data for reports or training models.

Data Pipeline 98

Data Pipeline Hospitality Data Validation Datasets 98

Precisely

MARCH 28, 2025

However, they faced a growing challenge: integrating and accessing data across a complex environment. Some departments used IBM Db2, while others relied on VSAM files or IMS databases creating complex data governance processes and costly data pipeline maintenance. The result?

Data Integration 69

Data Integration Pipeline-centric Database-centric Kafka 69

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Snowflake

MARCH 24, 2025

Todays organizations recognize the importance of data-driven decision-making, but the process of setting up a data pipeline thats easy to use, easy to track and easy to trust continues to be a complex challenge.

Cloud 72

Cloud Data Pipeline Data Workflow Data Engineer 72

Snowflake

APRIL 5, 2025

The Llama 4 Maverick and Llama 4 Scout models can be accessed within the secure Snowflake perimeter on Cortex AI. Integrated access via SQL and Python The Llama 4 series now available in preview on Cortex AI offer easy access through established SQL functions and standard REST API endpoints.

Architecture 74

Architecture SQL Accessibility Accessible 74

Precisely

FEBRUARY 20, 2025

Data integration ensures your AI initiatives are fueled by complete, relevant, and real-time enterprise data, minimizing errors and unreliable outcomes that could harm your business. Data integration solves key business challenges. Follow five essential steps for success in making your data AI ready with data integration.

Data Integration 69

Data Integration Government Datasets Data Pipeline 69

Striim

APRIL 18, 2025

Why AI and Analytics Require Real-Time, High-Quality Data To extract meaningful value from AI and analytics, organizations need data that is continuously updated, accurate, and accessible. Heres why: AI Models Require Clean Data: Machine learning models are only as good as their training data.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements. As a result, data often went underutilized.

Management 93

Management Government Cloud Unstructured Data 93

Striim

NOVEMBER 19, 2024

Real-Time Data Replication : Seamlessly transfer data from SQL Server to Fabric for immediate insights. Automated Data Pipelines : Benefit from automated initial load and real-time CDC pipelines, ensuring efficient data transfer. Striim automates the rest.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Snowflake

NOVEMBER 4, 2024

And for all you Python builders out there, don’t miss the instructor-led lab, where you will learn how to create an end-to-end data pipeline seamlessly in Python using Snowflake Notebooks with the Snowflake pandas API. We’ll also discuss moving to a lakehouse architecture: How will it change how your data works?

Unstructured Data 105

Unstructured Data Python Machine Learning Data Pipeline 105

Data Engineering Weekly

JANUARY 15, 2025

To safeguard sensitive information, compliance with frameworks like GDPR and HIPAA requires encryption, access control, and anonymization techniques. The AI Data Engineer: A Role Definition AI Data Engineers play a pivotal role in bridging the gap between traditional data engineering and the specialized needs of AI workflows.

Data Engineering 63

Data Engineering Data Engineer Unstructured Data Engineering 63

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

DataKitchen

NOVEMBER 5, 2024

By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment. This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Data Engineering Podcast

FEBRUARY 11, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. How does that change as a function of the type of data? What are the requirements around governance and auditability of data access that need to be addressed when sharing data? tabular, image, etc.)

Data Lake 147

Data Lake High Quality Data Government Machine Learning 147

Edureka

APRIL 22, 2025

Data Democratisation Focus Organizations are under more pressure to “democratize” data, which lets teams that aren’t experts access and use data. Data engineering services will introduce self-service analytics tools and easy-to-use data interfaces in 2025 to enhance data accessibility for all.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Data Engineering Podcast

DECEMBER 24, 2023

[Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Snowflake

JUNE 6, 2024

A look inside Snowflake Notebooks: A familiar notebook interface, integrated within Snowflake’s secure, scalable platform Keep all your data and development workflows within Snowflake’s security boundary, minimizing the need for data movement. Discover valuable business insights through exploratory data analysis. The best part?

SQL 115

SQL Python Machine Learning Data Workflow 115