Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product?

Data Process 147

Data Process Process Data Lake High Quality Data 147

The Pragmatic Engineer

NOVEMBER 21, 2023

Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process.  After this zero-byte file was deployed to prod, the Apache web server processes slowly picked up the empty configuration file. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

dbt Developer Hub

APRIL 20, 2025

We are committed to building the data control plane that enables AI to reliably access structured data from across your entire data lineage. We believe it is important for the industry to start coalescing on best practices for safe and trustworthy ways to access your business data via LLM. What is MCP?

Structured Data 98

Structured Data SQL BI Project 98

Snowflake

MARCH 31, 2025

For image data, running distributed PyTorch on Snowflake ML also with standard settings resulted in over 10x faster processing for a 50,000-image dataset when compared to the same managed Spark solution. Secure access to open source repositories via pip and the ability to bring in any model from hubs such as Hugging Face (see example here ).

Healthcare 93

Healthcare Medical Government Food 93

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Snowflake

NOVEMBER 6, 2024

But as technology speeds forward, organizations of all sizes are realizing that generative AI isn’t just aspirational: It’s accessible and applicable now. But getting a handle on all the emails, calls and support tickets had historically been a tedious and largely manual process. Cortex is doing a great job for us.”

Hospitality 96

Hospitality Medical Government Software Engineering 96

Engineering at Meta

JANUARY 22, 2025

This belief has led us to developing Privacy Aware Infrastructure (PAI) , which offers efficient and reliable first-class privacy constructs embedded in Meta infrastructure to address different privacy requirements, such as purpose limitation , which restricts the purposes for which data can be processed and used. Hack, C++, Python, etc.)

Data Warehouse 115

Data Warehouse SQL Programming Language Data 115

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 90

Metadata Management Data Governance Government 90

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. And who better to learn from than the tech giants who process more data before breakfast than most companies see in a year?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Start Data Engineering

MAY 28, 2024

Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Introduction 2. Project demo 3. Building efficient data pipelines with DuckDB 4.1. Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219

Netflix Tech

NOVEMBER 12, 2024

However, this category requires near-immediate access to the current count at low latencies, all while keeping infrastructure costs to a minimum. Introducing sufficient jitter to the flush process can further reduce contention. This process can also be used to track the provenance of increments.

Datasets 101

Datasets Computer Science Systems Kafka 101

Snowflake

JANUARY 30, 2025

Gen AI makes this all easy and accessible because anyone in an enterprise can simply interact with data by using natural language. What if our app doesnt have access to the right data and generates inaccurate results for stakeholders? Sales teams are usually boxed into dashboards to get insights.

Unstructured Data 96

Unstructured Data Entertainment Healthcare Telecommunication 96

Snowflake

FEBRUARY 12, 2025

Agents need to access an organization's ever-growing structured and unstructured data to be effective and reliable. As data connections expand, managing access controls and efficiently retrieving accurate informationwhile maintaining strict privacy protocolsbecomes increasingly complex. text, audio) and structured (e.g.,

Unstructured Data 83

Unstructured Data Government SQL Structured Data 83

The Pragmatic Engineer

APRIL 21, 2024

Other shipped things include  DALL·E 3  (image generation,)  GPT-4  (an advanced model,) and the  OpenAI API  which developers and companies use to integrate AI into their processes. Each word that spits out of ChatGPT is this same process repeated over and over again many times per second.

Engineering 363

Engineering Software Engineering Software Engineer Programming 363

Snowflake

JANUARY 29, 2025

As part of the private preview, we will focus on providing access inline with our product principles of ease, efficiency and trust. To request access during preview please reach out to your sales team. Once the model is generally available, customers can manage access to the model via role-based access control (RBAC).

Unstructured Data 94

Unstructured Data SQL Python Government 94

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements. As a result, data often went underutilized.

Management 85

Management Government Cloud Unstructured Data 85

Snowflake

NOVEMBER 26, 2024

Processing some 90,000 tables per day, the team oversees the ingestion of more than 100 terabytes of data from upward of 8,500 events daily. With an internal user base of 2,000 — and growing — the company particularly appreciated the seamless data access controls and the ability to securely share data with just a few simple clicks.

Data Warehouse 104

Data Warehouse Cloud PostgreSQL Hadoop 104

Snowflake

JANUARY 21, 2025

How can they get access to more transparency into where and why their marketing dollars are being spent (to reduce fraud, saturation and leverage for higher-level internal measurement practices, among other reasons)? Teams will also be able to work more efficiently when they can access all relevant data in one place.

Media 96

Media Unstructured Data Data Retail 96

Precisely

OCTOBER 31, 2024

Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 68

Data Architecture Architecture Metadata Government 68

Snowflake

NOVEMBER 11, 2024

Ingest data more efficiently and manage costs For data managed by Snowflake, we are introducing features that help you access data easily and cost-effectively. This reduces the overall complexity of getting streaming data ready to use: Simply create external access integration with your existing Kafka solution.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Netflix Tech

DECEMBER 17, 2024

This fragmentation leads to inconsistencies and wastes valuable time as teams end up reinventing metrics or seeking clarification on definitions that should be standardized and readily accessible. Enter DataJunction (DJ). DJ acts as a central store where metric definitions can live and evolve.

Engineering 85

Engineering Entertainment Amazon Web Services Utilities 85

Snowflake

NOVEMBER 11, 2024

This is not surprising when you consider all the benefits, such as reducing complexity [and] costs and enabling zero-copy data access (ideal for centralizing data governance).

Banking 97

Banking Finance Retail Food 97

Snowflake

MARCH 19, 2025

Attendees will discover how to accelerate their critical business workflows with the right data, technology and ecosystem access. Explore AI and unstructured data processing use cases with proven ROI: This year, retailers and brands will face intense pressure to demonstrate tangible returns on their AI investments.

Retail 92

Retail Media Entertainment Unstructured Data 92

Snowflake

APRIL 7, 2025

Our deep industry knowledge and understanding of these gaps gave us the insight to create solutions that simplify and automate compliance processes using AI. With advanced encryption, strict access controls and strong data governance, Snowflake helps us ensure the confidentiality and protection of our clients information.

Food 80

Food Data Transparency Software Engineering Software Engineer 80

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.

Data Storage 187

Data Storage Big Data Hadoop Datasets 187

Snowflake

DECEMBER 4, 2024

For years, an essential tenet of digital transformation has been to make data accessible, to break down silos so that the enterprise can draw value from all of its data. Overall, data must be easily accessible to AI systems, with clear metadata management and a focus on relevance and timeliness.

Unstructured Data 96

Unstructured Data Data Lake Deep Learning Structured Data 96

Marc Lamberti

SEPTEMBER 22, 2023

To access XComs, go to the user interface, then Admin and XComs. First thing first, xcom_push is accessible only from a task instance object. With the PythonOperator you can access it by passing the parameter ti to the Python callable function. Once we access the task instance object, we can call xcom_push. Yes, there is!

MySQL 246

MySQL Data Pipeline Database Python 246

Cloudera

DECEMBER 9, 2024

For example, a Cloudera customer saw a large productivity improvement in their contract review process with an application that extracts and displays a short summary of essential clauses for the reviewer. Benchmark tests indicate that Gemini Pro demonstrates superior speed in token processing compared to its competitors like GPT-4.

Machine Learning 75

Machine Learning Project Banking Accessible 75

Engineering at Meta

APRIL 28, 2025

Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process. However, conducting these processes outside of developer workflows presented challenges in terms of accuracy and timeliness.

Metadata 79

Metadata Data Utilities Data Warehouse 79

DataKitchen

NOVEMBER 5, 2024

The Medallion architecture is a design pattern that helps data teams organize data processing and storage into three distinct layers, often called Bronze, Silver, and Gold. The Silver layer aims to create a structured, validated data source that multiple organizations can access. How do you ensure data quality in every layer ?

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Snowflake

JANUARY 16, 2025

Customers can access these in Cortex AI via the complete function. This is done by combining parameter preserving model rewiring with lightweight fine-tuning to minimize the likelihood of knowledge being lost in the process. SwiftKV-optimized Llama 3.3 70B and Llama 3.1 405B models, referred to as Snowflake-LLama-3.3-70B

Algorithm 95

Algorithm Data Analysis Building Process 95

Ascend.io

NOVEMBER 14, 2024

But when data processes fail to match the increased demand for insights, organizations face bottlenecks and missed opportunities. Set Up Auto-Scaling: Configure auto-scaling for your data processing and storage resources. The ability to harness and analyze data effectively can make or break a company’s competitive edge.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Snowflake

APRIL 13, 2025

Manual processes can be time-consuming and error-prone. Agentic AI automates these processes, helping ensure data integrity and offering real-time insights. Leveraging advanced machine learning and natural language processing, these intelligent agents can efficiently manage and analyze vast data amounts.

Data Management 90

Data Management Management Consulting Unstructured Data 90

Precisely

FEBRUARY 20, 2025

It enables faster decision-making, boosts efficiency, and reduces costs by providing self-service access to data for AI models. Define clear goals, assess your data landscape, choose the right tools, ensure data quality and governance, and continuously optimize your integration processes. Thats where data integration comes in.

Data Integration 68

Data Integration Government Datasets Data Pipeline 68