Data Engineering Podcast

MAY 7, 2023

Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it. In this episode Kevin Niparko and Hanhan Wang share the details of how it is implemented and how you can use it to build and maintain rich customer profiles.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. The blog is an excellent summarization of the common patterns emerging in GenAI platforms. Pipeline breakpoint feature.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Data Engineering Weekly

JUNE 2, 2024

Learn More → AI Verify Foundation: Model AI Governance Framework for Generative AI Several countries are working on building governance rules for Gen AI. The author highlights the structured approach to building data infrastructure, data management, and metrics. TIL that the queryable state is deprecated, which surprises me too.

Data Engineering 103

Data Engineering Data Engineer Engineering Pipeline-centric 103

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process.

Process 95

Process Pipeline-centric Media Metadata 95

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing. How the UK football rely heavily on data?

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

DataKitchen

AUGUST 27, 2021

DataOps is fundamentally about eliminating errors, reducing cycle time, building trust and increasing agility. The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Analytics Data Lake 98

Cloudera

APRIL 22, 2021

Cloudera has partnered with Cisco in helping build the Cisco Validated design (CVD) for Apache Ozone. Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Cloudera will publish separate blog posts with results of performance benchmarks.

Pipeline-centric 101

Pipeline-centric Data Lake Hadoop Metadata 101

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. The post Cloudera Customer Story appeared first on Cloudera Blog. Please read the full story here.

Pipeline-centric 89

Pipeline-centric Professional Services BI Datasets 89

Snowflake

JUNE 28, 2023

Streamlit has rapidly become the de facto way to build UIs for LLM-powered apps. Get smarter about your data with native LLMs Additionally, Snowflake is building LLMs directly into the platform to help customers boost productivity and unlock new insights from their data. Read this blog. And with LLMs, it’s no different.

Pipeline-centric 94

Pipeline-centric Unstructured Data Data Government 94

Towards Data Science

MAY 19, 2023

Data engineers spend countless hours troubleshooting broken pipelines. Data plays a central role in modern organisations; the centricity here is not just a figure of speech, as data teams often sit between traditional IT and different business functions. More can be found in this blog. Know when to build and when to buy.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. The blog is a good overview of various components in a typical data stack. I often wonder if we are building a pyramid infrastructure scheme on top of the object storage.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Maxime Beauchemin

JANUARY 20, 2017

Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you? link] Nvidia: What Is Sovereign AI?

Data Engineering 67

Data Engineering Data Engineer Engineering Pipeline-centric 67

Cloudera

NOVEMBER 15, 2021

This blog discusses quantifications, types, and implications of data. The activity in the field of learning with limited data is reflected in a variety of courses , workshops , reports , blogs and a large number of academic papers (a curated list of which can be found here ). Quantifications of data. Addressing the challenges of data.

Unstructured Data 90

Unstructured Data Pipeline-centric Database-centric Entertainment 90

DoorDash Engineering

NOVEMBER 28, 2023

It is amusing for a human being to write an article about artificial intelligence in a time when AI systems, powered by machine learning (ML), are generating their own blog posts. We also shed light on how we drove value by taking a user-centered approach while building this internal tool.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Data Engineers are engineers responsible for uncovering trends in data sets and building algorithms and data pipelines to make raw data beneficial for the organization.

Data Engineering 98

Data Engineering Data Engineer Engineering Hadoop 98

Lyft Engineering

APRIL 3, 2023

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders. This blog mostly focuses on the mode selector to explain how rankings have evolved in the past years and briefly touches on the post request cross-sells.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87

DataKitchen

NOVEMBER 4, 2021

The DataKitchen Platform serves as a process hub that builds temporary analytic databases for daily and weekly ad hoc analytics work. These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. . The DataOps Advantage .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Workfall

AUGUST 29, 2023

Learn how to build your very own Amazon Price Tracker using Python. In this blog, we will cover: About Python About Amazon About BeautifulSoup Library Hands-On Conclusion About Python Python is a high-level, interpreted, and versatile programming language known for its simplicity and readability. appeared first on The Workfall Blog.

Python 93

Python Pipeline-centric Programming Language Coding 93

Knowledge Hut

SEPTEMBER 24, 2024

In this blog post, we will see the top Automation testing tools used in the software industry. Ranorex Ranorex is a set of paid and free tools to build sophisticated tests for desktop, web, and mobile applications. We have so many Automation tools available in the market to perform automation testing. Support integration with CICD.

Java 98

Java Programming Language Pipeline-centric Database-centric 98

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. At the end of 2022 we decided to collect the blogs we enjoyed the most over the year. For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

Pipeline-centric 52

Pipeline-centric Database-centric SQL Data Warehouse 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Database-centric 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Building the KPay payment system. Pillar 1 – Business function: Payment processing pipeline.

Kafka 94

Kafka Pipeline-centric Architecture Systems 94

Cloudera

OCTOBER 17, 2023

This blog is for anyone who was interested but unable to attend the conference, or anyone interested in a quick summary of what happened there. Use cases such as fraud monitoring, real-time supply chain insight, IoT-enabled fleet operations, real-time customer intent, and modernizing analytics pipelines are driving development activity.

Kafka 79

Kafka Database-centric Pipeline-centric Database 79

U-Next

AUGUST 17, 2022

SQL – A database may be used to build data warehousing, combine it with other technologies, and analyze the data for commercial reasons with the help of strong SQL abilities. Pipeline-centric: Pipeline-centric Data Engineers collaborate with data researchers to maximize the use of the info they gather.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Data Pipeline 52

Data Pipeline Healthcare Medical Pipeline-centric 52

Snowflake

JUNE 28, 2023

It provides familiar APIs for various data centric tasks, including data preparation, cleansing, preprocessing, model training, and deployments tasks. In the warehouse model, users can seamlessly run and operationalize data pipelines, ML models, and data applications with user-defined functions (UDFs) and stored procedures (sprocs).

Python 52

Python Accessibility Accessible Pipeline-centric 52

Data Engineering Weekly

APRIL 2, 2023

link] Tweet Search System (EarlyBird) Design [link] Google AI: Data-centric ML benchmarking - Announcing DataPerf’s 2023 challenges Data is the new code: it is the training data that determines the maximum possible quality of an ML solution. I echoed a similar statement here. Seriously, come on, Redshift team!!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

RandomTrees

SEPTEMBER 27, 2024

With One Lake serving as a primary multi-cloud repository, Fabric is designed with an open, lake-centric architecture. You can use Copilot to build reports, summarize insights, build pipelines, and develop ML models. What’s new ever since Fabric’s GA announcement in November 2023? This preview will roll out in stages.

Database-centric 52

Database-centric Pipeline-centric IT BI 52

Cloudera

SEPTEMBER 11, 2024

Pharmaceuticals : Pharmaceutical companies need to integrate sensitive data, including Personally Identifiable Information (PII), across the R&D pipeline to accelerate the development of life-saving medications. Healthcare : Healthcare providers are under pressure to leverage data to deliver patient centricity and a continuum of care.

Pharmaceutical 52

Pharmaceutical Cloud Pipeline-centric Healthcare 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. DoorDash writes about its journey to build a Metric Layer for experimentation.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

FEBRUARY 21, 2023

As a result, a less senior team member was made responsible for modifying a production pipeline. Build analytic data systems that have modular, reusable components. Build components that are idempotent on data. To add fuel to the fire, a few weeks ago, one of the most talented people in the group left.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Snowflake

AUGUST 21, 2023

In this blog post, we’ll review the core data mesh principles, highlight how both organizations and modern data platforms are putting those principles into action, and demonstrate just how achievable a secure and efficient data mesh architecture can be. It’s also crucial that teams consider the price tag of the data mesh.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Monte Carlo

JULY 28, 2022

These are particularly frustrating, because while they are breaking data pipelines constantly, it’s not their fault. As Convoy Head of Product, Data Platform, Chad Sanderson wrote in our blog , “The data in production tables are not intended for analytics or machine learning. He suggested : “Private vs. public methods.

Software Engineering 52

Software Engineering Software Engineer Pipeline-centric Database-centric 52

Edureka

JULY 14, 2024

In this blog, we’ll discuss DevOps release management, its process, best practices, and the advantages of release manager in Devops. It encompasses the planning, scheduling, and controlling of software builds and delivery pipelines. This is where DevOps Release Management comes in. What is Release Management?

Management 40

Management Pipeline-centric Coding Software Engineering 40