Data Engineering Weekly

NOVEMBER 3, 2024

impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team. The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development.

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Data Engineering Weekly

JANUARY 12, 2025

With Astro, you can build, run, and observe your data pipelines in one place, ensuring your mission critical data is delivered on time. meeting recordings and videos), which contrasts with traditional SQL-centric systems for structured data. Generative AI demands the processing of vast amounts of diverse, unstructured data (e.g.,

Pipeline-centric 57

Pipeline-centric Data Engineer Data Engineering Engineering 57

Pinterest Engineering

SEPTEMBER 20, 2023

Our commitment is evidenced by our history of building products that champion inclusivity. We know from experience that building for marginalized communities helps make the product work better for everyone. To ensure an unbiased approach, we also leveraged our skin tone and hair pattern signals when building this dataset.

Building 109

Building Pipeline-centric Machine Learning Datasets 109

DataKitchen

NOVEMBER 5, 2024

We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric. The need to copy data across layers, manage different schemas, and address data latency issues can complicate data pipelines. However, this architecture is not without its challenges.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Data Engineering Weekly

MARCH 30, 2025

One thing that stands out to me is As AI-driven data workflows increase in scale and become more complex, modern data stack tools such as drag-and-drop ETL solutions are too brittle, expensive, and inefficient for dealing with the higher volume and scale of pipeline and orchestration approaches.

Data Engineer 57

Data Engineer Data Engineering Engineering Pipeline-centric 57

Data Engineering Podcast

MAY 7, 2023

Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it. In this episode Kevin Niparko and Hanhan Wang share the details of how it is implemented and how you can use it to build and maintain rich customer profiles.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage. Pipeline breakpoint feature.

Data Engineer 109

Data Engineer Data Engineering Engineering Database-centric 109

ThoughtSpot

OCTOBER 18, 2024

In the fast-paced world of software development, the efficiency of build processes plays a crucial role in maintaining productivity and code quality. At ThoughtSpot , while Gradle has been effective, the growing complexity of our projects demanded a more sophisticated approach to understanding and optimizing our builds.

Building 52

Building Process Pipeline-centric Database-centric 52

François Nguyen

MARCH 22, 2021

How do we build data products ? You are starting to be an operation or technology centric data team. ” Code : all the code necessary to build a data product (data pipelines, API, policies). TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information? What has been your experience, both positive and negative, in building on top of serverless components?

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Data Engineering Weekly

JUNE 2, 2024

Learn More → AI Verify Foundation: Model AI Governance Framework for Generative AI Several countries are working on building governance rules for Gen AI. The author highlights the structured approach to building data infrastructure, data management, and metrics. TIL that the queryable state is deprecated, which surprises me too.

Data Engineer 103

Data Engineer Data Engineering Engineering Pipeline-centric 103

The Pragmatic Engineer

JANUARY 5, 2023

The first response has been frustration because of the chaos a breach like this causes: At a scaleup I talked with, infrastructure teams shut down all pipelines in order to replace secrets. Our customers are some of the most innovative, engineering-centric businesses on the planet, and helping them do great work will continue to be our focus.”

Pipeline-centric 130

Pipeline-centric Database-centric Coding Accessibility 130

Monte Carlo

JANUARY 21, 2025

Key Themes Data-Driven Decision-Making : Learn how to build a data-centric culture that drives better outcomes. Build Valuable Connections : Network with peers, analysts, and data visionaries who can offer fresh perspectives and collaborations. Build trust across domains : Set and uphold data SLAs.

Data Analytics 52

Data Analytics Pipeline-centric Food Data Lake 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. And don’t forget to thank them for their continued support of this show!

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing. What's your question?

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

The Modern Data Company

DECEMBER 29, 2022

This means moving beyond product-centric thinking to a data-driven customer experience model that’s consistent across all channels. Next, the wealth management industry is also shifting away from a product focus to a client-centric model. Data will enable this industry to shift to scalable solutions and ensure greater customer loyalty.

Banking 52

Banking Pipeline-centric Building Finance 52

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 104

Data Pipeline Data Engineer Data Engineering Engineering 104

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 95

Process Pipeline-centric Media Metadata 95

DataKitchen

AUGUST 27, 2021

DataOps is fundamentally about eliminating errors, reducing cycle time, building trust and increasing agility. The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Analytics Data Lake 98

Confluent

SEPTEMBER 25, 2019

Of course, this is not to imply that companies will become only software (there are still plenty of people in even the most software-centric companies), just that the full scope of the business is captured in an integrated software defined process. Our approach to building this platform is from the bottom up. Confluent’s mission.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

Snowflake

JUNE 28, 2023

Streamlit has rapidly become the de facto way to build UIs for LLM-powered apps. Get smarter about your data with native LLMs Additionally, Snowflake is building LLMs directly into the platform to help customers boost productivity and unlock new insights from their data. And with LLMs, it’s no different. Read this blog.

Pipeline-centric 95

Pipeline-centric Unstructured Data Data Government 95

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud.

Pipeline-centric 89

Pipeline-centric Professional Services BI Datasets 89

Cloudera

APRIL 22, 2021

Cloudera has partnered with Cisco in helping build the Cisco Validated design (CVD) for Apache Ozone. Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Cloudera and Cisco have tested together with dense storage nodes to make this a reality. .

Pipeline-centric 101

Pipeline-centric Data Lake Hadoop Big Data 101

DataKitchen

NOVEMBER 4, 2021

The DataKitchen Platform serves as a process hub that builds temporary analytic databases for daily and weekly ad hoc analytics work. These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. . The DataOps Advantage .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you? Part 1: Why did we need to build our own SIEM?

Data Engineer 67

Data Engineer Data Engineering Engineering Pipeline-centric 67

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Data Engineers are engineers responsible for uncovering trends in data sets and building algorithms and data pipelines to make raw data beneficial for the organization.

Data Engineer 98

Data Engineer Data Engineering Engineering Hadoop 98

The Modern Data Company

JULY 29, 2024

Already operating at capacity, data teams often find themselves repeating efforts, rebuilding similar data pipelines and models for each new project. Yet, data teams find themselves perpetually playing catch-up, struggling to meet immediate demands rather than building scalable, future-proof solutions.

Data Architecture 52

Data Architecture Architecture Building Pipeline-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. Data is simply too centric to the company’s activity to have limitation around what roles can manage its flow.

Data Engineer 130

Data Engineer Data Engineering Engineering ETL Tools 130

Towards Data Science

MAY 19, 2023

Data engineers spend countless hours troubleshooting broken pipelines. Data plays a central role in modern organisations; the centricity here is not just a figure of speech, as data teams often sit between traditional IT and different business functions. Know when to build and when to buy. Data Observability.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Dataquest

JANUARY 25, 2017

This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use. If you’re the type of person that likes building and tweaking systems, data engineering might be right for you. A data pipeline — input data is transformed in a series of phases into output data.

Data Engineer 52

Data Engineer Data Engineering Pipeline-centric Database-centric 52

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. I often wonder if we are building a pyramid infrastructure scheme on top of the object storage.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

DoorDash Engineering

NOVEMBER 28, 2023

DoorDash has been building an internal Machine Learning Workbench over the past year to enhance data operations and assist our data scientists, analysts, and AI/ML engineers. We also shed light on how we drove value by taking a user-centered approach while building this internal tool.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Cloudera

NOVEMBER 15, 2021

Related to the neglect of data quality, it has been observed that much of the efforts in AI have been model-centric, that is, mostly devoted to developing and improving models , given fixed data sets. Data Cascades are said to be pervasive, to lack immediate visibility, but to eventually impact the world in a negative manner. Conclusions.

Unstructured Data 90

Unstructured Data Pipeline-centric Database-centric Entertainment 90

Monte Carlo

MARCH 28, 2024

In a nutshell, DataOps engineers are responsible not only for designing and building data pipelines, but iterating on them via automation and collaboration as well. While a DataOps engineer is primarily focused on ensuring pipelines run smoothly, data engineers are more focused on designing and implementing those pipelines themselves.

Engineering 75

Engineering Pipeline-centric BI Google Cloud 75

U-Next

AUGUST 17, 2022

SQL – A database may be used to build data warehousing, combine it with other technologies, and analyze the data for commercial reasons with the help of strong SQL abilities. Pipeline-centric: Pipeline-centric Data Engineers collaborate with data researchers to maximize the use of the info they gather.

Data Engineer 52

Data Engineer Data Engineering Database-centric Pipeline-centric 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Building the KPay payment system. Pillar 1 – Business function: Payment processing pipeline.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94