Data Engineering Weekly

JULY 28, 2024

Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage. Pipeline breakpoint feature. I like testing people on their practical knowledge rather than artificial coding challenges. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 111

Data Engineering Data Engineer Engineering Database-centric 111

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information? How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information?

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

The Pragmatic Engineer

JANUARY 5, 2023

The first response has been frustration because of the chaos a breach like this causes: At a scaleup I talked with, infrastructure teams shut down all pipelines in order to replace secrets. Our customers are some of the most innovative, engineering-centric businesses on the planet, and helping them do great work will continue to be our focus.”

Pipeline-centric 130

Pipeline-centric Database-centric Coding Accessible 130

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 96

Process Pipeline-centric Media Metadata 96

François Nguyen

MARCH 22, 2021

You are starting to be an operation or technology centric data team. How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum.

Technology 130

Technology Architecture Google Cloud Metadata 130

Cloudera

JANUARY 24, 2023

Next, it needed to enhance the company’s customer-centric approach for a needs-based alignment of products and services. We are positive that our continuing partnership with Cloudera and Blutech Consulting will be foundational to our customer-centric approach, considerably improving our customer responsiveness,” he said.

Banking 86

Banking Data Analytics Pipeline-centric IT 86

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 105

Data Pipeline Data Engineering Data Engineer Engineering 105

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Data Engineering Weekly

JUNE 2, 2024

The resulting solution was SnowPatrol, an OSS app that alerts on anomalous Snowflake usage, powered by ML Airflow pipelines. link] Adevinta: How we moved from local scripts and spreadsheets shared by email to Data Products Data Product Thinking Shaping the data management to build a reliable, customer-centric data application.

Data Engineering 104

Data Engineering Data Engineer Engineering Pipeline-centric 104

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. In the first part he treats about the history of modeling and the main concepts. When it comes to modeling it's hard not to mention dbt.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. In the first part he treats about the history of modeling and the main concepts. When it comes to modeling it's hard not to mention dbt.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Data Engineering Weekly

MARCH 3, 2024

2) Why High-Quality Data Products Beats Complexity in Building LLM Apps - Ananth Packildurai I will walk through the evolution of model-centric to data-centric AI and how data products and DPLM (Data Product Lifecycle Management) systems are vital for an organization's system.

Data Engineering 69

Data Engineering Data Engineer Engineering Pipeline-centric 69

Cloudera

APRIL 22, 2021

Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Seamlessly scale the architecture to thousands of nodes with a single pane of glass management using Cisco Application Centric Infrastructure (ACI).

Pipeline-centric 104

Pipeline-centric Data Lake Hadoop Metadata 104

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud.

Pipeline-centric 93

Pipeline-centric Professional Services BI Datasets 93

Cloudera

NOVEMBER 15, 2021

Related to the neglect of data quality, it has been observed that much of the efforts in AI have been model-centric, that is, mostly devoted to developing and improving models , given fixed data sets. Data Cascades are said to be pervasive, to lack immediate visibility, but to eventually impact the world in a negative manner. Conclusions.

Unstructured Data 96

Unstructured Data Pipeline-centric Database-centric Entertainment 96

Confluent

SEPTEMBER 25, 2019

Of course, this is not to imply that companies will become only software (there are still plenty of people in even the most software-centric companies), just that the full scope of the business is captured in an integrated software defined process. Here, the bank loan business division has essentially become software.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

DataKitchen

AUGUST 27, 2021

The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds. Perhaps more importantly, data engineers and scientists may change any part of the automated pipelines related to data at any time. That’s the power of DataOps automation.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Analytics Data Lake 98

Confluent

JUNE 18, 2019

It involves many moving parts, from data preparation to building indexing and query pipelines. Building an indexing pipeline at scale with Kafka Connect. Always keep an eye on their performance and make sure they run in the expected time to allow your pipeline to function properly. Scaling indexing. Interested in more?

Architecture 97

Architecture Building Kafka Database-centric 97

Monte Carlo

MARCH 28, 2024

In a nutshell, DataOps engineers are responsible not only for designing and building data pipelines, but iterating on them via automation and collaboration as well. While a DataOps engineer is primarily focused on ensuring pipelines run smoothly, data engineers are more focused on designing and implementing those pipelines themselves.

Engineering 75

Engineering Pipeline-centric BI Google Cloud 75

Dataquest

JANUARY 25, 2017

This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use. Roughly, the operations in a data pipeline consist of the following phases: Ingestion — this involves gathering in the needed data. A data scientist is only as good as the data they have access to.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Christophe Blefari

DECEMBER 19, 2022

But this article is not about the pricing which can be very subjective depending on the context—what is 1200$ for dev tooling when you pay them more than $150k per year, yes it's US-centric but relevant. But before sending your code to production you still want to validate some stuff, static or not, in the CI/CD pipelines.

Management 130

Management Pipeline-centric Database-centric SQL 130

DataKitchen

NOVEMBER 4, 2021

These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. . The data pipelines must contend with a high level of complexity – over seventy data sources and various cadences, including daily/weekly updates and builds.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Towards Data Science

MAY 19, 2023

Data engineers spend countless hours troubleshooting broken pipelines. Data plays a central role in modern organisations; the centricity here is not just a figure of speech, as data teams often sit between traditional IT and different business functions. Every “minor” change upstream results in mayhem. trillion to the U.S.

Pipeline-centric 104

Pipeline-centric Data Software Engineering Software Engineer 104

Snowflake

DECEMBER 19, 2023

Sometimes they need feedback on touchpoints very quickly, while other pipelines don’t need as much acceleration. Acadia, a digital media agency, wanted to accelerate end-to-end pipeline for its clients while also enhancing security for clients’ PII. One conversation quickly coming to the forefront is first-party data.

Building 86

Building Pipeline-centric Database-centric Digital Media 86

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. The author writes about integrating uwheel with DataFusion (a query engine for building high-quality data-centric systems in Rust , using the Apache Arrow in-memory format).

Data Engineering 72

Data Engineering Data Engineer Engineering Database-centric 72

U-Next

AUGUST 17, 2022

Pipeline-centric: Pipeline-centric Data Engineers collaborate with data researchers to maximize the use of the info they gather. Data Engineers focused on pipelines require a solid understanding of decentralized technology and computer engineering. They are frequently found in midsize businesses.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Data is simply too centric to the company’s activity to have limitation around what roles can manage its flow. The modern data warehouse is a more public institution than it was historically, welcoming data scientists, analysts, and software engineers to partake in its construction and operation.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

DataKitchen

NOVEMBER 5, 2024

We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric. The need to copy data across layers, manage different schemas, and address data latency issues can complicate data pipelines. However, this architecture is not without its challenges.

Architecture 81

Architecture Raw Data Pipeline-centric Data Ingestion 81

DoorDash Engineering

NOVEMBER 28, 2023

I frequently check Pipeline Runs and Sensor Ticks, but, often verify with Dagit.” As we spoke to users, we realized that this also was an opportunity for us to observe what DoorDash’s ML pipeline looked like. Would be really helpful to have a dropdown box as we’re typing feature search keywords (contextual search).”

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Snowflake

JUNE 28, 2023

Since it’s all part of Snowflake’s single platform, data engineers and developers can also perform inference by programmatically calling the built-in or fine-tuned models, like in pipelines with Streams and Tasks or in applications. Learn More: Learn more about how Snowflake is building a data-centric platform for generative AI and LLM.

Pipeline-centric 77

Pipeline-centric Unstructured Data Data Government 77

Ascend.io

MAY 7, 2024

” Key Partnership Benefits: Cost Optimization and Efficiency : The collaboration is poised to reduce IT and data management costs significantly, including an up to 68% reduction in data stack spend and the ability to build data pipelines 7.5x Discover how Wizeline’s AI-centric approach can revolutionize your business at wizeline.com.

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Knowledge Hut

APRIL 25, 2024

Kubernetes is a container-centric management software that allows the creation and deployment of containerized applications with ease. To check if the Pod is running in the Kubernetes cluster: Command: kubectl get pods --watch --show-labels Learn more about DevOps and writing pipelines with the Best DevOps Courses Online.

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Monte Carlo

MAY 30, 2024

It can involve prompt engineering, vector databases like Pinecone , embedding vectors and semantic layers, data modeling, data orchestration, and data pipelines – all tailored for RAG. Like RAG, fine tuning requires building effective data pipelines that can make proprietary data available to the fine tuning process in the first place.

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team.

Data Engineering 64

Data Engineering Data Engineer Engineering Pipeline-centric 64

Lyft Engineering

APRIL 3, 2023

Specifically, Lyft’s in-house distributed hyperparameter optimization pipeline is used for the majority of its business critical models. That said, in 2020, Lyft moved towards a more user centric approach — preselecting a user’s most frequently used mode. Screenshots are illustrative. May not capture the current experience.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87