Data Engineering Weekly

JUNE 2, 2024

Learn More → AI Verify Foundation: Model AI Governance Framework for Generative AI Several countries are working on building governance rules for Gen AI. The author highlights the structured approach to building data infrastructure, data management, and metrics. TIL that the queryable state is deprecated, which surprises me too.

Data Engineering 103

Data Engineering Data Engineer Engineering Pipeline-centric 103

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process.

Process 93

Process Pipeline-centric Media Metadata 93

Confluent

JUNE 18, 2019

Software projects of all sizes and complexities have a common challenge: building a scalable solution for search. Building a resilient and scalable solution is not always easy. It involves many moving parts, from data preparation to building indexing and query pipelines. You might be wondering, is this a good solution?

Architecture 97

Architecture Building Kafka Database-centric 97

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. The post Cloudera Customer Story appeared first on Cloudera Blog. Please read the full story here.

Pipeline-centric 87

Pipeline-centric Professional Services Datasets BI 87

RandomTrees

SEPTEMBER 27, 2024

With One Lake serving as a primary multi-cloud repository, Fabric is designed with an open, lake-centric architecture. You can use Copilot to build reports, summarize insights, build pipelines, and develop ML models. What’s new ever since Fabric’s GA announcement in November 2023? This preview will roll out in stages.

Database-centric 52

Database-centric Pipeline-centric IT BI 52

Cloudera

SEPTEMBER 11, 2024

Pharmaceuticals : Pharmaceutical companies need to integrate sensitive data, including Personally Identifiable Information (PII), across the R&D pipeline to accelerate the development of life-saving medications. Healthcare : Healthcare providers are under pressure to leverage data to deliver patient centricity and a continuum of care.

Pharmaceutical 52

Pharmaceutical Cloud Pipeline-centric Healthcare 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DoorDash Engineering

NOVEMBER 28, 2023

It is amusing for a human being to write an article about artificial intelligence in a time when AI systems, powered by machine learning (ML), are generating their own blog posts. We also shed light on how we drove value by taking a user-centered approach while building this internal tool.

Machine Learning 90

Machine Learning Pipeline-centric Data Science Designing 90

Meltano

JANUARY 25, 2023

A curated list of the top 9 must read blogs on data. At the end of 2022 we decided to collect the blogs we enjoyed the most over the year. For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

Pipeline-centric 52

Pipeline-centric Database-centric SQL Data Warehouse 52

Snowflake

JUNE 28, 2023

Streamlit has rapidly become the de facto way to build UIs for LLM-powered apps. Get smarter about your data with native LLMs Additionally, Snowflake is building LLMs directly into the platform to help customers boost productivity and unlock new insights from their data. Read this blog. And with LLMs, it’s no different.

Pipeline-centric 75

Pipeline-centric Unstructured Data Data Government 75

Towards Data Science

MAY 19, 2023

Data engineers spend countless hours troubleshooting broken pipelines. Data plays a central role in modern organisations; the centricity here is not just a figure of speech, as data teams often sit between traditional IT and different business functions. More can be found in this blog. Know when to build and when to buy.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Snowflake

JUNE 28, 2023

It provides familiar APIs for various data centric tasks, including data preparation, cleansing, preprocessing, model training, and deployments tasks. In the warehouse model, users can seamlessly run and operationalize data pipelines, ML models, and data applications with user-defined functions (UDFs) and stored procedures (sprocs).

Python 52

Python Accessible Accessibility Pipeline-centric 52

U-Next

AUGUST 17, 2022

SQL – A database may be used to build data warehousing, combine it with other technologies, and analyze the data for commercial reasons with the help of strong SQL abilities. Pipeline-centric: Pipeline-centric Data Engineers collaborate with data researchers to maximize the use of the info they gather.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Lyft Engineering

APRIL 3, 2023

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders. This blog mostly focuses on the mode selector to explain how rankings have evolved in the past years and briefly touches on the post request cross-sells.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87

DataKitchen

NOVEMBER 5, 2024

We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric. The need to copy data across layers, manage different schemas, and address data latency issues can complicate data pipelines. However, this architecture is not without its challenges.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Data Engineering Weekly

APRIL 2, 2023

link] Tweet Search System (EarlyBird) Design [link] Google AI: Data-centric ML benchmarking - Announcing DataPerf’s 2023 challenges Data is the new code: it is the training data that determines the maximum possible quality of an ML solution. I echoed a similar statement here. Seriously, come on, Redshift team!!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

AUGUST 27, 2021

DataOps is fundamentally about eliminating errors, reducing cycle time, building trust and increasing agility. The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Lake Data Analytics 98

Edureka

JULY 14, 2024

In this blog, we’ll discuss DevOps release management, its process, best practices, and the advantages of release manager in Devops. It encompasses the planning, scheduling, and controlling of software builds and delivery pipelines. This is where DevOps Release Management comes in. What is Release Management?

Management 40

Management Pipeline-centric Coding Software Engineer 40

Knowledge Hut

SEPTEMBER 24, 2024

In this blog post, we will see the top Automation testing tools used in the software industry. Ranorex Ranorex is a set of paid and free tools to build sophisticated tests for desktop, web, and mobile applications. We have so many Automation tools available in the market to perform automation testing. Support integration with CICD.

Java 98

Java Programming Language Pipeline-centric Database-centric 98

Workfall

AUGUST 29, 2023

Learn how to build your very own Amazon Price Tracker using Python. In this blog, we will cover: About Python About Amazon About BeautifulSoup Library Hands-On Conclusion About Python Python is a high-level, interpreted, and versatile programming language known for its simplicity and readability. appeared first on The Workfall Blog.

Python 93

Python Pipeline-centric Programming Language Coding 93

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. DoorDash writes about its journey to build a Metric Layer for experimentation.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

NOVEMBER 15, 2021

This blog discusses quantifications, types, and implications of data. The activity in the field of learning with limited data is reflected in a variety of courses , workshops , reports , blogs and a large number of academic papers (a curated list of which can be found here ). Quantifications of data. Addressing the challenges of data.

Unstructured Data 88

Unstructured Data Pipeline-centric Database-centric Entertainment 88

Cloudera

APRIL 22, 2021

Cloudera has partnered with Cisco in helping build the Cisco Validated design (CVD) for Apache Ozone. Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Cloudera will publish separate blog posts with results of performance benchmarks.

Pipeline-centric 100

Pipeline-centric Data Lake Hadoop Metadata 100

DataKitchen

FEBRUARY 21, 2023

As a result, a less senior team member was made responsible for modifying a production pipeline. Build analytic data systems that have modular, reusable components. Build components that are idempotent on data. To add fuel to the fire, a few weeks ago, one of the most talented people in the group left.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

OCTOBER 17, 2023

This blog is for anyone who was interested but unable to attend the conference, or anyone interested in a quick summary of what happened there. Use cases such as fraud monitoring, real-time supply chain insight, IoT-enabled fleet operations, real-time customer intent, and modernizing analytics pipelines are driving development activity.

Kafka 78

Kafka Database-centric Pipeline-centric Database 78

Snowflake

AUGUST 21, 2023

In this blog post, we’ll review the core data mesh principles, highlight how both organizations and modern data platforms are putting those principles into action, and demonstrate just how achievable a secure and efficient data mesh architecture can be. It’s also crucial that teams consider the price tag of the data mesh.

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Data Pipeline 52

Data Pipeline Healthcare Medical Pipeline-centric 52

Picnic Engineering

DECEMBER 19, 2023

In this blog, we’d like to give you a glimpse into some of the major developments in Picnic Tech in 2023. Combining efficient incident handling, establishing resilience by design, and strict adherence to SLOs are pivotal in ensuring our services remain resilient, reliable, stable, and user-centric. Join us and have a read! The impact?

Transportation 52

Transportation Pipeline-centric Database-centric Python 52

DataKitchen

NOVEMBER 4, 2021

The DataKitchen Platform serves as a process hub that builds temporary analytic databases for daily and weekly ad hoc analytics work. These limited-term databases can be generated as needed from automated recipes (orchestrated pipelines and qualification tests) stored and managed within the process hub. . The DataOps Advantage .

Process 98

Process Data Process Pharmaceutical Data Lake 98

Monte Carlo

JULY 28, 2022

These are particularly frustrating, because while they are breaking data pipelines constantly, it’s not their fault. As Convoy Head of Product, Data Platform, Chad Sanderson wrote in our blog , “The data in production tables are not intended for analytics or machine learning. He suggested : “Private vs. public methods.

Pipeline-centric 52

Pipeline-centric Software Engineer Software Engineering Database-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Databand.ai

DECEMBER 29, 2022

Chad writes on data management, contracts, and products on his Substack blog and serves as an advisor and investor to several startups. He recently joined Databand’s MAD Data Podcast to talk about how his team is building one of the most advanced experimentation and machine learning platforms in the world from the ground up.

BI 52

BI Consulting Data Science Data Governance 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. Elasticsearch has become ubiquitous as an index centric datastore for search and rose in tandem with the popularity of the internet and Web2.0.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

Knowledge Hut

MARCH 22, 2024

In this blog post, I will guide you through crafting a compelling response to "Why should we hire you," provide examples, discuss “Why should we hire you answers,” and highlight common pitfalls to avoid in the process.

Recruitment 52

Recruitment Healthcare Pipeline-centric Hospitality 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Building the KPay payment system. Pillar 1 – Business function: Payment processing pipeline.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94