phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. 2019 - Delta Lake Databricks released Delta Lake as an open-source project.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Engineering at Meta

APRIL 6, 2023

Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In particular, we support Sapling-based file systems. Why rebuild Buck?

Building 144

Building Systems Java Coding 144

ArcGIS

NOVEMBER 30, 2023

Learn how to filter coordinate systems based on a spatial extent, GCS, or projection property.

Project 129

Project Systems 129

dbt Developer Hub

APRIL 20, 2025

In particular, we expect both Business Intelligence and Data Engineering will be driven by AI operating on top of the context defined in your dbt Projects. Both AI agents and business stakeholders will then operate on top of LLM-driven systems hydrated by the dbt MCP context. Why does this matter? MCP addresses this challenge.

Structured Data 98

Structured Data SQL BI Project 98

The Pragmatic Engineer

MARCH 12, 2024

The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. Two secretaries.

Engineering 329

Engineering Programming Language Hospitality Programming 329

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Datafold has invested a lot of time into integrating with the workflow of dbt projects to add early verification that the changes you are making are correct. What are the parallels to that in data projects?

Project 130

Project Building Data Lake Machine Learning 130

Knowledge Hut

MAY 23, 2024

You can find the online PMP exam application on the Project Management Institute (PMI)® website. Check Project Management professional preparation course to get started with your PMP preparation. Work Experience Your project management expertise is questioned in the next area of the online form.

Project 98

Project Certification Education Telecommunication 98

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Can you start by outlining what elements of observability are most relevant for dbt projects? Over the past ~3 years there were numerous data observability systems/products created. How is Elementary designed/implemented?

Project 130

Project Data Lake High Quality Data Data Workflow 130

The Pragmatic Engineer

SEPTEMBER 22, 2023

It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. Bun has no such constraint. If you are one of these people: thank you!

Programming Language 316

Programming Language Project Coding Engineering 316

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. OpenAI’s impossible business projections.  According This event is shameful and unprecedented in the history of open source on the web.

Engineering 357

Engineering Government Project AWS 357

Analytics Vidhya

FEBRUARY 3, 2023

Say Harish and Lisa are two people working on the same project but on two different systems(say windows and […] The post Getting Started with The Basics of Docker appeared first on Analytics Vidhya. ” If you read the above quote, you must think, what does this all mean? Well, my friend, this is what Docker is.

Coding 264

Coding Project Systems IT 264

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. Facebook created Cassandra, which ultimately became an Apache Software Foundation project. It is well-known for its rapid write […] The post Top 6 Cassandra Interview Questions appeared first on Analytics Vidhya.

NoSQL 259

NoSQL Database Project Systems 259

Netflix Tech

DECEMBER 19, 2024

Many of these projects are under constant development by dedicated teams with their own business goals and development best practices, such as the system that supports our content decision makers , or the system that ranks which language subtitles are most valuable for a specific piece ofcontent.

Machine Learning 81

Machine Learning Project Data Warehouse Coding 81

The Pragmatic Engineer

OCTOBER 17, 2024

We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Christophe Blefari

JANUARY 11, 2025

There are multiple ways to start a new year, either with new projects, new ideas, new resolutions or by just keeping doing the same music. AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. I hope you will enjoy 2025.

Data 130

Data Data Warehouse Coding Programming Language 130

Start Data Engineering

MAY 28, 2024

Project demo 3. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Introduction 2. Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3.

Data Pipeline 219

Data Pipeline Python Building Data 219

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. The SDE3 level expects leadership on projects in which this engineer is involved. It’s not a checklist, but some expectations that could be considered: Lead a complex project.

Software Engineering 246

Software Engineering Software Engineer Engineering Designing 246

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! As a response, cloud providers started the Valkey project, which could become the “new and still permissive Redis.”

Consulting 347

Consulting AWS Engineering Software Engineering 347

The Pragmatic Engineer

AUGUST 22, 2023

Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process.

Software Engineering 235

Software Engineering Software Engineer Consulting Entertainment 235

Snowflake

DECEMBER 4, 2024

Beyond working with well-structured data in a data warehouse, modern AI systems can use deep learning and natural language processing to work effectively with unstructured and semi-structured data in data lakes and lakehouses. AI projects should not be about “the latest” or “the best.” Leadership will be the antidote to AI exhaustion.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 271

Banking Utilities Database Engineering 271

Snowflake

JANUARY 28, 2025

Thats why we are announcing that SnowConvert , Snowflakes high-fidelity code conversion solution to accelerate data warehouse migration projects, is now available for download for prospects, customers and partners free of charge. And today, we are announcing expanded support for code conversions from Amazon Redshift to Snowflake.

Data Warehouse 92

Data Warehouse Professional Services SQL Coding 92

DataKitchen

DECEMBER 9, 2024

The project showed that smaller, empowered teams achieve higher impact than larger ones. These small, cross-functional teams ensured that members were deeply involved in the project operations, the technical setup, and the feedback cycle, leading to fewer delays, fewer bottlenecks, and faster decision-making.

Pharmaceutical 64

Pharmaceutical Data Lake Cloud Storage Project 64

KDnuggets

JANUARY 22, 2025

Are you looking for an easier way to manage files across different storage systems? fsspec is a Python library that simplifies file handling by providing a unified interface for file management.

Python 104

Python Management Project Systems 104

Yelp Engineering

MARCH 25, 2024

We present a high level overview of our in-house phone masking system and dive into the details of the engineering challenge of optimizing the usage of proxy phone number resources at Yelp’s scale. Background Every year, millions of requests for quotes, consultations or other messages are sent to businesses on Yelp.

Project 103

Project Consulting Engineering Systems 103

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? If you've learned something or tried out a project from the show then tell us about it!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. So we had a quarterly planning process to ensure all project dependencies were incorporated into each team’s roadmap. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineering 314

Software Engineering Software Engineer Engineering Building 314

Data Engineering Podcast

OCTOBER 29, 2023

Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data projects are notoriously complex. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects.

Database 189

Database SQL BI Machine Learning 189

Snowflake

JANUARY 28, 2025

Thats why we are announcing that SnowConvert , Snowflakes high-fidelity code conversion solution to accelerate data warehouse migration projects, is now available for download for prospects, customers and partners free of charge. And today, we are announcing expanded support for code conversions from Amazon Redshift to Snowflake.

Data Warehouse 85

Data Warehouse Professional Services SQL Data 85

The Pragmatic Engineer

OCTOBER 10, 2023

They called it Office 365, and in 2010, this was a really exciting project to work on. I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked.

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

Snowflake

MARCH 6, 2025

Document RAG preparation : Ingesting, cleaning and chunking documents before embedding them into vector representations, enabling efficient retrieval and enhanced LLM responses in retrieval-augmented generation (RAG) systems. An efficient batch processing system scales in a cost-effective manner to handle growing volumes of unstructured data.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

The Pragmatic Engineer

AUGUST 8, 2024

” A few days ago on 27 February, Klarna shared progress, a month after the project went live. With clever-enough probing, this system prompt can be revealed. ” What is the system prompt for Klarna’s bot? ’ The two companies have worked together ever since.” Translate to English if needed.

IT 253

IT Software Engineering Software Engineer Systems 253

Cloudera

NOVEMBER 13, 2024

Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Metadata 90

Metadata Management Data Governance Government 90

The Pragmatic Engineer

APRIL 19, 2023

I still remember being in a meeting where a Very Respected Engineer was explaining how they are building a project, and they said something along the lines of "and, of course, idempotency is non-negotiable." I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager.

Software Engineering 343

Software Engineering Software Engineer Engineering Media 343

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. Replay traffic enabled us to test our new systems and algorithms at scale before launch, while also making the traffic as realistic as possible.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Cloudera

DECEMBER 4, 2024

System metrics, such as inference latency and throughput, are available as Prometheus metrics. Users can manage all of their models and applications on the Cloudera AI Inference service with common CI/CD systems using Cloudera service accounts, also known as machine users.

Architecture 73

Architecture Machine Learning BI Deep Learning 73