Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress.org – which has a complex, intertwined setup with Automattic, and was also cofounded by Matt Mullenweg –  bans  WP Engine from accessing its plugin repository and updates infrastructure. 25 Sep: Block. This penalizes WP Engine customers. 27 Sep: Unblock.  Wordpress.org 

Consulting 305

Consulting AWS Engineering Software Engineer 305

Rockset

JUNE 21, 2024

We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI. With this acquisition, what we’ve developed over the years will help make AI accessible to all in a safe and beneficial way. Read more: OpenAI’s blog post FAQs for Rockset customers

Database 145

Database Cloud Accessibility Accessible 145

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 66

Building Metadata Software Engineer Software Engineering 66

Workfall

DECEMBER 19, 2023

For Part 1 of this blog, refer here. Whether delving into team-building activities with the “Team Builder Guide” or seeking cinematic recommendations through the “Movie Recommendation” app, PartyRock delivers an expansive array of possibilities. Navigate to the below given URL to access the PartyRock site.

Building 81

Building AWS Accessibility Accessible 81

Snowflake

DECEMBER 11, 2023

While digital advertisers and paid media professionals are on the hook to build ample sales pipeline and maximize return on ad spend (ROAS), they’re also expected to deliver personalized advertising content while navigating evolving privacy requirements and adhering to consumer expectations—all while extracting insights from siloed ad platforms.

Building 92

Building Cloud Media Algorithm 92

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Go to dataengineeringpodcast.com/dagster today to get started.

Database 162

Database Technology Data Lake High Quality Data 162

LinkedIn Engineering

DECEMBER 20, 2023

At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships. In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform.

Building 91

Building Algorithm Kafka Machine Learning 91

The Pragmatic Engineer

JANUARY 12, 2023

Source: The Pragmatic Engineer blog. I suspect the company was able to attract good tech talent despite not overpaying in markets like London or Warsaw, thanks to supporting remote working, possessing a good brand, and being focused on building a diverse team – at which it did a good job. Tier 1: hyperlocal compensation.

Software Engineer 187

Software Engineer Software Engineering Engineering Raw Data 187

Snowflake

FEBRUARY 21, 2023

Welcome to Snowflake’s Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. I’ve worked on building API platforms throughout my career. Unlocking the creativity of developers to build with data. Propel is an API platform for developers to build web and mobile apps with their Snowflake data.

AWS 123

AWS Building Raw Data Architecture 123

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Your host is Tobias Macey and today I'm interviewing Oren Eini about the work of designing and building a NoSQL database engine Interview Introduction How did you get involved in the area of data management? Your first 30 days are free!

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Engineering at Meta

OCTOBER 23, 2023

To optimize build and performance, we developed our own build system called Buck , which was first open-sourced in 2013. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Meta has a very large monorepo, with many different programming languages.

Programming Language 117

Programming Language Java Python Programming 117

Lyft Engineering

APRIL 24, 2024

We want to foster a culture of continuous learning by providing resources and forums that are easily accessible to everyone. See our previous blog post for more on the motivations behind TLC. Some recent examples can be found in our tech blog: reinforcement learning and structural causal modeling.

Data Science 68

Data Science Building Computer Science Software Engineer 68

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Tweag

MARCH 6, 2024

Three years ago, a blog post introduced destination-passing style (DPS) programming in Haskell, focusing on array processing, for which the API was made safe thanks to Linear Haskell. The present blog post is mostly based on my recent paper Destination-passing style programming: a Haskell implementation , published at JFLA 2024.

Programming 126

Programming Data Programming Language Coding 126

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages.

Project 144

Project Programming Language Building Systems 144

Tweag

NOVEMBER 20, 2024

Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. will require special care to build both static and dynamic flavours of those libraries.

Coding 137

Coding Programming Project Process 137

Snowflake

APRIL 24, 2024

Building top-tier enterprise-grade intelligence using LLMs has traditionally been prohibitively expensive and resource-hungry, and often costs tens to hundreds of millions of dollars. license provides ungated access to weights and code. We will be releasing corresponding Medium.com blog posts daily over the next month.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Data Engineering Weekly

JULY 21, 2024

The author highlights key lessons learning building an analytical org from scratch with key highlights, 1. Build reliable data, maintain usable data models, and ensure the data is interpreted correctly for decision-making. The blog is an excellent summary of what one needs to know about Gen-AI to start.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Data Engineering Weekly

JUNE 24, 2024

link] Sponsored: 2024 State of Apache Airflow Report Gain access to the latest trends and insights shaping the world of Apache Airflow—the go-to platform for data pipeline development and orchestration. The blog contains a summary of each talk and a link to the YouTube channel with all the talks.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Snowflake

AUGUST 5, 2024

Welcome to Snowflake’s Startup Spotlight, where we learn about companies building their businesses on Snowflake. These moments are constant reminders of why we do what we do — they fuel our motivation to keep building “Big Things!” We unlock the value of geospatial data and make it accessible to your entire organization.

Business Intelligence 110

Business Intelligence Technology Accessibility Accessible 110

Snowflake

DECEMBER 20, 2023

Accessing the necessary resources from cloud providers demands careful planning and up to month-long wait times due to the high demand for GPUs. You can read more about their experience with Snowpark Container Services in this two-part blog series ( part 1 , part 2 ). What makes Snowpark Container Services unique?

Scala 119

Scala Government Java Cloud 119

Snowflake

MAY 29, 2024

Step 3: Commercial Models to Unlock Revenue at Scale Modeling the business value comes down to two things: (1) What data are you making accessible; and (2) How many partners are you willing (and able) to engage? Once you have a plan on paper, you can begin sizing the data clean room opportunity for your business.

Retail 89

Retail Media Accessibility Accessible 89

Cloudera

JUNE 2, 2022

Over the last two years, the Cloudera DataFlow team has been hard at work building Cloudera DataFlow for the Public Cloud (CDF-PC). This blog aims to answer two questions: What is a universal data distribution service? .” CDF-PC is a cloud native universal data distribution service powered by Apache NiFi on Kubernetes, ??allowing

Systems 105

Systems Data Lake Google Cloud Cloud 105

Zalando Engineering

NOVEMBER 8, 2023

Introduction This blog post describes a recent contribution from Zalando to the Postgres JDBC driver to address a long-standing issue with the driver’s integration with Postgres’ logical replication that resulted in runaway Write-Ahead Log (WAL) growth. However as you may imagine, this blog post concerns a path that is anything but happy.

PostgreSQL 138

PostgreSQL Java Database Bytes 138

Cloudera

AUGUST 30, 2022

This blog lays out some steps to help you incrementally advance efforts to be a more data-driven, customer-centric organization. Cloudera refers to this as universal data distribution, as explored further in this blog post. Data-fuelled innovation requires a pragmatic strategy. Embrace incremental progress.

Cloud Storage 108

Cloud Storage Government Data Governance Retail 108

Christophe Blefari

APRIL 5, 2024

On my side I'll talk about Apache Superset and what you can do to build a complete application with it. How we built Text-to-SQL at Pinterest — Pinterest open-sourced a tool called Querybook that they used to access Pinterest data every day. This week they released a new open model that performs great.

SQL 130

SQL Data Python Datasets 130

Rockset

MARCH 1, 2023

But, these two functions directly compete for the available compute resources, creating a fundamental limitation that makes it difficult to build efficient, reliable real-time applications at scale. Elasticsearch, Clickhouse, Apache Druid and Apache Pinot are the databases commonly used for building real-time applications.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

Robinhood

OCTOBER 21, 2024

Today, we’re excited to build upon that effort for customers in the UK by announcing the launch of margin investing, with some of the most competitive rates in the industry. Historically, UK retail investors have had challenges accessing features like margin investing that provide other ways to help maximise their investments.

Portfolio 95

Portfolio Insurance Retail Banking 95

Pinterest Engineering

FEBRUARY 22, 2024

In this blog post, we share our journey of unlocking AI-assisted development, from the initial idea to the General Availability (GA) stage. The first decision we had to make was build versus buy. To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site.

Scala 96

Scala Engineering Software Engineer Software Engineering 96

Tweag

OCTOBER 25, 2023

The Nix language allows you to define the “recipe” to build anything into a package, like the sources and the steps to make the package, but also the dependencies it needs to build. Behind the zlib variable we use here also lies a derivation, another recipe to build said library. Is it a bird? It’s a graph!

Metadata 114

Metadata Accessibility Accessible Python 114

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. To briefly recap, here are a few of the things that we took away from this journey: Accurately simulating real traffic helps build confidence in new systems and algorithms more quickly.

Algorithm 139

Algorithm Metadata Kafka Systems 139

Snowflake

JUNE 3, 2024

Polaris Catalog builds on standards created by the Iceberg community to address the challenges described above. The remainder of this blog post provides more detail on functionality and hosting options. You can host it in Snowflake managed infrastructure or your infrastructure of choice.

Amazon Web Services 126

Amazon Web Services Google Cloud Data Architect Government 126

Data Engineering Weekly

JUNE 16, 2024

One of the big benefits of having a Hive Meastore catalog is that it enables many query engines to build executing engines on top of it, which creates a strong ecosystem. Along the same line, NVIDIA open sources nmotran4 give developers a free, scalable way to generate synthetic data that can help build powerful LLMs.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Netflix Tech

NOVEMBER 9, 2020

Studio Engineering is building hundreds of applications and tools that power these workflows. Due to the loose coupling, we observed that many teams spent considerable effort building duplicative data-fetching code and aggregation layers to support their product needs. 2) Materialized views for data from other teams?—?some

IT 144

IT Architecture Engineering Building 144

Data Engineering Weekly

AUGUST 6, 2024

CDC enables real-time data flows from operational systems to analytics platforms, ensuring decision-makers can access the most current data. By capturing and integrating data changes from diverse sources, CDC helps build and maintain the context-rich environments that modern AI applications demand.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125