Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses.

IT 147

IT Data Lake Metadata Data Warehouse 147

The Pragmatic Engineer

JUNE 1, 2023

Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js See it in action, here : A screenshot of the interactive “Rides” app. Here’s how Juraj approached the undertaking: Phase 0: make a plan (Oct).

Education 363

Education Project PostgreSQL Software Engineering 363

Data Engineering Podcast

MAY 18, 2024

In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Code Comments Podcast Logo]([link] Putting new technology to use is an exciting prospect. Closing Announcements Thank you for listening!

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Cloudera

NOVEMBER 1, 2024

It also provided a moment for us to launch an important initiative for Cloudera: our Women Leaders in Technology (WLIT) initiative. Building Inclusive Data-Driven Organizations: Leadership Strategies for the Modern Workplace As it stands, women currently account for approximately 25% of the technology workforce.

Technology 95

Technology Recruitment Banking AWS 95

The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry.

Engineering 301

Engineering Bytes Cloud Computing AWS 301

Christophe Blefari

MARCH 15, 2024

AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora. AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Snowflake

AUGUST 1, 2024

technology seamlessly integrates retrieval and generation to deliver unmatched accuracy for RAG workloads without any additional, error-prone system tuning. This helps ensure that AI-powered applications provide responses that are not only accurate, relevant, and consistent, but also aligned with business needs.

Cloud 103

Cloud IT Technology Government 103

The Pragmatic Engineer

APRIL 21, 2024

Sometimes the best explanations of how a technology solution works come from the software engineers who built it. See a longer version of this article here: Scaling ChatGPT: Five Real-World Engineering Challenges. To explain how ChatGPT (and other large language models) operate, I turned to the ChatGPT engineering team. "How

Engineering 351

Engineering Software Engineering Software Engineer Programming 351

Precisely

DECEMBER 16, 2024

As technology continues its rapid ongoing evolution, IT environments have become increasingly complex which leaves businesses needing to adapt at unprecedented speeds. Key Takeaways: Centralized visibility of data is key. Predictive of AIOps capabilities will revolutionize IT operations.

Systems 59

Systems IT Machine Learning Insurance 59

Christophe Blefari

MARCH 1, 2023

First let's understand why dbt exists. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. With the public clouds—e.g. It was the previous tag line dbt Labs had on their website.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 18, 2024

Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Analytics Vidhya

MARCH 5, 2023

HDInsight works seamlessly with the Hadoop ecosystem, which includes technologies like MapReduce, Hive, […] The post Top 6 Microsoft HDFS Interview Questions appeared first on Analytics Vidhya. A distributed file system runs on commodity hardware and manages massive data collections.

Hadoop 246

Hadoop Data Collection Cloud Technology 246

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Bun has other contributors, but Jared writes the lion’s share of code. Many developers are giving Bun a go.

Programming Language 254

Programming Language Project Coding Engineering 254

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When Andrew Lee started work on Shortwave he was focused on making email more productive.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Machine Learning Data 208

Jesse Anderson

OCTOBER 5, 2023

Data quality problems are solved by more than just technology. Data products need to be exposed with the right technologies for the various use cases. The technologies speak Kafka’s binary protocol, removing the need for any client-side changes. A few examples: Everything should use Kafka – No, it shouldn’t.

Kafka 195

Kafka Finance Cloud Designing 195

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” Clearly, this would generate a handsome return for investors and founders.

Software Engineering 275

Software Engineering Software Engineer Programming Language Media 275

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. The internet has been speculating the past few days on which crypto company spent $65M on Datadog in 2022. I confirmed it was Coinbase, and here are the details of what happened. Originally published on 11 May 2023. Can you possibly shed a little more light?“

AWS 318

AWS Software Engineering Software Engineer Google Cloud 318

Analytics Vidhya

MARCH 14, 2023

Introduction In today’s world, technology has increased tremendously, and many people are using the internet. This results in the generation of so much data daily. This generated data is stored in the database and will maintain it. SQL is a structured query language used to read and write these databases.

SQL 202

SQL Database Technology Data 202

The Pragmatic Engineer

APRIL 19, 2023

When I think back on the software engineers I looked up to, they all shared this trait where they never took anything at face value. They regularly questioned statements that did not make sense to them, no matter how small the topic was: even if it involved admitting they did not understand a concept. This person explained it. Or does it?

Software Engineering 294

Software Engineering Software Engineer Engineering Media 294

Data Engineering Podcast

JUNE 25, 2023

Contact Info tobymao on GitHub @captaintobs on Twitter Website Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

The Pragmatic Engineer

OCTOBER 10, 2023

We don’t frequently have CEOs on The Pragmatic Engineer: in fact, today is the first issue when we’re talking with a CEO – partially – about their CEO job. I’ve always wondered what it would be like to go from a developer, to eventually become the CEO of a large and growing company.

Software Engineering 268

Software Engineering Software Engineer Architecture Media 268

The Pragmatic Engineer

AUGUST 1, 2023

Senior engineers are expected to tackle problems where the business problem (or customer case) is well defined, but the technology strategy is not. Wise Wise – formerly Transferwise – is a publicly traded Fintech, valued at $6.5B at time of publication. The company employs more than 5,000 people in 17 countries. A lead role.

Software Engineering 245

Software Engineering Software Engineer Engineering Designing 245

The Pragmatic Engineer

MAY 25, 2023

In this issue, we cover one out of six topics from today’s subscriber-only The Scoop issue. To get full articles twice a week, subscribe here. I got a message from a software engineer working at a company which laid off 30% of staff in December 2022. Also, there is business sense in doing this for reputational reasons.

Software Engineering 315

Software Engineering Software Engineer AWS Engineering 315

Data Engineering Podcast

MARCH 24, 2024

Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

The Pragmatic Engineer

JULY 20, 2023

Startups that are running out of money, and are unable to raise more capital, are having to forcibly shut down. Of course, it’s not just grim news: startup funding might be harder to come by, but it’s not impossible to get. And it’s not just startups shutting down: there are plenty of startups being acquired. How did you hire?

Building 207

Building Programming Language Programming Python 207

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. With this, it’s over to Lou. Use tech debt payments to get into the flow and stay in it A good reason to add new comments to old code before you change it is to speed up a code review. Adding comments is also a good way to reduce cognitive load.

Recruitment 233

Recruitment Java Coding Project 233

Data Engineering Podcast

FEBRUARY 4, 2024

Contact Info yingjunwu on GitHub Personal Website LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Closing Announcements Thank you for listening!

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

JUNE 9, 2024

In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Can you describe what Datorios is and the story behind it? What are some of the unique challenges posed by Flink?

Process 147

Process Data Lake High Quality Data Machine Learning 147