The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry.

Engineering 325

Engineering Bytes Cloud Computing AWS 325

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product? Closing Announcements Thank you for listening!

Data Process 147

Data Process Process Data Lake High Quality Data 147

The Pragmatic Engineer

APRIL 21, 2024

Sometimes the best explanations of how a technology solution works come from the software engineers who built it. Other shipped things include  DALL·E 3  (image generation,)  GPT-4  (an advanced model,) and the  OpenAI API  which developers and companies use to integrate AI into their processes.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

The Pragmatic Engineer

JUNE 1, 2023

Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js Change deploys to be a one-command process, instead of multiple steps See blog posts #1-11 for details on all the steps. Incremental progress.

Education 363

Education Project PostgreSQL Software Engineer 363

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

LinkedIn Engineering

OCTOBER 5, 2023

The magic in search happens because of the technology that powers it. And one of the key technologies behind LinkedIn's search and recommendation features is embedding based retrieval (EBR). LinkedIn is on the forefront of leveraging EBR technology to revolutionize the way we approach search and recommendation systems.

IT 133

IT Metadata Designing Cloud 133

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses.

IT 147

IT Data Lake Metadata Data Warehouse 147

Cloudera

APRIL 2, 2024

Moreover, as emerging technologies like generative AI proliferate across Federal use cases, the need for trusted data that is secure, governed and ready for AI has never been more acute. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 101

Designing Process Government Metadata 101

Snowflake

MARCH 14, 2024

Changes to SAP’s warehousing strategy over time mean many customers may need to patch together and manage multiple technologies to extract data in a useful format. To improve the last mile of this process, SNP built a Snowflake Native App, called SNP Glue Connector for SAP. Try SNP Glue Connector for SAP: Get it now!

IT 109

IT Data Ingestion Data AWS 109

Christophe Blefari

MARCH 15, 2024

AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora. AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Monte Carlo

JANUARY 3, 2024

Sure, you could integrate your AI model more deeply into your organization’s processes, but that feels risky. Data teams are scrambling to answer the call. But building a generative AI model that actually drives business value is hard. And in the long run, a quick integration with the OpenAI API won’t cut it. It’s GenAI, but where’s the moat?

Technology 113

Technology Database Data Engineering Data Engineer 113

Analytics Vidhya

MARCH 5, 2023

It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data. HDInsight works seamlessly with the Hadoop ecosystem, which includes technologies like MapReduce, Hive, […] The post Top 6 Microsoft HDFS Interview Questions appeared first on Analytics Vidhya.

Hadoop 246

Hadoop Data Collection Cloud Technology 246

Knowledge Hut

MAY 6, 2024

Through this article, I will take a look at how Agile methodology is adopted in software development organizations and what modifications are made to the pure Agile processes to suit the adopting organizations to get optimum benefit. Organizations learn through the process during initial iterations for pilot projects.

IT 98

IT Banking Retail Project 98

Scott Logic

JULY 16, 2024

However, based on my experience of applying GenAI to enterprise implementations, I am seeing first-hand the sustainability challenges threatening to implode the first generation of this technology. Sadly, those with non-trivial use cases (those that stretch too far beyond language processing) are quickly met with disappointment.

IT 98

IT Architecture Media Technology 98

Knowledge Hut

MAY 2, 2024

Pharma, education technology, industries, science and space, digital inventions, maps and navigation, robotics – you name the domain and you will have instances of machine learning innovations made in it. Doesn’t this piece of information gives you a glimpse of the wondrous possibilities of machine learning and its potential uses?

Machine Learning 98

Machine Learning IT Deep Learning Algorithm 98

Christophe Blefari

MARCH 1, 2023

First let's understand why dbt exists. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. With the public clouds—e.g. It was the previous tag line dbt Labs had on their website.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

The Pragmatic Engineer

MAY 25, 2023

In this issue, we cover one out of six topics from today’s subscriber-only The Scoop issue. To get full articles twice a week, subscribe here. I got a message from a software engineer working at a company which laid off 30% of staff in December 2022. Also, there is business sense in doing this for reputational reasons.

Software Engineering 329

Software Engineering Software Engineer AWS Engineering 329

Precisely

FEBRUARY 6, 2025

For IT operations (ITOps) teams, 2025 means reassessing technology stacks, processes, and people. What to do with all the technology? While a cloud-first company may not have on-prem legacy systems, most companies are running an IBM Z or IBM i for transactional data processes. Where does the legacy sit with the new?

IT 64

IT Management Datasets Systems 64

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” Clearly, this would generate a handsome return for investors and founders.

Software Engineering 293

Software Engineering Software Engineer Programming Language Media 293

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

The Pragmatic Engineer

APRIL 19, 2023

When I think back on the software engineers I looked up to, they all shared this trait where they never took anything at face value. They regularly questioned statements that did not make sense to them, no matter how small the topic was: even if it involved admitting they did not understand a concept. This person explained it. Or does it?

Software Engineer 319

Software Engineer Software Engineering Engineering Media 319

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

Precisely

DECEMBER 16, 2024

As technology continues its rapid ongoing evolution, IT environments have become increasingly complex which leaves businesses needing to adapt at unprecedented speeds. Key Takeaways: Centralized visibility of data is key. Predictive of AIOps capabilities will revolutionize IT operations.

Systems 59

Systems IT Machine Learning Insurance 59

The Pragmatic Engineer

SEPTEMBER 19, 2024

My goal was to fix the debt of hardcoded strings, but I learned a lot about the codebase and our process as I did it.  Without the backing of management, a large-scale rewrite is likely to fail. With this, it’s over to Lou. Adding comments is also a good way to reduce cognitive load. Clarifying the code is even better.

Recruitment 260

Recruitment Java Coding Project 260

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment. Datafold : ![Datafold]([link]

BI 208

BI SQL Machine Learning Data 208

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Jesse Anderson

OCTOBER 5, 2023

Data quality problems are solved by more than just technology. Data products need to be exposed with the right technologies for the various use cases. The technologies speak Kafka’s binary protocol, removing the need for any client-side changes. A few examples: Everything should use Kafka – No, it shouldn’t.

Kafka 195

Kafka Finance Cloud Designing 195

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

The Pragmatic Engineer

JULY 20, 2023

We have a pretty involved interview process for an early stage startup, and we kept tweaking it. Startups that are running out of money, and are unable to raise more capital, are having to forcibly shut down. Of course, it’s not just grim news: startup funding might be harder to come by, but it’s not impossible to get.

Building 207

Building Programming Language Programming Python 207

The Pragmatic Engineer

JUNE 22, 2023

However, on, 16 June 2023, Google VP Matt Madrigal emailed Google employees and shared more details on what exactly is happening: “No transfer of technology is included as part of this agreement. To get full issues twice a week, subscribe here. Google Domains was a breath of fresh air when launched in 2015.

Google Cloud 272

Google Cloud Media Cloud Engineering 272

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. Who needs to be involved in the process of defining and developing that program? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

MAY 12, 2024

In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Code Comments Podcast Logo]([link] Putting new technology to use is an exciting prospect. Don't forget to check out our other shows.

Management 147

Management Data Lake High Quality Data Machine Learning 147

The Pragmatic Engineer

AUGUST 22, 2023

Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process. I’d love to get a glimpse into the game development world, and how it compares to the software engineering approaches at tech companies.”

Software Engineer 206

Software Engineer Software Engineering Consulting Entertainment 206

Data Engineering Podcast

JUNE 30, 2024

Petr emphasizes the need for a holistic approach to data reliability, integrating data systems into broader business processes. For a team who is onboarding onto Synq, what are the steps required to get it integrated into their technology stack and workflows? What does a typical incident/error resolution process look like with Synq?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147