Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Powered by Trino, the query engine Apache Iceberg was designed for, Starburst is an open platform with support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Cloudera

APRIL 2, 2024

This designation reaffirms Cloudera’s dedication to delivering enterprise-grade, government-scale data management capabilities while maintaining a hyper-focus on data security. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 107

Designing Process Government Metadata 107

The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry. ARPANET was designed to maintain communications during outages. Subscribe here.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Start Data Engineering

MAY 28, 2024

Use DuckDB to process data, not for multiple users to access data 4.2. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Building efficient data pipelines with DuckDB 4.1. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB?

Data Pipeline 219

Data Pipeline Python Building Data 219

Data Engineering Podcast

DECEMBER 3, 2023

What was the design of your initial solution to the problem? What was the design of your initial solution to the problem? What was the original problem that you were trying to solve? What were the shortcomings of other options in the ecosystem that led you to building a new system?

Systems 130

Systems Designing Data Lake SQL 130

Striim

SEPTEMBER 11, 2024

A data pipeline is a systematic sequence of components designed to automate the extraction, organization, transfer, transformation, and processing of data from one or more sources to a designated destination. Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Analytics Vidhya

FEBRUARY 27, 2023

Delta Lake allows businesses to access and break new data down in real time. Delta Lake is an open-source warehouse layer designed to run on top of data lakes analogous to […] The post A Comprehensive Guide on Delta Lake appeared first on Analytics Vidhya.

Data Lake 215

Data Lake Business Intelligence Designing Accessible 215

Data Engineering Podcast

FEBRUARY 18, 2024

What are the differences in terms of pipeline design/access and usage patterns when using a Trino/Iceberg lakehouse as compared to other popular warehouse/lakehouse structures? For someone who is interested in building a data lakehouse with Trino and Iceberg, how does that influence their selection of other platform elements?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Knowledge Hut

MAY 21, 2024

The average salary for a UI/UX Designer is $73,000 (Source:Indeed.com) compared to the average salary of a Front End Developer which is $62,000. So you want to be a designer or developer? But which one should you choose: Front end Developer vs UX designer or UI designer? Almost every business today has a website.

Designing 98

Designing Programming Language Portfolio Media 98

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

The Pragmatic Engineer

APRIL 21, 2024

The engineering, product, and design organization that makes and scales these products is called "Applied," and was founded in 2020 when GPT-3 was released. ChatGPT works impressively well with human language, and has access to more information than any one human could handle.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

The Pragmatic Engineer

MARCH 19, 2024

Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research. Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool. In 1959, the programming language COBOL was designed by software engineer Grace Hopper.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

APRIL 9, 2024

On top of the ability to post jobs, they had access to The Pragmatic Engineer Talent Collective. Companies stayed customers longer, as they were interested in having access to the "latest drops" in the talent collective. (A There was a similar trend in companies paying to access the talent collective and job board.

Software Engineer 271

Software Engineer Software Engineering Engineering Recruitment 271

DoorDash Engineering

FEBRUARY 22, 2023

Figure 1 shows a high-level design of our inventory ingestion pipeline, which is an asynchronous system ingesting inventory from multiple different sources, processing them, and passing them to downstream systems where the view is served to customer-facing entities. Then we will review the primary components in the following sections.

Designing 125

Designing Database Architecture SQL 125

The Pragmatic Engineer

JANUARY 12, 2023

Full subscribers have access to all tech compensation numbers, including for data science, data engineering design, and leadership roles. How much were data scientists, data engineers, designers, leadership and other tech functions paid? How was pay calibrated across these regions? Tech compensation numbers.

Software Engineer 187

Software Engineer Software Engineering Engineering Raw Data 187

The Pragmatic Engineer

MARCH 13, 2023

The issue is that banks (not only SVB) aren’t designed to work like this: they cannot fulfil simultaneous requests for withdrawals by all (or even most) customers. The FDIC stated “all insured depositors will have full access to their insured deposits no later than Monday morning, March 13, 2023.”

Banking 192

Banking Insurance Portfolio Media 192

Data Engineering Podcast

DECEMBER 24, 2023

When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity? When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Snowflake

SEPTEMBER 13, 2023

In the coming months, Salesforce and Snowflake plan to launch BYOL Data Federation so Snowflake data can be accessed within Salesforce Data Cloud, completing the bidirectional data sharing capability, to close the loop and facilitate data activation and streamline value for joint customers.

Finance 112

Finance AWS Accessible Accessibility 112

Data Engineering Podcast

JUNE 23, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. How have the design and goals/scope changed since you first started working on it? What are the core design philosophies that you rely on to prioritize the ongoing development of the project?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Engineering at Meta

DECEMBER 6, 2023

Our aim is to ensure that everyone’s personal messages on Messenger can only be accessed by the sender and the intended recipients, and that everyone can be sure the messages they receive are from an authentic sender. Third-party scrutiny E2EE implies confidentiality even if the provider wants to access the contents of a communication.

Building 145

Building Designing Consulting Accessible 145

ArcGIS

AUGUST 21, 2024

Explore simplified and advanced AI capabilities across desktop, enterprise, and cloud environments, designed to make geospatial intelligence accessible to everyone. Whether you're leveraging pre-trained models for quick insights or building custom AI solutions, ArcGIS offers flexible, powerful tools for every workflow.

Cloud 126

Cloud Designing Accessible Accessibility 126

Data Engineering Podcast

MAY 26, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Knowledge Hut

MARCH 28, 2024

The process of developing digital assets that are both aesthetically pleasing and simple to use is known as user interface/user experience design, or UI/UX design. While UX designers concentrate on the user's journey and how they engage with the product, UI designers are more concerned with the appearance and feel of a product.

Designing 105

Designing Accessible Accessibility Technology 105

The Pragmatic Engineer

MARCH 12, 2024

They] research alternative design strategies. ” This setup made sense when access to computers was more limited because machine time was expensive, but, today, it would be wasteful to not have even an entry-level engineer writing production code. The copilot is described as: “knows all the code intimately.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Engineering at Meta

JUNE 27, 2023

A more ergonomic alternative is EdenFS, which checks out everything in a few seconds but then only actually downloads the files from the server when they are accessed. Buck2 works well without Sapling, but has specific design considerations to enable Sapling and EdenFS.

Java 132

Java Programming Language Algorithm Coding 132

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Snowflake

APRIL 24, 2024

Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourced them to make LLMs more accessible and cost-effective for the community. license provides ungated access to weights and code.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Knowledge Hut

MARCH 22, 2024

These locks are designed to automatically unlock during emergencies or power outages, ensuring swift evacuation or access for first responders. These sophisticated locks operate on the principle of maintaining secure access control, even in the event of power failure or system malfunction.

Systems 105

Systems Electronics Hospitality Architecture 105

Engineering at Meta

MARCH 12, 2024

We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We have been openly designing our GPU hardware platforms beginning with our Big Sur platform in 2015.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Support Data Engineering Podcast Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Email hosts@dataengineeringpodcast.com ) with your story.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JUNE 30, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

APRIL 21, 2024

How have the design and goals of the product changed since you started it? How have the design and goals of the product changed since you started it? What are the strengths that email maintains as a protocol and ecosystem? From a product perspective, what are the data challenges that are posed by email?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Cloudera

DECEMBER 12, 2024

Were proud to announce that Cloudera signed the Cybersecurity and Infrastructure Security Agency (CISA) Secure by Design pledge , joining a network of industry leaders dedicated to embedding security at every stage of the product lifecycle. What is the Secure by Design Pledge? Why is Secure by Design so Important?

Designing 64

Designing AWS Architecture Technology 64

Data Engineering Podcast

JUNE 9, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Snowflake

APRIL 10, 2024

One of our key objectives at Snowflake is to help enterprises fully unlock the value of their data, and an important aspect of that is making data both accessible and actionable to as many people as possible, regardless of their role or technical skill set.

Finance 115

Finance Media Cloud Data 115