Data Engineering Podcast

JUNE 14, 2021

Typically this means downloading files from object storage, or querying a database. To speed up the process, why not build the model inside the database so that you don’t have to move the information? Can you start by giving an overview of the current state of the market for databases that support in-process machine learning?

Machine Learning 100

Machine Learning Database Data Warehouse Hadoop 100

Data Engineering Podcast

DECEMBER 11, 2022

Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. No more shipping and praying, you can now know exactly what will change in your database! Can you describe how Planetscale is implemented?

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

JANUARY 13, 2019

How has the market for timeseries databases changed since we last spoke? How has the market for timeseries databases changed since we last spoke? Can you refresh our memory about what TimescaleDB is? What has changed in the focus and features of the TimescaleDB project and company? Toward the end of 2018 you launched the 1.0

Database 100

Database PostgreSQL SQL MongoDB 100

Netflix Tech

DECEMBER 14, 2018

System Requirements Support for Structured Data The growth of NoSQL databases has broadly been accompanied with the trend of data “schemalessness” (e.g., We have chosen the high data capacity and high performance Cassandra (C*) database as the backend implementation that serves as the source of truth for all our data.

Media 99

Media Database Metadata Data Schemas 99

Cloudera

SEPTEMBER 23, 2020

Database object security. Database object-level security is available through the centralized authorization framework of Apache Ranger. . Both fine-grained access control of database objects and access to metadata is provided. This was Part 2 of the Operational Database Security blogpost. Security levels.

Database 84

Database Data Lake Metadata Java 84

Team Data Science

SEPTEMBER 1, 2020

There were database developers, database guys, web interface specialists and yeah. So, let's bring Hadoop into play here. Everyone suddenly started talking about Hadoop. Everyone should learn Hadoop. There was a time when people said, "Okay, let's look at Hadoop and become a Hadoop expert.

Data Science 130

Data Science Hadoop Kafka Big Data 130

Cloudera

MAY 18, 2021

Prior the introduction of CDP Public Cloud, many organizations that wanted to leverage CDH, HDP or any other on-prem Hadoop runtime in the public cloud had to deploy the platform in a lift-and-shift fashion, commonly known as “Hadoop-on-IaaS” or simply the IaaS model. Introduction. The case of backup and disaster recovery costs .

Hadoop 86

Hadoop Cloud AWS Utilities 86

Monte Carlo

APRIL 28, 2025

It supports a ton of connectorsfrom SQL databases to machine learning modelsso if youre juggling different tools and platforms, this one can help bring everything together. Apache Atlas Source: Apache Atlas Apache Atlas is more enterprise-focused and really shines if youre in a Hadoop-heavy environment. Its simple, but it works.

Metadata 64

Metadata Hadoop Data SQL 64

Cloudera

DECEMBER 8, 2021

CDP Operational Database (COD) is a real-time auto-scaling operational database powered by Apache HBase and Apache Phoenix. COD is easy-to-provision and is autonomous, that means developers can provision a new database instance within minutes and start creating prototypes quickly. You can access COD right from your CDP console.

Database 93

Database AWS Datasets Cloud Storage 93

The Pragmatic Engineer

JUNE 13, 2023

Ten years ago, this data cluster was 300GB as a Hadoop cluster; that’s around a 100,000-fold increase in data stored! For transactional databases, it’s mostly the Microsoft SQL Server, but also other databases like PostgreSQL, ScyllaDB and Couchbase. It uses Spark for the data platform.

Cloud 242

Cloud Database Utilities BI 242

Data Engineering Podcast

FEBRUARY 5, 2023

Look no further than Materialize, the streaming database you already know how to use. Look no further than Materialize, the streaming database you already know how to use. Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Christophe Blefari

MARCH 1, 2023

dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. Generate databases constraints with dbt. In this resource hub I'll mainly focus on dbt Core— i.e. dbt. First let's understand why dbt exists. How to monitor dbt models.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Knowledge Hut

APRIL 25, 2024

If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.

Big Data 98

Big Data Technology Hadoop NoSQL 98

phData: Data Engineering

NOVEMBER 8, 2024

Evolution of Open Table Formats Here’s a timeline that outlines the key moments in the evolution of open table formats: 2008 - Apache Hive and Hive Table Format Facebook introduced Apache Hive as one of the first table formats as part of its data warehousing infrastructure, built on top of Hadoop.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Striim

MARCH 21, 2025

A streaming ETL for Snowflake approach loads data to Snowflake from diverse sources such as transactional databases, security systems logs, and IoT sensors/devices in real time , while simultaneously meeting scalability, latency, security, and reliability requirements.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Cloudera

AUGUST 19, 2020

For those who are new to HBase or are evaluating it for a new project, HBase is a non-relational distributed database that is trusted by architects and developers who want to process large volumes of data in a timely and reliable manner. . Hadoop) or a banking system to access and view account statements . 100% READ operations.

Database 75

Database Cloud Hadoop Banking 75

Knowledge Hut

DECEMBER 21, 2023

To establish a career in big data, you need to be knowledgeable about some concepts, Hadoop being one of them. Hadoop tools are frameworks that help to process massive amounts of data and perform computation. You can learn in detail about Hadoop tools and technologies through a Big Data and Hadoop training online course.

Hadoop 52

Hadoop Big Data NoSQL Unstructured Data 52

Christophe Blefari

JANUARY 20, 2024

Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. In order to understand today's data engineering I think that this is important to at least know Hadoop concepts and context and computer science basics.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. Your first three Miro boards are free when you sign up today at [dataengineeringpodcast.com/miro]([link] Support Data Engineering Podcast Summary Databases and analytics architectures have gone through several generational shifts.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Knowledge Hut

DECEMBER 26, 2023

Most Popular Programming Certifications C & C++ Certifications Oracle Certified Associate Java Programmer OCAJP Certified Associate in Python Programming (PCAP) MongoDB Certified Developer Associate Exam R Programming Certification Oracle MySQL Database Administration Training and Certification (CMDBA) CCA Spark and Hadoop Developer 1.

Certification 98

Certification Programming MongoDB R (Programming) 98

Knowledge Hut

DECEMBER 28, 2023

That's where Hadoop comes into the picture. Hadoop is a popular open-source framework that stores and processes large datasets in a distributed manner. Organizations are increasingly interested in Hadoop to gain insights and a competitive advantage from their massive datasets. Why Are Hadoop Projects So Important?

Hadoop 52

Hadoop Project Big Data Datasets 52

Monte Carlo

OCTOBER 31, 2024

The foundational skills are similar between traditional data engineers and AI data engineers are similar, with AI data engineers more heavily focused on machine learning data infrastructure, AI-specific tools, vector databases, and LLM pipelines. Let’s dive into the tools necessary to become an AI data engineer.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Knowledge Hut

JULY 27, 2023

The interesting world of big data and its effect on wage patterns, particularly in the field of Hadoop development, will be covered in this guide. As the need for knowledgeable Hadoop engineers increases, so does the debate about salaries. You can opt for Big Data training online to learn about Hadoop and big data.

Hadoop 52

Hadoop Programming Language Banking Big Data 52

Cloudera

FEBRUARY 8, 2022

Cloudera has been recognized as a Visionary in 2021 Gartner® Magic Quadrant for Cloud Database Management Systems (DBMS) and for the first time, evaluated CDP Operational Database (COD) against the 12 critical capabilities for Operational Databases. Evolutionary schema is supported. What Cloudera COD customers are saying .

Database 67

Database Systems Cloud Management 67

Knowledge Hut

MAY 20, 2024

Enter Hadoop , which lets you store data on a massive scale at low cost (compared with similarly scaled commercial databases). That sounds great, but where do you find qualified people who know how to use Pig, Hive, Scoop and other tools needed to run Hadoop?

Hadoop 52

Hadoop Big Data BI Business Intelligence 52

Cloudera

NOVEMBER 9, 2023

To deploy high-performance applications at scale, a rugged operational database is essential. Cloudera Operational Database (COD) is a high-performance and highly scalable operational database designed for powering the biggest data applications on the planet at any scale. We tested for two cloud storages, AWS S3 and Azure ABFS.

Cloud Storage 74

Cloud Storage Database Cloud AWS 74

Cloudera

OCTOBER 5, 2021

This means many manually implemented Ranger HDFS policies, Hadoop ACLs, or POSIX permissions created solely for this purpose can now be removed, if desired. Instead, it generates a mapping that allows the Ranger Plugin in HDFS to make run-time decisions based on the Hadoop SQL grants.

Hadoop 96

Hadoop SQL Database Accessibility 96

Christophe Blefari

NOVEMBER 11, 2022

Mastodon and Hadoop are on a boat. Introduction to Snowflake's Micro-Partitions — I think that explaination about databases internals are my favourite tech articles. Slowly, years after years, graph databases time is coming up. EdgeDB is an hybrid open-source graph database developed on top of Postgres.

BI 130

BI Data Warehouse Data Database 130

Data Engineering Podcast

APRIL 2, 2023

To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links MAD Landscape First Mark Capital Bayesian Learning AI Winter Databricks Cloud Native Landscape LUMA Scape Hadoop Ecosystem Modern Data Stack Reverse ETL Generative AI dbt Transform Podcast Episode Snowflake IPO Dataiku Iceberg Podcast (..)

Hadoop 130

Hadoop Machine Learning Python Architecture 130

Cloudera

NOVEMBER 10, 2021

This blog post provides CDH users with a quick overview of Ranger as a Sentry replacement for Hadoop SQL policies in CDP. Apache Sentry is a role-based authorization module for specific components in Hadoop. It is useful in defining and enforcing different levels of privileges on data for users on a Hadoop cluster.

Hadoop 78

Hadoop SQL Database Kafka 78

Snowflake

OCTOBER 16, 2024

For organizations who are considering moving from a legacy data warehouse to Snowflake, are looking to learn more about how the AI Data Cloud can support legacy Hadoop use cases, or are struggling with a cloud data warehouse that just isn’t scaling anymore, it often helps to see how others have done it.

Digital Media 103

Digital Media Media Data Lake Data Warehouse 103

Data Engineering Podcast

FEBRUARY 9, 2020

You deserve Clickhouse, the open source analytical database that deploys and scales wherever and whenever you want it to and turns data into actionable insights. Is there any utility in data vault modeling in a data lake context (S3, Hadoop, etc.)? What are the steps for establishing and evolving a data vault model in an organization?

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Podcast

AUGUST 19, 2019

One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system. Despite being older than the Hadoop platform it doesn’t seem that HPCC Systems has seen the same level of growth and popularity.

Big Data 100

Big Data Hadoop Data Lake Media 100

Knowledge Hut

DECEMBER 26, 2023

Data Engineers are skilled professionals who lay the foundation of databases and architecture. Using database tools, they create a robust architecture and later implement the process to develop the database from zero. Data engineers who focus on databases work with data warehouses and develop different table schemas.

Data Engineer 98

Data Engineer Data Engineering Engineering Hadoop 98

Maxime Beauchemin

JANUARY 20, 2017

This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. This includes tasks like setting up and operating platforms like Hadoop/Hive/HBase, Spark, and the like.

Data Engineer 130

Data Engineer Data Engineering Engineering ETL Tools 130

Cloudera

SEPTEMBER 15, 2022

It was designed as a native object store to provide extreme scale, performance, and reliability to handle multiple analytics workloads using either S3 API or the traditional Hadoop API. Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases.

Systems 87

Systems Hadoop Metadata Telecommunication 87

Data Engineering Podcast

FEBRUARY 11, 2018

The landscape of time series databases is extensive and oftentimes difficult to navigate. Which came first, Timescale the business or Timescale the database, and what is your strategy for ensuring that the open source project and the company around it both maintain their health?

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100