Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

KDnuggets

SEPTEMBER 8, 2023

Understand the relevance of databases in data science. Also learn the fundamentals of relational databases, NoSQL database categories, and more.

Database 146

Database Data Science NoSQL Relational Database 146

Data Engineering Podcast

APRIL 14, 2024

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Analytics Vidhya

FEBRUARY 22, 2023

Introduction Data replication is also known as database replication, which is copying data to ensure that all information remains consistent across all data resources in real-time. data replication is like a safety net that keeps your information safe from disappearing or falling through the cracks.

Database 269

Database Data NoSQL Datasets 269

KDnuggets

MARCH 5, 2024

Looking to learn SQL and databases to level up your data science skills? Learn SQL, database internals, and much more with these free university courses.

SQL 149

SQL Database Data Science Data 149

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. In this episode he explains how he used the combination of Apache Arrow, Flight, Datafusion, and Parquet to lay the foundation of the newest version of his time-series database. Data lakes are notoriously complex.

Database 162

Database Technology Data Lake High Quality Data 162

KDnuggets

OCTOBER 18, 2023

Cloud databases have made it easier and cheaper to develop enterprise-level applications, offering flexibility, convenience, and standard database functionality. See what KDnuggets recommends.

Database 148

Database Cloud IT Data Engineering 148

KDnuggets

SEPTEMBER 12, 2023

This blog discusses vector databases, specifically pinecone vector databases. A vector database is a type of database that stores data as mathematical vectors, which represent features or attributes. These vectors have multiple dimensions, capturing complex data relationships.

Database 142

Database Machine Learning Data Analysis Systems 142

KDnuggets

JUNE 28, 2024

Explore the world of modern databases that are fast, secure, and cost-efficient, designed to tackle large-scale and diverse data challenges.

Database 145

Database SQL Designing Data 145

KDnuggets

SEPTEMBER 13, 2024

Learn how to install ClickHouse DBMS, create a database, and run SQL queries using native and Python clients.

Database 134

Database SQL Python Data Engineering 134

KDnuggets

NOVEMBER 13, 2023

This week, we delve into the vital world of Databases, SQL, Data Management, and Statistical Concepts in Data Science. Welcome back to Week 2 of KDnuggets’ "Back to Basics" series.

Database 144

Database SQL Data Management Management 144

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Analytics Vidhya

FEBRUARY 6, 2023

Especially while working with databases, it is often considered a good practice to follow a design pattern. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? The pattern is not an actual code but a template that can be used to solve problems in different situations.

Accessible 234

Accessible Accessibility Python Database 234

KDnuggets

NOVEMBER 6, 2023

Graph databases are quickly becoming a core part of the analytics toolset for enterprise IT organizations. If you know SQL, you can easily learn Cypher and open up a huge opportunity for data analysis.

Database 142

Database Data Analysis SQL Data 142

KDnuggets

SEPTEMBER 13, 2023

Getting Started with SQL in 5 Steps • Introduction to Databases in Data Science • Time 100 AI: The Most Influential?

Data Science 148

Data Science Database SQL Data 148

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Database 130

Database Building PostgreSQL BI 130

Analytics Vidhya

JUNE 24, 2023

Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases.

Database 211

Database Utilities Data Data Science 211

KDnuggets

APRIL 19, 2024

Learn about Vectors and How Storing Data Can Be Used in LLM Applications.

Database 147

Database Data 147

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. With Materialize, you can!

BI 208

BI SQL Data Machine Learning 208

KDnuggets

NOVEMBER 22, 2023

This week on KDnuggets: Learn how to perform data quality checks using pandas, from detecting missing records to outliers, inconsistent data entry and more • The top vector databases are known for their versatility, performance, scalability, consistency, and efficient algorithms in storing, indexing, and querying vector embeddings for AI applications (..)

Database 137

Database Algorithm Data 137

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. With Materialize, you can! Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!

Building 189

Building SQL BI Python 189

Christophe Blefari

MARCH 2, 2024

Mistral ( credits ) Hello all, this is the Data News, this week edition might be smaller than usual in term of comments as I'm working on a Data News related project that takes me a bit of time, which will probably lead to a series of articles. In 2024 we are more than ever tools to move data from sources to destinations.

Data 162

Data Python Database BI 162

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. It is meant to handle massive volumes of data across many commodity servers while maintaining high availability with no single point of failure.

NoSQL 252

NoSQL Database Project Systems 252

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Want to see Starburst in action?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Analytics Vidhya

MARCH 1, 2023

Introduction Cassandra is an Apache-developed free and open-source distributed NoSQL database management system. It manages huge volumes of data across many commodity servers, ensures fault tolerance with the swift transfer of data, and provides high availability with no single point of failure.

NoSQL 223

NoSQL Java Database Big Data 223

The Pragmatic Engineer

OCTOBER 17, 2024

The current database includes 2,000 server types in 130 regions and 340 zones. Storing data: data collected is stored to allow for historical comparisons. Results are stored in git and their database, together with benchmarking metadata. Visualizing the data: the frontend that allows querying of live and historic data.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Analytics Vidhya

JANUARY 30, 2023

Whether we are analyzing IoT data streams, managing scheduled events, processing document uploads, responding to database changes, etc. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions? appeared first on Analytics Vidhya.

Coding 237

Coding Database Management Process 237

Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 246

Big Data Data Pipeline Data IT 246

Analytics Vidhya

MARCH 14, 2023

This results in the generation of so much data daily. This generated data is stored in the database and will maintain it. SQL is a structured query language used to read and write these databases. Introduction In today’s world, technology has increased tremendously, and many people are using the internet.

SQL 202

SQL Database Technology Data 202

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineering Data Engineer Engineering 147

Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162