Data Engineering Podcast

AUGUST 6, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Rockset

JUNE 16, 2020

Data-Driven Applications Need Real-Time Aggregations and Joins Developers of data-driven applications face many challenges. Applications of today often operate on data from multiple sources—databases like MongoDB, streaming platforms, and data lakes. Each index is optimized for different types of queries.

MongoDB 52

MongoDB Data Lake PostgreSQL Kafka 52

Snowflake

JUNE 4, 2024

This includes pipelines and transformations with Snowpark, Streams, Tasks and Dynamic Tables (public preview soon); extending AI and ML to Iceberg with Snowflake Cortex AI; performing storage maintenance with capabilities like automatic clustering and compaction; as well as securely collaborating on live data shares.

Government 84

Government Data Ingestion Data PostgreSQL 84

Striim

JANUARY 4, 2023

That’s because you don’t know how many target environments can be used to ingest data from your operational systems. Maybe, you first load data into a data warehouse and later go on to load data into a data lake. Cover schemas in data contracts.

PostgreSQL 52

PostgreSQL Data Warehouse Data Data Lake 52

Rockset

AUGUST 30, 2021

Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection. The majority are still draining streaming data into a data lake or a warehouse and are doing batch analytics.

SQL 52

SQL Kafka MongoDB MySQL 52

Ascend.io

SEPTEMBER 14, 2023

Use Case: Transforming monthly sales data to weekly averages import dask.dataframe as dd data = dd.read_csv('large_dataset.csv') mean_values = data.groupby('category').mean().compute() compute() Data Storage Python extends its mastery to data storage, boasting smooth integrations with both SQL and NoSQL databases.

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Knowledge Hut

NOVEMBER 28, 2023

Concepts of IaaS, PaaS, and SaaS are the trend, and big companies expect data engineers to have the relevant knowledge. Kafka Kafka is one of the most desired open-source messaging and streaming systems that allows you to publish, distribute, and consume data streams. ETL is central to getting your data where you need it.

Data Engineer 52

Data Engineer Data Engineering Engineering Generalist 52

Silectis

JANUARY 3, 2021

Data Ingestion The process by which data is moved from one or more sources into a storage destination where it can be put into a data pipeline and transformed for later analysis or modeling. Data Integration Combining data from various, disparate sources into one unified view.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40

Zalando Engineering

SEPTEMBER 20, 2015

Named after the Javanese word for “queue,” Saiki is built mostly in Python and includes components that provide a scalable Change Data Capture infrastructure, consume PostgreSQL replication logs, and perform other relevant tasks. presumed the prior integration of data distributed over a significant number of sources.

Data Integration 40

Data Integration PostgreSQL Amazon Web Services Kafka 40

AltexSoft

NOVEMBER 7, 2022

However, the platform is compatible with solutions supporting near real-time and real-time analytics — such as Apache Kafka or Apache Spark. For production purposes, choose from PostgreSQL 10+, MySQL 8+, and MsSQL. The Good and the Bad of Power BI Data Visualization. The Good and the Bad of Hadoop Big Data Framework.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Rockset

JANUARY 28, 2022

Rockset works well with a wide variety of data sources, including streams from databases and data lakes including MongoDB , PostgreSQL , Apache Kafka , Amazon S3 , GCS (Google Cloud Service) , MySQL , and of course DynamoDB. Results, even for complex queries, would be returned in milliseconds.

NoSQL 52

NoSQL PostgreSQL MongoDB ETL Tools 52

Striim

DECEMBER 8, 2023

CDC is a technique designed to efficiently capture and track changes made in a source database, thereby enabling real-time data synchronization and streamlining the process of updating data warehouses, data lakes, or other systems. See enabling Kafka Streams in Striim Platform (self-hosted).

Kafka 64

Kafka Database Data Warehouse Data 64

Rockset

NOVEMBER 28, 2022

First, CDC theoretically allows companies to analyze and react to data in real time, as it’s generated. It works with existing streaming systems like Apache Kafka, Amazon Kinesis, and Azure Events Hubs, making it easier than ever to build a real-time data pipeline. This method offers a few enormous advantages over batch updates.

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB SQL 52

Rockset

OCTOBER 20, 2021

Let’s walk through an example workflow for setting up real-time streaming ELT using dbt + Rockset: Write-Time Data Transformations Using Rollups and Field Mappings Rockset can easily extract and load semi-structured data from multiple sources in real-time. For instance, let’s say you have streaming data coming in from Kafka or Kinesis.

SQL 52

SQL PostgreSQL MongoDB NoSQL 52

Confluent

OCTOBER 16, 2019

Trains are an excellent source of streaming data—their movements around the network are an unbounded series of events. Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. As with any real system, the data has “character.”

Kafka 19

Kafka Building Data Coding 19

Rockset

JULY 25, 2019

encompasses your data pipeline that sources data from various sources deposits it into your data lake or data warehouse runs various transformations to extract insights, and then. In this respect, it is very similar to transactional databases like Oracle, PostgreSQL, etc.

Software Engineering 40

Software Engineering Software Engineer Engineering PostgreSQL 40

Databand.ai

DECEMBER 13, 2022

Bob also hosts The Engineering Side of Data podcast , which is dedicated to discussions around data engineering and features a variety of guests from the data engineering space. His specialties include Microsoft SQL Server, Azure Databricks, Azure Data Factory, SQL Server Integration Services (SSIS), and Azure Data Lake.

Data Engineer 52

Data Engineer Data Engineering Engineering AWS 52

ProjectPro

JULY 27, 2021

Non-relational databases are ideal if you need flexibility for storing the data since you cannot create documents without having a fixed schema. E.g. PostgreSQL, MySQL, Oracle, Microsoft SQL Server. E.g. Redis, MongoDB, Cassandra, HBase , Neo4j, CouchDB What is data modeling? A data warehouse can contain unstructured data too.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40

ProjectPro

DECEMBER 10, 2021

Azure Backup is a cloud-based solution offered by Microsoft that allows you to backup Azure Windows VMs, Azure Managed Disks, Azure File shares, SQL Server databases, SAP HANA databases, Azure PostgreSQL databases, etc. It is responsible for faster and cost-effective processing of vast amounts of data in a configurable framework.

BI 52

BI Cloud Computing SQL Database 52

ProjectPro

AUGUST 21, 2015

Unorganized and raw data that cannot be categorized as semi-structured or structured data is referred to as unstructured data. are all examples of unstructured data. It is not possible to use Apache Kafka without Zookeeper because if the Zookeeper is down Kafka cannot serve client request.

Hadoop 40

Hadoop Big Data Metadata Java 40