Data Engineering Podcast

FEBRUARY 11, 2018

Links Timescale PostGreSQL Citus Timescale Design Blog Post MIT NYU Stanford SDN Princeton Machine Data Timeseries Data List of Timeseries Databases NoSQL Online Transaction Processing (OLTP) Object Relational Mapper (ORM) Grafana Tableau Kafka When Boring Is Awesome PostGreSQL RDS Google Cloud SQL Azure DB Docker Continuous Aggregates Streaming Replication (..)

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop ETL Tools Java 93

Rockset

SEPTEMBER 25, 2020

According to over 40,000 developers, MongoDB is the most popular NOSQL database in use right now. From a developer perspective, MongoDB is a great solution for supporting modern data applications. This blog post will look at three of them: tailing MongoDB with an oplog, using MongoDB change streams, and using a Kafka connector.

MongoDB 52

MongoDB Kafka Database NoSQL 52

Rockset

JUNE 9, 2020

In the course of implementing the Rockset connector to MongoDB , we did a fair amount of research on the MongoDB user experience, both online and through user interviews. Sharding What is MongoDB Sharding and the Best Practices? This was a recurring theme we heard when speaking with MongoDB users.

MongoDB 52

MongoDB NoSQL Kafka Database 52

Knowledge Hut

APRIL 25, 2024

There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data. The most popular NoSQL database systems include MongoDB, Cassandra, and HBase.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Rockset

JUNE 16, 2020

MongoDB.live took place last week, and Rockset had the opportunity to participate alongside members of the MongoDB community and share about our work to make MongoDB data accessible via real-time external indexing. We would be responsible for building and maintaining pipelines from these sources to MongoDB.

MongoDB 52

MongoDB Data Lake PostgreSQL Kafka 52

ProjectPro

JUNE 6, 2025

They include relational databases like Amazon RDS for MySQL, PostgreSQL, and Oracle and NoSQL databases like Amazon DynamoDB. Database Variety: AWS provides multiple database options such as Aurora (relational), DynamoDB (NoSQL), and ElastiCache (in-memory), letting startups choose the best-fit tech for their needs.

AWS 40

AWS Database Amazon Web Services MySQL 40

Rockset

AUGUST 25, 2020

In Part One , we discussed how to first identify slow queries on MongoDB using the database profiler, and then investigated what the strategies the database took doing during the execution of those queries to understand why our queries were taking the time and resources that they were taking.

MongoDB 40

MongoDB NoSQL SQL Relational Database 40

Confluent

AUGUST 19, 2019

It points to best practices for anyone writing Kafka Connect connectors. In a nutshell, the document states that sources and sinks are verified as Gold if they’re functionally equivalent to Kafka Connect connectors. Over the years, we’ve since seen wide adoption of Kafka Connect.

Programming 15

Programming Kafka Database-centric MongoDB 15

AltexSoft

MAY 14, 2021

NoSQL databases. NoSQL databases, also known as non-relational or non-tabular databases, use a range of data models for data to be accessed and managed. The “NoSQL” part here stands for “Non-SQL” and “Not Only SQL”. Cassandra is an open-source NoSQL database developed by Apache. Apache Kafka.

Big Data 78

Big Data Data Analytics IT NoSQL 78

ProjectPro

JUNE 6, 2025

Recommended Reading: Top 50 NLP Interview Questions and Answers 100 Kafka Interview Questions and Answers 20 Linear Regression Interview Questions and Answers 50 Cloud Computing Interview Questions and Answers HBase vs Cassandra-The Battle of the Best NoSQL Databases 3) Name few other popular column oriented databases like HBase.

Hadoop 40

Hadoop Bytes Metadata MongoDB 40

ProjectPro

JUNE 6, 2025

They get used in NoSQL databases like Redis, MongoDB , data warehousing. DB used in AWS MariaDB, Postgres, MongoDB, Oracle, MySQL are some common databases used in AWS. Use cases for EBS are Software development and testing, NoSQL databases, organization-wide application.

Cloud Computing 53

Cloud Computing Cloud Amazon Web Services AWS 53

Knowledge Hut

DECEMBER 26, 2023

They use technologies like Storm or Spark, HDFS, MapReduce, Query Tools like Pig, Hive, and Impala, and NoSQL Databases like MongoDB, Cassandra, and HBase. They also make use of ETL tools, messaging systems like Kafka, and Big Data Tool kits such as SparkML and Mahout.

Data Science 97

Data Science BI Data Mining Business Intelligence 97

ProjectPro

JUNE 6, 2025

Highlight the Big Data Analytics Tools and Technologies You Know The world of analytics and data science is purely skills-based and there are ample skills and technologies like Hadoop, Spark, NoSQL, Python, R, Tableau, etc. that you need to learn to pursue a lucrative career in the industry.

Hadoop 40

Hadoop Recruitment Big Data NoSQL 40

Knowledge Hut

DECEMBER 21, 2023

For a data engineer career, you must have knowledge of data storage and processing technologies like Hadoop, Spark, and NoSQL databases. Understanding of Big Data technologies such as Hadoop, Spark, and Kafka. Familiarity with database technologies such as MySQL, Oracle, and MongoDB. Knowledge of Hadoop, Spark, and Kafka.

Data Engineer 52

Data Engineer Data Engineering Engineering MongoDB 52

Cloud Academy

JANUARY 27, 2022

Microsoft SQL Server Document-oriented database: MongoDB (classified as NoSQL) The Basics of Data Management, Data Manipulation and Data Modeling This learning path focuses on common data formats and interfaces. MongoDB Configuration and Setup Watch an example of deploying MongoDB to understand its benefits as a database system.

Certification 52

Certification Data Engineer Data Engineering Engineering 52

Rockset

NOVEMBER 9, 2021

CDC with Update Timestamps and Kafka One of the simplest ways to implement a CDC solution in both MySQL and Postgres is by using update timestamps. Kafka Connect also has connectors to target systems that can then write these records for you. To simplify this process we can use Kafka Connect.

MySQL 52

MySQL Kafka AWS Database 52

Knowledge Hut

APRIL 25, 2023

Database management: Data engineers should be proficient in storing and managing data and working with different databases, including relational and NoSQL databases. Data processing: Data engineers should know data processing frameworks like Apache Spark, Hadoop, or Kafka, which help process and analyze data at scale.

Data Engineer 52

Data Engineer Data Engineering Engineering Google Cloud 52

Rockset

JANUARY 28, 2022

However, Seesaw’s DynamoDB database stored the data in its own NoSQL format that made it easy to build applications, just not analytical ones. And that was only possible if both internal and external users could drill down into the freshest data possible in order to get the answers they needed.

NoSQL 52

NoSQL MongoDB PostgreSQL ETL Tools 52

Knowledge Hut

JUNE 23, 2023

Other Competencies You should have proficiency in coding languages like SQL, NoSQL, Python, Java, R, and Scala. Equip yourself with the experience and know-how of Hadoop, Spark, and Kafka, and get some hands-on experience in AWS data engineer skills, Azure, or Google Cloud Platform. You can also post your work on your LinkedIn profile.

Data Engineer 52

Data Engineer Data Engineering Engineering Non-relational Database 52

Rockset

APRIL 3, 2020

Rockset This SaaS service allows fast SQL on NoSQL data from varied sources like Kafka, DynamoDB, S3 and more. For example, we could have used either DynamoDB or Cassandra or MongoDB for this project and all would be able to meet the requirement of handling IoT data at scale. We had selected Amazon MSK to run Kafka and Spark.

Data Pipeline 40

Data Pipeline Systems Management NoSQL 40

Rockset

MARCH 26, 2023

Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack. You can learn more about Confluent vs. Kafka over on Confluent’s site.

Cloud 52

Cloud PostgreSQL Kafka Relational Database 52

Knowledge Hut

OCTOBER 27, 2023

cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image, threshold(gray_image, 127, 255, cv2.THRESH_BINARY) THRESH_BINARY) contours, _ = cv2.findContours(thresh, findContours(thresh, cv2.RETR_TREE, RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) boundingRect(max_cnt) else: return None image = cv2.imread("fingerprint.jpg")

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Rockset

AUGUST 11, 2022

Our talk follows an earlier video roundtable hosted by Rockset CEO Venkat Venkataramani, who was joined by a different but equally-respected panel of data engineering experts, including: DynamoDB author Alex DeBrie ; MongoDB director of developer relations Rick Houlihan ; Jeremy Daly , GM of Serverless Cloud.

Bytes 52

Bytes Consulting Kafka MongoDB 52

Rockset

OCTOBER 20, 2021

For instance, let’s say you have streaming data coming in from Kafka or Kinesis. S3 or GCS), NoSQL databases (e.g. DynamoDB or MongoDB), and relational databases (e.g. For high velocity data, most commonly coming from data streams, you can roll it up at write-time. PostgreSQL or MySQL).

SQL 52

SQL MongoDB PostgreSQL NoSQL 52

Knowledge Hut

JUNE 20, 2023

Languages Python, SQL, Java, Scala R, C++, Java Script, and Python Tools Kafka, Tableau, Snowflake, etc. Kafka: Kafka is a top engineering tool highly valued by big data experts. You should be skilled in SQL and knowledgeable about NoSQL databases like Cassandra, MongoDB, and HBase.

Machine Learning 52

Machine Learning Data Engineer Data Engineering Engineering 52

Ascend.io

SEPTEMBER 14, 2023

compute() Data Storage Python extends its mastery to data storage, boasting smooth integrations with both SQL and NoSQL databases. Be it PostgreSQL, MySQL, MongoDB, or Cassandra, Python ensures seamless interactions. Use Case: Processing streaming tweets from pyspark.streaming import StreamingContext from pyspark.

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

AltexSoft

MARCH 14, 2023

Many components of a modern data stack (such as Apache Airflow, Kafka, Spark, and others) are open-source and free. Also, there are NoSQL databases that can be home to all sorts of data, including unstructured and semi-structured (images, PDF files, audio, JSON, etc.) Some popular databases are Postgres and MongoDB.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Knowledge Hut

SEPTEMBER 6, 2023

Big Data Frameworks : Familiarity with popular Big Data frameworks such as Hadoop, Apache Spark, Apache Flink, or Kafka are the tools used for data processing. Implement ETL & Data Pipelines with Bash, Airflow & Kafka; architect, populate, deploy Data Warehouses; create BI reports & interactive dashboards.

Big Data 52

Big Data Certification Hadoop Generalist 52

Rockset

JUNE 7, 2021

Translating the commands from source to target can be tricky especially if you’re capturing changes to a SQL database and reflecting them in a NoSQL database, as the way commands are written are different. Reference Debezium Architecture To handle the queuing of changes, Debezium uses Kafka.

IT 40

IT Kafka MongoDB Database 40

ProjectPro

JUNE 6, 2025

Hadoop , MongoDB, and Kafka are popular Big Data tools and technologies a data engineer needs to be familiar with. They must be skilled at creating solutions that use the Azure Cosmos DB for NoSQL API. Data engineers don't just work with conventional data; and they're often entrusted with handling large amounts of data.

Data Engineer 49

Data Engineer Data Engineering Engineering Certification 49

AltexSoft

OCTOBER 8, 2021

There are also out-of-the-box connectors for such services as AWS, Azure, Oracle, SAP, Kafka, Hadoop, Hive, and more. They include NoSQL databases (e.g., MongoDB), SQL databases (e.g., TOS supports both on-premise and cloud ELT jobs as well as Big Data implementations using databases like NoSQL, Hadoop, Spark.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

ProjectPro

JUNE 6, 2025

E.g. Redis, MongoDB, Cassandra, HBase , Neo4j, CouchDB Check Out ProjectPro's Complete Data Engineering Traning with Enterprise-Grade Data Engineering Projects ! Prepare for Your Next Big Data Job Interview with Kafka Interview Questions and Answers How is a data warehouse different from an operational database? What is data modeling?

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40

ProjectPro

NOVEMBER 14, 2015

Tools/Tech stack used: The tools and technologies used for such weblog trend analysis using Apache Hadoop are NoSql, MapReduce, and Hive. Tools/Tech stack used: The tools and technologies used for such healthcare data management using Apache Hadoop are MapReduce and MongoDB. This system can even handle emergency situations if required.

Hadoop 52

Hadoop Project Big Data Healthcare 52

ProjectPro

JUNE 17, 2016

Highlight the Big Data Analytics Tools and Technologies You Know The world of analytics and data science is purely skills-based and there are ample skills and technologies like Hadoop, Spark, NoSQL, Python, R, Tableau, etc. that you need to learn to pursue a lucrative career in the industry.

Hadoop 40

Hadoop Recruitment Big Data NoSQL 40

ProjectPro

JULY 6, 2016

Recommended Reading: Top 50 NLP Interview Questions and Answers 100 Kafka Interview Questions and Answers 20 Linear Regression Interview Questions and Answers 50 Cloud Computing Interview Questions and Answers HBase vs Cassandra-The Battle of the Best NoSQL Databases 3) Name few other popular column oriented databases like HBase.

Hadoop 40

Hadoop Bytes Metadata MongoDB 40

Knowledge Hut

APRIL 24, 2024

cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image, threshold(gray_image, 127, 255, cv2.THRESH_BINARY) THRESH_BINARY) contours, _ = cv2.findContours(thresh, findContours(thresh, cv2.RETR_TREE, RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) CHAIN_APPROX_SIMPLE) max_area = 0 max_cnt = None for cnt in contours: area = cv2.contourArea(cnt)

Software Engineer 52

Software Engineer Software Engineering Coding Project 52