The Pragmatic Engineer

MAY 11, 2023

The company racked up huge bills for the likes of AWS, Snowflake, and also Datadog. A quick summary of these technologies: Prometheus : a time series database. A fast and open-source column-oriented database management system, which is a popular choice for log management. And so, the $65M bill was for Datadog, for 2021.

AWS 339

AWS Software Engineer Software Engineering Google Cloud 339

DoorDash Engineering

MARCH 7, 2023

As backend developers, we needed to stay unblocked while the infrastructure — in this case AWS resources — was being created. We knew we’d be deploying a Docker container to Fargate as well as using an Amazon Aurora PostgreSQL database and Terraform to model our infrastructure as code. Additionally, some require a paid subscription.

AWS 119

AWS PostgreSQL Database SQL 119

Knowledge Hut

NOVEMBER 16, 2023

There is a clear shortage of professionals certified with Amazon Web Services (AWS). As far as AWS certifications are concerned, there is always a certain debate surrounding them. AWS certification helps you reach new heights in your career with improved pay and job opportunities. What is AWS?

AWS 98

AWS Certification IT Amazon Web Services 98

Snowflake

NOVEMBER 11, 2024

Unify transactional and analytical workloads in Snowflake for greater simplicity Many businesses must maintain two separate databases: one to handle transactional workloads and another for analytical workloads.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Analytics Vidhya

MARCH 5, 2023

Introduction S3 is Amazon Web Services cloud-based object storage service (AWS). It stores and retrieves large amounts of data, including photos, movies, documents, and other files, in a durable, accessible, and scalable manner.

Amazon Web Services 268

Amazon Web Services AWS Cloud Accessibility 268

ProjectPro

FEBRUARY 8, 2023

AWS Glue is here to put an end to all your worries! Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Well, AWS Glue is the answer to your problems! In 2023, more than 5140 businesses worldwide have started using AWS Glue as a big data tool.

AWS 98

AWS Scala Metadata Data Lake 98

Analytics Vidhya

MARCH 5, 2023

Introduction Amazon Athena is an interactive query tool supplied by Amazon Web Services (AWS) that allows you to use conventional SQL queries to evaluate data stored in Amazon S3. Athena is a serverless service. Thus there are no servers to operate, and you pay for the queries you perform.

Amazon Web Services 246

Amazon Web Services AWS SQL Data Analysis 246

Cloudera

JULY 9, 2021

Using Operational Database Replication Plugin. The Operational Database Replication Plugin is available both as a standalone plugin as well as installed automatically via Cloudera Replication Manager. Operational Database Replication Plugin uses PAM authentication to validate the machine user credentials. Implementation Details.

Database 100

Database Cloud AWS Systems 100

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. Andy Pavlo researches autonomous database systems, and out of that research he created OtterTune to find the optimal set of parameters to use for your specific workload. How does it relate to your work with NoisePage?

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Towards Data Science

APRIL 6, 2023

But, instead of GCP, we’ll be using AWS. AWS is, by far, the most popular cloud computing platform, it has an absurd number of products to solve every type of specific problem you imagine. So, join me on this post to develop a full data pipeline from scratch using some pieces from the AWS toolset. S3 is AWS’ blob storage.

AWS 98

AWS Data Pipeline Amazon Web Services Python 98

Cloudera

DECEMBER 8, 2021

CDP Operational Database (COD) is a real-time auto-scaling operational database powered by Apache HBase and Apache Phoenix. COD is easy-to-provision and is autonomous, that means developers can provision a new database instance within minutes and start creating prototypes quickly. AWS EC2 instance configurations.

Database 93

Database AWS Datasets Cloud Storage 93

Knowledge Hut

NOVEMBER 17, 2023

When we talk of top cloud computing providers, there are 2 names that are ruling the markets right now- AWS and Google Cloud. Hosting sites at AWS and Google Cloud has become fairly easy. When it comes to public cloud adoption, AWS is still the leader. All the traffic between the data centers is now encrypted by default.

Google Cloud 98

Google Cloud AWS Cloud Cloud Computing 98

Pinterest Engineering

JANUARY 4, 2024

While KVStore was the client facing abstraction, we also built a storage service called Rockstorewidecolumn : a wide column, schemaless NoSQL database built using RocksDB. Additionally, the last section explains how this new database supports a key platform in the product. All names, addresses, phone numbers are illustrative/not real.

Database 98

Database Building Datasets Relational Database 98

Cloudera

SEPTEMBER 9, 2021

The CDP Operational Database ( COD ) builds on the foundation of existing operational database capabilities that were available with Apache HBase and/or Apache Phoenix in legacy CDH and HDP deployments. AWS and Azure standards) reducing cost, complexity and ensuing risk mitigation in HA scenarios: . Savings opportunity on AWS.

Database 95

Database AWS Relational Database Cloud 95

Snowflake

NOVEMBER 11, 2024

Deliver multimodal analytics with familiar SQL syntax Database queries are the underlying force that runs the insights across organizations and powers data-driven experiences for users. Expanded multimodal support enriches responses for diverse tasks such as summarization, classification and entity extraction across various media types.

Unstructured Data 95

Unstructured Data SQL AWS Healthcare 95

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 53

Kafka MySQL Database Software Engineer 53

Data Engineering Podcast

DECEMBER 24, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

DECEMBER 11, 2022

Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. No more shipping and praying, you can now know exactly what will change in your database! Can you describe how Planetscale is implemented?

Database 100

Database MySQL Data Lake MongoDB 100

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. data before the last 2 hours, since GokuS allows only 2 hours of backfill old data in most cases), it stores a copy of the finalized data on AWS EFS (deep persistent storage). Once the data becomes immutable (i.e.

Database 111

Database Bytes Kafka Architecture 111

Data Engineering Podcast

MAY 25, 2021

To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. Mention that you’re a Data Engineering Podcast listener, and they’ll send you a free t-shirt.

Database 100

Database Building Data Warehouse Machine Learning 100

Data Engineering Podcast

AUGUST 6, 2022

For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services.

Machine Learning 100

Machine Learning Database MySQL MongoDB 100

Data Engineering Podcast

MAY 29, 2022

Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required. By supporting fast, in-memory row-based queries and columnar on-disk representation, it lets your transactional and analytical workloads run in the same database.

Database 100

Database Architecture Data Architecture PostgreSQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Summary The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases that continue to be adopted due to their proven reliability and robust features.

Database 100

Database MySQL Cloud MongoDB 100

Towards Data Science

FEBRUARY 19, 2024

Image from Unsplash Building a Semantic Book Search: Scale an Embedding Pipeline with Apache Spark and AWS EMR Serverless Using OpenAI’s Clip model to support natural language search on a collection of 70k book covers In a previous post I did a little PoC to see if I could use OpenAI’s Clip model to build a semantic book search.

AWS 90

AWS Building Python Bytes 90

Zalando Engineering

FEBRUARY 19, 2024

TL;DR : Database per service pattern in the microservices world brings an overhead on operating database instances, observing its health status and anomalies. Often, microservices are implemented with a datastore following the design pattern – database per service , where each service deploys its own database instances.

AWS 72

AWS Utilities Database SQL 72

Cloudera

NOVEMBER 9, 2023

To deploy high-performance applications at scale, a rugged operational database is essential. Cloudera Operational Database (COD) is a high-performance and highly scalable operational database designed for powering the biggest data applications on the planet at any scale. We tested for two cloud storages, AWS S3 and Azure ABFS.

Cloud Storage 74

Cloud Storage Database Cloud AWS 74

Precisely

APRIL 11, 2024

In an era where cloud technology is not just an option but a necessity for competitive business operations, the collaboration between Precisely and Amazon Web Services (AWS) has set a new benchmark for mainframe and IBM i modernization. Solution page Precisely on Amazon Web Services (AWS) Precisely brings data integrity to the AWS cloud.

AWS 71

AWS Amazon Web Services Cloud Cloud Computing 71

Data Engineering Weekly

OCTOBER 27, 2024

Astasia Myers: The three components of the unstructured data stack LLMs and vector databases significantly improved the ability to process and understand unstructured data. I never thought of PDF as a self-contained document database, but that seems a reality that we can’t deny.

Data Engineer 57

Data Engineer Data Engineering Engineering Unstructured Data 57

Sync Computing

JANUARY 27, 2025

Databricks clusters and AWS EC2 In todays landscape, big data, which is data too large to fit into a single node machine, is transformed and managed by clusters. M6GD instances are general-purpose EC2 instances equipped with AWS Graviton2 processors and local NVMe-based SSD storage, offering a balanced mix of compute, memory, and storage.

AWS 52

AWS Data Lake Machine Learning Big Data 52

Towards Data Science

MARCH 20, 2023

Infrastructure as Code for database practitioners Continue reading on Towards Data Science »

MySQL 98

MySQL AWS Data Science Database 98

Team Data Science

MARCH 31, 2020

I built a serverless architecture for my simulated credit card complaints stream using, AWS S3 AWS Lambda AWS Kinesis the above picture gives a high-level view of the data flow. Instead of running database queries over stored data, stream processing applications process data continuously in realtime, even before it is stored.

Process 130

Process AWS Python Architecture 130

Cloudera

SEPTEMBER 1, 2022

Apache HBase has long been the database of choice for business-critical applications across industries. This is primarily because HBase provides unmatched scale, performance, and fault-tolerance that few other databases can come close to. It’s a cloud-native data service that is available on AWS, Azure, and GCP.

Database 84

Database Cloud NoSQL SQL 84

Hevo

AUGUST 2, 2024

AWS offers robust tools to facilitate this, including the AWS Database Migration Service (DMS).Most In the modern data-centric world, efficient data transfer and management are essential to staying competitive. In 2024, over 11441 companies1 […]

AWS 52

AWS Database-centric Data Warehouse Data Storage 52

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Precisely

AUGUST 26, 2024

Key Takeaways: Enhance capabilities through partnerships: AWS, Confluent, and Precisely accelerate mainframe modernization efforts, providing you with essential tools for success. To explore this topic, experts from AWS , Confluent , and Precisely came together to discuss the challenges and opportunities of this migration.

AWS 63

AWS Cloud Data Integration Accessibility 63

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Start Data Engineering

MARCH 27, 2021

This event can be a file creation on S3, a new database row, API call, etc. A common use case is to process a file after it lands on a cloud storage system.

AWS 100

AWS Cloud Storage Database Cloud 100