The Pragmatic Engineer

OCTOBER 10, 2024

We have seen other similar stories play out recently: In 2021, Elastisearch faced a similar “freerider” challenge from AWS. In response, AWS, GCP, Oracle, Snap and others are backing – and migrating to – the Valkey fork, which remains open source. In response, Elasticsearch ceased  being open source.

Consulting 347

Consulting AWS Engineering Software Engineer 347

Snowflake

NOVEMBER 11, 2024

With Provisional Throughput (public preview soon on AWS), customers can reserve dedicated throughput, ensuring consistent and predictable performance for their workloads. To learn more about these new features and related updates check out our Cortex Analyst blog post.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Knowledge Hut

JANUARY 11, 2024

Are you planning to appear for the AWS Cloud Practitioner Certification in 2023? Here is a complete and comprehensive guide of all the things you can expect from the AWS exams and how to best prepare for them! Amazon Web Services (AWS) is the global leader in cloud computing. Looking for some guidance in the right direction?

AWS 96

AWS Amazon Web Services Cloud Cloud Computing 96

ProjectPro

FEBRUARY 8, 2023

AWS Glue is here to put an end to all your worries! Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Well, AWS Glue is the answer to your problems! Table of Contents What is AWS Glue? How Does AWS Glue Work?

AWS 98

AWS Scala Metadata Data Lake 98

databricks

NOVEMBER 28, 2023

We’re thrilled to share that Databricks has won the AWS ISV Partner of the Year award for North America. This award recognizes top I.

AWS 111

AWS 111

Cloudera

JANUARY 7, 2025

A prominent public health organization integrated data from multiple regional health entities within a hybrid multi-cloud environment (AWS, Azure, and on-premise). A leading meal kit provider migrated its data architecture to Cloudera on AWS, utilizing Cloudera’s Open Data Lakehouse capabilities.

Cloud 82

Cloud Government AWS Unstructured Data 82

databricks

MAY 10, 2023

We're excited to announce the general availability of Databricks Fleet clusters on AWS. What are Fleet clusters? Databricks Fleet clusters unlock the potential.

AWS 96

AWS 96

Workfall

MAY 16, 2023

In this blog, we shall look at how we can create a Dockerfile to create an image with this executable. In this blog, we will cover: Docker AWS EC2 Hands-on Conclusion Docker We are going to containerize our application using Docker to make it easier to deploy on different target platforms. Enter the application name.

AWS 81

AWS Cloud Computing Coding Cloud 81

Data Engineering Weekly

OCTOBER 27, 2024

The blog is an excellent summary of the existing unstructured data landscape. It is exciting to read probably the first blog on building a vector search infrastructure at scale. The blog from Meta discusses how it designed a privacy-preserving storage. Luckily, the Flink community is actively innovating on this.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Precisely

DECEMBER 14, 2023

As 2023 comes to an end we’re counting down the Top 5 Data Integrity blog posts of the year. #5. Trusted Generative AI using Precisely and AWS Welcome to an era where machines are breaking free from their traditional analytical roles to unleash boundless creativity and innovation. Read more > #2.

Data Integration 69

Data Integration AWS Machine Learning Data 69

Data Engineering Weekly

APRIL 20, 2025

The blog took out the last edition’s recommendation on AI and summarized the current state of AI adoption in enterprises. The simplistic model expressed in the blog made it easy for me to reason about the transactional system design. The popularity also exposes its Achilles heel, the replication and network bottlenecks.

Data Engineering 68

Data Engineering Data Engineer Engineering Kafka 68

Cloudera

NOVEMBER 3, 2023

The post EMEA Summit Partner Awards for AWS, Accenture, IBM, Talend a Qlik Company, Tech Data, and Dell Technologies appeared first on Cloudera Blog.

AWS 100

AWS Technology Amazon Web Services Insurance 100

KDnuggets

JANUARY 23, 2023

This blog post will make you less likely to run into issues in this 15+ step process. JupyterHub is a multi-user, container-friendly version of the Jupyter Notebook. However, it can be difficult to setup.

AWS 112

AWS Process IT 112

Data Engineering Weekly

NOVEMBER 24, 2024

The blog highlights how moving from 6-character base-64 to 20-digit base-2 file distribution brings more distribution in S3 and reduces request failures. The blog is a good summary of how to use Snowflake QUERY_TAG to measure and monitor query performance. The blog post made me curious to understand DataFusion's internals.

Data Engineering 58

Data Engineering Data Engineer Engineering Insurance 58

databricks

MARCH 22, 2023

We are excited to announce that PrivateLink and using customer-managed keys (CMK) for encryption are now Generally Available (GA) for Databricks on AWS.

AWS 98

AWS Management 98

Cloudera

SEPTEMBER 15, 2023

Next — working backwards — collaboration tooling such as dashboards and developer environments like those on Amazon Web Services (AWS), are also key components to consider. Building your data monetization strategy on Cloudera and AWS AWS provides several products and solutions to support this approach.

Telecommunication 80

Telecommunication AWS Amazon Web Services Retail 80

databricks

JULY 28, 2023

For the final part of our Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS series, we'll cover an important.

AWS 98

AWS Cloud Engineering 98

databricks

JANUARY 26, 2023

This is part two of a three-part series in Best Practices and Guidance for Cloud Engineers to deploy Databricks on AWS. You can.

AWS 98

AWS Cloud Engineering 98

Yelp Engineering

APRIL 30, 2024

This is the next installment of our blog series on improving our autoscaling infrastructure. In the previous blog posts (Open-sourcing Clusterman, Recycling kubernetes nodes) we explained the architecture and inner-working of Clusterman. Spoiler alert: Karpenter blog post is coming soon!)

AWS 64

AWS Architecture 64

Team Data Science

MAY 10, 2020

The remaining tech (stages 3, 4, 7 and 8) are all AWS technologies. What's Next I'll be documenting how I build this setup in the AWS console (with screenshots). I can now begin drafting my data ingestion/ streaming pipeline without being overwhelmed.

Data Pipeline 130

Data Pipeline Data Ingestion AWS Kafka 130

Edureka

APRIL 22, 2025

Enter Amazon EventBridge, a fully managed serverless event bus service that makes it easier to build event-driven applications using data from your AWS services, custom applications, or SaaS providers. Overall, Amazon EventBridge is a foundational service for anyone looking to embrace modern, event-driven architecture on AWS.

AWS 40

AWS Architecture Media Cloud 40

Confluent

MAY 14, 2024

In the blog we will focus specifically on real-time analysis of AWS audit logs. Explore the practical applications of using the Destinations EventBridge API to send data in real time to Confluent, enabling a myriad of use cases.

AWS 69

AWS Cloud Data 69

The Pragmatic Engineer

OCTOBER 18, 2024

WP Engine is the challenger for the most popular managed WordPress hosting service – generating likely around $400M/year in revenue ( as per Automattic ), versus Automattic’s circa $500M/year, as per Automattic’s CEO, in a now-edited blog post. Automattic raised $980M in venture funding and was valued at $7.5B

Engineering 357

Engineering Government Project AWS 357

Sync Computing

JANUARY 27, 2025

Databricks clusters and AWS EC2 In todays landscape, big data, which is data too large to fit into a single node machine, is transformed and managed by clusters. M6GD instances are general-purpose EC2 instances equipped with AWS Graviton2 processors and local NVMe-based SSD storage, offering a balanced mix of compute, memory, and storage.

AWS 52

AWS Data Lake Big Data Machine Learning 52

Confluent

DECEMBER 7, 2021

In our February 2020 blog post Celebrating Over 100 Supported Apache Kafka® Connectors, we announced support for more than 100 connectors on Confluent Platform. Since then, we have been focused […].

AWS 104

AWS Kafka Management 104

Hevo

AUGUST 29, 2024

AWS Glue is a fully managed serverless ETL service that simplifies preparing and loading data for analytics. In this blog, we will discuss the AWS Glue architecture so you can fully understand how it works and optimize your data better. […] But how does it work?

AWS 52

AWS Architecture Management IT 52

databricks

JUNE 23, 2023

This is a collaborative post from Databricks and Amazon Web Services (AWS). We thank Venkat Viswanathan, Data and Analytics Strategy Leader, Partner Solutions.

Amazon Web Services 98

Amazon Web Services AWS Data 98

Edureka

APRIL 22, 2025

It provides real multi-cloud flexibility in its operations on AWS , Azure, and Google Cloud. Additionally, it offers genuine multi-cloud flexibility by integrating easily with AWS, Azure, and GCP. Snowflake: Offers multi-cloud support, which is present on AWS, Azure, and Google Cloud.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

databricks

APRIL 30, 2024

We are excited to announce that Databricks on AWS GovCloud is now in public preview and that we recently earned our first FedRAMP® High agency ATO! We are ready today to support your International Traffic in Arms Regulations (ITAR) and HIPAA use cases; the Provisional Authorization for DoD Impact Level 5 (IL5) is expected soon.

AWS 59

AWS 59

Cloudyard

APRIL 22, 2025

Read Time: 2 Minute, 34 Second Introduction In modern data pipelines, especially in cloud data platforms like Snowflake, data ingestion from external systems such as AWS S3 is common. In this blog, we introduce a Snowpark-powered Data Validation Framework that: Dynamically reads data files (CSV) from an S3 stage.

Data Validation 52

Data Validation Data Ingestion Data Pipeline AWS 52

Cloudyard

NOVEMBER 5, 2024

In this blog, we’ll: Show how to split a large file using Snowpark’s DataFrame transformations. AWS S3 CLI (cp utility) : For S3-stored files, the AWS CLI cp command supports parallel uploads and efficient file handling. Though it’s a valuable exercise for users looking to deepen their understanding of Snowpark.

AWS 52

AWS Utilities Database Data Engineering 52

Hevo

AUGUST 22, 2024

AWS Glue and Informatica are prominent players offering unique features and benefits. In this blog, AWS Glue vs Informatica […] Businesses today rely heavily on efficient data integration and ETL (Extract, Transform, Load) tools to manage and analyze their data.

AWS 52

AWS Utilities Data Integration Management 52

Cloudyard

DECEMBER 18, 2024

Snowflake, with its robust Snowpark API and features like COPY FILES and REMOVE, allows us to develop fully managed solutions without relying on external automation like AWS Lambda or lifecycle policies.

Retail 52

Retail Data Ingestion AWS Data Pipeline 52

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. The control plane manages various aspects of the system: It runs on a single host inside an AWS® Auto Scaling Group with a minimum and maximum host count of 1. or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineer 50

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. link] CapitalOne: Serverless ML - Lessons from Capital One CapitalOne writes about its experience building Serverless ML on top of AWS Lambda.

Data Engineering 63

Data Engineering Data Engineer Engineering Pipeline-centric 63

Hevo

AUGUST 27, 2024

In this blog, we will explore how to build a data pipeline using AWS Glue S3. AWS Glue is a tool that makes building and managing your data pipelines easier. We will go through every step of the process, and by the end, you will see how straightforward it can be. It’s fully managed, so […]

AWS 52

AWS Building Data Pipeline Management 52

Hevo

AUGUST 4, 2024

AWS Database Migration Service (DMS) offers a comprehensive database migration and replication solution, including support for Change Data Capture (CDC). This blog will delve into configuring AWS DMS CDC Oracle, providing a step-by-step guide and […]

AWS 52

AWS Database Data 52