Simon Späti

FEBRUARY 26, 2025

Traditionally, answering this question would require expensive GIS (Geographic Information Systems) software or complex database setups. Today, DuckDB offers a simpler, more accessible approach for data engineers to tackle spatial problems without specialized infrastructure.

Database 130

Database Data Engineer Data Engineering Accessible 130

The Pragmatic Engineer

NOVEMBER 21, 2023

  We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. We dabbled in network engineering, database management, and system administration. Our deployments were initially manual. What was the other driver of adoption?

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. The current database includes 2,000 server types in 130 regions and 340 zones. Results are stored in git and their database, together with benchmarking metadata. Each benchmarking task is evaluated sequentially.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Snowflake

NOVEMBER 11, 2024

Unify transactional and analytical workloads in Snowflake for greater simplicity Many businesses must maintain two separate databases: one to handle transactional workloads and another for analytical workloads. Sensitive data can have enormous value but is oftentimes locked down due to privacy requirements.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Precisely

JANUARY 9, 2025

These are all big questions about the accessibility, quality, and governance of data being used by AI solutions today. And then a wide variety of business intelligence (BI) tools popped up to provide last mile visibility with much easier end user access to insights housed in these DWs and data marts.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Christophe Blefari

OCTOBER 19, 2023

Private means the model is accessible only within the same group—a model can be only in one group. Once you have dbt build the core project a manifest.json will be generated and tables will be created in the database. Or even more, versioning models. Or even more, versioning models. It can be private, protected or public.

Project 264

Project Finance SQL Government 264

Marc Lamberti

SEPTEMBER 22, 2023

One solution could be to store the accuracies in a database and fetch them back in the task choosing_model with an SQL request. Keep in mind that Airflow stores XComs in the database. To access XComs, go to the user interface, then Admin and XComs. First thing first, xcom_push is accessible only from a task instance object.

MySQL 246

MySQL Data Pipeline Database Python 246

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 187

Data Storage Big Data Hadoop Datasets 187

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements.

Management 93

Management Government Cloud Unstructured Data 93

Data Engineering Podcast

DECEMBER 24, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Snowflake

NOVEMBER 26, 2024

Before it migrated to Snowflake in 2022, WHOOP was using a catalog of tools — Amazon Redshift for SQL queries and BI tooling, Dremio for a data lake, PostgreSQL databases and others — that had ultimately become expensive to manage and difficult to maintain, let alone scale. million in cost savings annually.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Snowflake

APRIL 7, 2025

With advanced encryption, strict access controls and strong data governance, Snowflake helps us ensure the confidentiality and protection of our clients information. We chose Snowflake for its robust, scalable and secure data infrastructure, perfectly suited for handling complex regulatory and quality data efficiently.

Food 90

Food Data Transparency Software Engineer Software Engineering 90

Engineering at Meta

JANUARY 22, 2025

Data lineage refers to the process of tracing the journey of data as it moves through various systems, illustrating how data transitions from one data asset, such as a database table (the source asset), to another (the sink asset). In this blog, we will delve into an early stage in PAI implementation: data lineage.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

The Pragmatic Engineer

JUNE 13, 2023

For transactional databases, it’s mostly the Microsoft SQL Server, but also other databases like PostgreSQL, ScyllaDB and Couchbase. queries per second as total load, spread across its managed database-as-a-service (DBAAS.) It uses Spark for the data platform. At peak load, Agoda sees around 7.5M

Cloud 251

Cloud Database Utilities BI 251

The Pragmatic Engineer

APRIL 11, 2024

” It’s hard to get too much out of these vague reports, but here’s my attempt at decrypting what might have happened: An Oracle version was updated, and/or database schema changes were made. The changes messed up all major databases in some unexpected way. Sella needs Oracle’s help to figure things out.

Banking 271

Banking Utilities Database Engineering 271

Snowflake

NOVEMBER 11, 2024

Optimize performance and cost with a broader range of model options Cortex AI provides easy access to industry-leading models via LLM functions or REST APIs, enabling you to focus on driving generative AI innovations. We offer a broad selection of models in various sizes, context window lengths and language supports.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Precisely

MARCH 28, 2025

However, they faced a growing challenge: integrating and accessing data across a complex environment. Some departments used IBM Db2, while others relied on VSAM files or IMS databases creating complex data governance processes and costly data pipeline maintenance. The result?

Data Integration 69

Data Integration Pipeline-centric Database-centric Kafka 69

Monte Carlo

APRIL 8, 2025

It’s the difference between knowing which documents can be shared in a public Slack channel versus which ones need encrypted storage and limited access. And most importantlywho really needs access to this data? Step 2: Hunt Down the Sensitive Stuff Now its time to play detective in your database. Databases change.

PostgreSQL 52

PostgreSQL Medical Database Data 52

The Pragmatic Engineer

APRIL 6, 2023

Our hope is that making salary ranges more accessible on Comprehensive.io on the backend, and Postgres for database storage.” We are super excited to start playing vector databases - ones that store and index vector embeddings we get from natural language processing models like OpenAI embeddings.

Software Engineering 236

Software Engineering Software Engineer Datasets Database 236

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A fast and open-source column-oriented database management system, which is a popular choice for log management. Ukraine is one of the few countries for which we have access to nationwide data, through job site Djinni.

AWS 347

AWS Software Engineer Software Engineering Google Cloud 347

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. This article highlights the performance optimizations implemented to initialize Atlas, our in-house Graph database, in less than two minutes.

Metadata 52

Metadata PostgreSQL Java Database 52

Netflix Tech

DECEMBER 17, 2024

Metric definitions are often scattered across various databases, documentation sites, and code repositories, making it difficult for analysts and data scientists to find reliable information quickly. Enter DataJunction (DJ).

Engineering 84

Engineering Entertainment Amazon Web Services Utilities 84

KDnuggets

OCTOBER 12, 2023

SQL is the essential data science language due to its universal database accessibility, efficient data cleaning capabilities, seamless integration with other languages, and requirement for most data science jobs.

Data Science 154

Data Science SQL Database Data 154

Snowflake

APRIL 9, 2025

It connects structured and unstructured databases across sources and uses a no-code UI or Python for advanced and predictive analytics. Users can work with the data by defining business concepts instead of writing database queries, and data structures can be reoptimized without major infrastructure changes as business needs evolve.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Rockset

JUNE 21, 2024

We developed our search and analytics database, taking full advantage of the cloud, to eliminate the complexity inherent in the data infrastructure needed for these apps. With this acquisition, what we’ve developed over the years will help make AI accessible to all in a safe and beneficial way.

Database 145

Database Cloud Accessible Accessibility 145

The Pragmatic Engineer

SEPTEMBER 19, 2024

These scripts mixed database access, HTML generation, and logic in unexpected ways. If it isn’t working, then there might be problems delivering electricity. At ISO-NE, the electricity price publishing system was a pile of Bash, Perl, PHP, and C. Sometimes a script would generate another script.

Recruitment 297

Recruitment Java Coding Project 297

Analytics Vidhya

MARCH 5, 2023

It stores and retrieves large amounts of data, including photos, movies, documents, and other files, in a durable, accessible, and scalable manner. Introduction S3 is Amazon Web Services cloud-based object storage service (AWS). S3 […] The post Top 6 Amazon S3 Interview Questions appeared first on Analytics Vidhya.

Amazon Web Services 268

Amazon Web Services AWS Cloud Accessible 268

Pinterest Engineering

APRIL 7, 2025

However, as we were migrating our widecolumn database , we saw significant performance degradation across many clusters, especially for our bulk-updated workloads. For these use cases, typically datasets are generated offline in batch jobs and get bulk uploaded from S3 to the database running on EC2.

AWS 57

AWS Bytes Database Data Ingestion 57

Christophe Blefari

MARCH 2, 2024

Currently it support around 250 sources, which is a subset of all Airbyte sources (only the ones written in Python) and it seems it does not support connecting to classic databases. ingestr — ingestr is a CLI tool to copy data between any databases with a single command seamlessly. It's built on top of dlt. Written in Go.

Data 162

Data Python Database BI 162

Data Engineering Weekly

OCTOBER 27, 2024

Astasia Myers: The three components of the unstructured data stack LLMs and vector databases significantly improved the ability to process and understand unstructured data. I never thought of PDF as a self-contained document database, but that seems a reality that we can’t deny. What are you waiting for?

Data Engineer 58

Data Engineer Data Engineering Engineering Unstructured Data 58

Rockset

MARCH 1, 2023

Every database built for real-time analytics has a fundamental limitation. When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. So they are not suitable for real-time analytics.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

Snowflake

JUNE 10, 2024

Declaratively manage database objects : Embrace a declarative approach for defining and managing Snowflake objects, using Python or SQL, with Database Change Management. A simple pip install snowflake grants developers access, eliminating the need to juggle between SQL and Python or wrestle with cumbersome syntax.

Python 119

Python Data Pipeline SQL Database 119

phData: Data Engineering

NOVEMBER 8, 2024

Data Versioning and Time Travel Open Table Formats empower users with time travel capabilities, allowing them to access previous dataset versions. This feature is essential in environments where multiple users or applications access, modify, or analyze the same data simultaneously. Amazon S3, Azure Data Lake, or Google Cloud Storage).

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Edureka

APRIL 22, 2025

Meanwhile, customers are responsible for protecting resources within the cloud, including operating systems, applications, data, and the configuration of security controls such as Identity and Access Management (IAM) and security groups.

Amazon Web Services 40

Amazon Web Services AWS Cloud Data Governance 40