databricks

OCTOBER 14, 2024

Over the last few years, we've seen tremendous growth and adoption of Databricks SQL , our intelligent data warehouse purpose-built on the Data.

SQL 131

SQL Data Warehouse Data 131

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Engineering at Meta

JANUARY 22, 2025

These stages propagate through various systems including function-based systems that load, process, and propagate data through stacks of function calls in different programming languages (e.g., For simplicity, we will demonstrate these for the web, the data warehouse, and AI, per the diagram below. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Monte Carlo

FEBRUARY 11, 2025

Think of your data warehouse like a well-organized library. Thats where data warehouse schemas come in. A data warehouse schema is a blueprint for how your data is structured and linkedusually with fact tables (for measurable data) and dimension tables (for descriptive attributes).

Data Warehouse 52

Data Warehouse Electronics Retail Data 52

databricks

JANUARY 31, 2024

Welcome to the blog series covering product advancements in 2023 for Databricks SQL, the serverless data warehouse from Databricks. This is part 2.

SQL 130

SQL Programming Data Warehouse Data 130

Start Data Engineering

JANUARY 16, 2021

Introduction Setup Code Conditional logic to read from mock input Custom macro to test for equality Setup environment specific test Run ELT using dbt Conclusion Further reading Introduction With the recent advancements in data warehouses and tools like dbt most transformations(T of ELT) are being done directly in the data warehouse.

SQL 130

SQL Data Warehouse Coding IT 130

Data Engineering Podcast

JUNE 25, 2023

You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Christophe Blefari

JANUARY 11, 2025

He listed 4 things that are the most difficult data integration tasks: from mutable data to IT migrations, everything adds complexity to ingestion systems. The software development lifecycle within a modern data engineering framework — A great deep-dive about a data platform using dltHub, dbt and Dagster.

Data 130

Data Data Warehouse Coding Programming Language 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free! Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Cloudera

MARCH 2, 2021

Contrast that with the skills honed over decades for gaining access, building data warehouses, performing ETL, creating reports and/or applications using structured query language (SQL). The declarative nature of the SQL language makes it a powerful paradigm for getting data to the people who need it.

SQL 112

SQL Java Data Lake Scala 112

Towards Data Science

JUNE 6, 2023

It runs locally, has extensive SQL support and can run queries directly on Pandas data, Parquet, JSON data. The fact it’s insanely fast and does (mostly) all processing in memory make it a good choice for building my personal data warehouse. Extra points for its seamless integration with Python and R.

Data Warehouse 98

Data Warehouse Data SQL Data Analysis 98

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. BigQuery supports DELETE to delete partitions in a SQL query. Arrow doing a lot of the data operation heavy lifting. Give a lot of insights on the market. this is more common sense but always works.

Metadata 272

Metadata Data Data Warehouse Software Engineer 272

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Engineering 100

databricks

AUGUST 10, 2023

At this year's Data+AI Summit, Databricks SQL continued to push the boundaries of what a data warehouse can be, leveraging AI across the.

SQL 98

SQL Data Warehouse Data 98

Data Engineering Podcast

MAY 1, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. No more shipping and praying, you can now know exactly what will change in your database!

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Christophe Blefari

JUNE 21, 2024

Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014. you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with 3) Spark 4.0

Metadata 147

Metadata Data Warehouse BI MySQL 147

databricks

FEBRUARY 12, 2023

Oracle is a well-known technology for hosting Enterprise Data Warehouse solutions. However, many customers like Optum and the U.S. Citizenship and Immigration Services.

Coding 122

Coding SQL Data Warehouse Technology 122

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Try Cloudera Data Warehouse (CDW) by signing up for a 60 day trial , or test drive CDP.

Data Warehouse 82

Data Warehouse Java Metadata Data 82

Towards Data Science

DECEMBER 10, 2024

Take advantage of old school databasetricks In the last 1015 years weve seen massive changes to the data industry, notably big data, parallel processing, cloud computing, data warehouses, and new tools (lots and lots of newtools). Created by the author using SQL-WatchPup Thats it. No server hosting costs.

PostgreSQL 75

PostgreSQL Data Python SQL 75

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

databricks

MAY 3, 2023

Caching is an essential technique for improving the performance of data warehouse systems by avoiding the need to recompute or fetch the same.

SQL 105

SQL Data Warehouse Systems Data 105

Data Engineering Podcast

FEBRUARY 5, 2023

Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Weekly

NOVEMBER 24, 2024

link] Jon Osborn: Best Practices for Using QUERY_TAG in Snowflake The modern data warehouses are good at running at scale, given the cost is not a constraint. link] JBarti: Write Manageable Queries With The BigQuery Pipe Syntax Our quest to simplify SQL is always an adventure.

Data Engineering 58

Data Engineering Data Engineer Engineering Insurance 58

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Sync Computing

NOVEMBER 7, 2024

Data volume and velocity, governance, structure, and regulatory requirements have all evolved and continue to. Despite these limitations, data warehouses, introduced in the late 1980s based on ideas developed even earlier, remain in widespread use today for certain business intelligence and data analysis applications.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Striim

MARCH 21, 2025

Batch processing: data is typically extracted from databases at the end of the day, saved to disk for transformation, and then loaded in batch to a data warehouse. Batch data integration is useful for data that isn’t extremely time-sensitive. Electric bills are a relevant example.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 72

SQL Data Warehouse AWS Accessibility 72

Data Engineering Podcast

JANUARY 8, 2023

In this episode Ori Rafael explains how they are automating the creation and scheduling of orchestration flows and their related transforations in a unified SQL interface. Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Missing data?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessibility 162

Christophe Blefari

SEPTEMBER 15, 2023

How to reduce warehouse costs? — Hugo propose 7 hacks to optimise data warehouse cost. Dimensional data modeling with dbt — A great 6-steps process to create a simple dim-fact model with dbt. teej/titan — Titan is a Python library to manage data warehouse infrastructure. Crazy amounts.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Data Engineering Podcast

JANUARY 15, 2023

With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

DECEMBER 28, 2022

Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Pricing for SQLake is simple.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130