Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Data Engineering Podcast

MAY 1, 2022

Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Cloudera

MARCH 2, 2022

Similarly, with better usage of available memory more users can query the data at any given time, so more people can use the warehouse at the same time. This post explains the novel technique for how Impala, offered within the Cloudera Data Platform (CDP), is now able to get much more mileage out of the memory at its disposal.

Data Warehouse 89

Data Warehouse Bytes Data Business Intelligence 89

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. The data warehouse needs to reflect the business, and the business should have clarity on how it thinks about analytics.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineering 174

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

Analytics Vidhya

FEBRUARY 27, 2023

Introduction Enterprises here and now catalyze vast quantities of data, which can be a high-end source of business intelligence and insight when used appropriately. Delta Lake allows businesses to access and break new data down in real time.

Data Lake 222

Data Lake Business Intelligence Designing Accessible 222

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. Instead, Facebook came to realize that the work we were doing transcended classic business intelligence. We were data engineers! Data Engineering? We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Podcast

DECEMBER 18, 2022

Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to reduce the potential for broken pipelines some teams have started to adopt the idea of data contracts.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. Two popular approaches that have emerged in recent years are data warehouse and big data. Data warehousing offers several advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery.

Coding 130

Coding Building PostgreSQL Data Lake 130

Sync Computing

NOVEMBER 7, 2024

Data is central to modern business and society. Depending on what sort of leaky analogy you prefer, data can be the new oil , gold , or even electricity. Of course, even the biggest data sets are worthless, and might even be a liability, if they arent organized properly.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Data Engineering Podcast

JANUARY 8, 2023

Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. Missing data? Start trusting your data with Monte Carlo today!

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Weekly

FEBRUARY 2, 2025

Notably, the process includes an RL step to create a specialized reasoning model (R1-Zero) capable of excelling in reasoning tasks without labeled SFT data, highlighting advancements in training methodologies for AI models. It employs a two-tower model approach to learn query and item embeddings from user engagement data.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

Cloudera

SEPTEMBER 24, 2020

Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Both Impala and Hive can operate at an unprecedented and massive scale, with many petabytes of data. So, why choose?

Data Warehouse 71

Data Warehouse SQL Engineering Metadata 71

Knowledge Hut

JUNE 28, 2023

In our data-driven world, our lives are governed by big data. The TV shows we watch, the social media we follow, the news we read, and even the optimized routes we take to work are all influenced by the power of big data analytics. The answer lies in the strategic utilization of business intelligence for data mining (BI).

Data Mining 52

Data Mining Business Intelligence BI Structured Data 52

Knowledge Hut

APRIL 25, 2024

However, with Business intelligence dashboards, knowledge is dispersed throughout the organization, enabling users to produce interactive reports, utilize data visualization, and disseminate the knowledge with internal and external stakeholders. What is a Business Intelligence Dashboard?

Business Intelligence 52

Business Intelligence BI Raw Data Data Warehouse 52

Knowledge Hut

JANUARY 19, 2024

One way to do so is by analyzing the data generated by various business activities like consumer purchase patterns. Every organization has tons of data units stored. For example, all these data sets have information about the consumers' age, gender, and preferences associated with the business.

Business Intelligence 52

Business Intelligence BI Raw Data Data Warehouse 52

Ascend.io

OCTOBER 28, 2024

What is Data Transformation? Data transformation is the process of converting raw data into a usable format to generate insights. It involves cleaning, normalizing, validating, and enriching data, ensuring that it is consistent and ready for analysis. This understanding forms the basis for effective data transformation.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Edureka

FEBRUARY 27, 2023

Business Intelligence Trends: Business intelligence (BI) is becoming an ever more critical element in the success of a business. We’ll also look into ways that businesses can successfully incorporate BI into their practices to gain competitive advantages. What is Business Intelligence?

Business Intelligence 52

Business Intelligence BI Data Mining Data Warehouse 52

Knowledge Hut

MAY 31, 2023

Business Intelligence (BI) comprises a career field that supports organizations to make driven decisions by offering valuable insights. Business Intelligence is closely knitted to the field of data science since it leverages information acquired through large data sets to deliver insightful reports.

Business Intelligence 52

Business Intelligence BI Business Analyst Consulting 52

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. And don’t forget to thank them for their continued support of this show!

Engineering 100

Engineering Business Intelligence Data Warehouse BI 100

Striim

MARCH 21, 2025

Business transactions captured in relational databases are critical to understanding the state of business operations. Since the value of data quickly drops over time, organizations need a way to analyze data as it is generated. What is Change Data Capture?

IT 52

IT Data Lake Data Warehouse Relational Database 52

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

ProjectPro

JANUARY 31, 2023

Want to know who is a business intelligence engineer, what does a business intelligence engineer do, and how these BI engineers turn mountains of data into actionable insights? According to Fortune Business Insights, the global market for business intelligence is likely to grow at a CAGR of 8.7%

Business Intelligence 52

Business Intelligence Engineering BI Data Warehouse 52

Edureka

SEPTEMBER 11, 2023

Today’s business world is very complex and always changing, so businesses have to be able to respond quickly to changes in technology, consumer behaviour, and market conditions. The huge amount of data created every second is one of the main reasons for this complexity. Table of Contents What is business intelligence (BI)?

Business Intelligence 52

Business Intelligence BI Database-centric Raw Data 52

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Christophe Blefari

JUNE 21, 2024

This year, the Snowflake Summit was held in San Francisco from June 2 to 5, while the Databricks Data+AI Summit took place 5 days later, from June 10 to 13, also in San Francisco. Using a quick semantic analysis, "The" means both want to be THE platform you need when you're doing data.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Data Engineering Podcast

DECEMBER 29, 2022

Summary Making effective use of data requires proper context around the information that is being used. Rehgan Avon co-founded AlignAI to help address this challenge through a more purposeful platform designed to collect and distribute the knowledge of how and why data is used in a business. Missing data?

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

U-Next

SEPTEMBER 7, 2022

As data generation and consumption continue to soar, Business Intelligence (BI) has become more relevant in this digital world. With the data generation of more than 2.5 quintillion bytes daily , the significance of Big Data and Data Analytics can be recognized. What Is Business Intelligence Dashboard?

Business Intelligence 52

Business Intelligence BI Finance Raw Data 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Warehouse? .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Knowledge Hut

DECEMBER 26, 2023

The strategic, tactical, and operational business decisions of a company are directly impacted by Business intelligence. BI encourages using historical data to promote fact-based decision-making instead of assumptions and intuition. What is Business Intelligence (BI)?

Business Intelligence 52

Business Intelligence BI Data Mining Data Analysis 52

Knowledge Hut

JANUARY 19, 2024

The key has the right tools, starting with knowing what data is important for your business. Business intelligence (BI) and business analytics (BA) are two terms that are often used interchangeably, but there is some important difference between business intelligence and business analytics.

Business Intelligence 52

Business Intelligence BI Business Analyst Aggregated Data 52

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Data Validation 52

Jesse Anderson

NOVEMBER 15, 2022

My guest this week is Kulani Likotsi , the Head of Data Management and Data Governance at one of the four biggest banks in Africa. She’s had a rising career journey going from an analyst, to a Business Intelligence developer, to the data warehouse team, to the data governance team.

Data Governance 130

Data Governance Government Banking Business Intelligence 130