Towards Data Science

MARCH 23, 2023

The difference between tables and views and how to use them Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Data Science Data Cloud Computing 98

Monte Carlo

OCTOBER 31, 2024

The rise of AI and GenAI has brought about the rise of new questions in the data ecosystem – and new roles. One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Weekly

FEBRUARY 2, 2025

Notably, the process includes an RL step to create a specialized reasoning model (R1-Zero) capable of excelling in reasoning tasks without labeled SFT data, highlighting advancements in training methodologies for AI models. It employs a two-tower model approach to learn query and item embeddings from user engagement data.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Christophe Blefari

NOVEMBER 11, 2022

On a social note, today I've joined data-folks Mastodon server, you can follow me there. I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Ian describes how Riot Games uses data and what machine learning means.

BI 130

BI Data Warehouse Data Database 130

Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. Two popular approaches that have emerged in recent years are data warehouse and big data. Data warehousing offers several advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Grouparoo

JANUARY 11, 2022

When it comes to storing large volumes of data, a simple database will be impractical due to the processing and throughput inefficiencies that emerge when managing and accessing big data. This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. The current data state is heavily dependent on infrastructure, wether it's cloud, on-premise or semi-related we need to understand where the data lands and where the code runs. Often they are right.

Kafka 130

Kafka Data Storage Data Lake Data 130

Sync Computing

NOVEMBER 7, 2024

Data is central to modern business and society. Depending on what sort of leaky analogy you prefer, data can be the new oil , gold , or even electricity. Of course, even the biggest data sets are worthless, and might even be a liability, if they arent organized properly.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Big Data 147

Big Data Cloud Storage Hadoop SQL 147

Monte Carlo

NOVEMBER 21, 2024

Let’s set the scene: your company collects data, and you need to do something useful with it. Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Christophe Blefari

JUNE 21, 2024

This year, the Snowflake Summit was held in San Francisco from June 2 to 5, while the Databricks Data+AI Summit took place 5 days later, from June 10 to 13, also in San Francisco. Using a quick semantic analysis, "The" means both want to be THE platform you need when you're doing data.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Data Engineering Weekly

OCTOBER 11, 2024

The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent.

Manufacturing 109

Manufacturing Transportation Data Warehouse Unstructured Data 109

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data. What Are Big Data T echnologies?

Big Data 98

Big Data Technology Hadoop NoSQL 98

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Most importantly, these pipelines enable your team to transform data into actionable insights, demonstrating tangible business value.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Monte Carlo

MARCH 3, 2025

The real disruption lies with data + AI. In other words, when organizations combine their first-party data with LLMs to unlock unique insights, automate processes, or accelerate specialized workflows. We saw this with software and application observability; data and data observability; and soon data + AI and data + AI observability.

Unstructured Data 52

Unstructured Data Data Cloud Computing Banking 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Cloudyard

JANUARY 21, 2025

Read Time: 6 Minute, 6 Second In modern data pipelines, handling data in various formats such as CSV, Parquet, and JSON is essential to ensure smooth data processing. However, one of the most common challenges faced by data engineers is the evolution of schemas as new data comes in. Technical Implementation: 1.

Data Schemas 59

Data Schemas Data Pipeline Data Warehouse Data Storage 59

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. This hands-on experience will allow you to develop an end-to-end data lifecycle, from extracting data from your Google Calendar to presenting it in a Snowflake analytics dashboard. See Github repo.

Data Engineer 97

Data Engineer Data Engineering Project Engineering 97

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. What is data pipeline observability?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Podcast

AUGUST 14, 2021

Summary The vast majority of data tools and platforms that you hear about are designed for working with structured, text-based data. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Machine Learning Data Lake SQL 100

Cloudera

DECEMBER 12, 2018

After having rebuilt their data warehouse, I decided to take a little bit more of a pointed role, and I joined Oracle as a database performance engineer. I spent eight years in the real-world performance group where I specialized in high visibility and high impact data warehousing competes and benchmarks. Greg Rahn: Sure.

Data Warehouse 42

Data Warehouse Relational Database Hadoop Database 42

Monte Carlo

APRIL 27, 2023

Introduction Thanks to the continued push towards a privacy-first internet, first-party customer data has never been more important to digital organizations. Doing so capitalizes on existing cloud investments while bringing data and marketing teams closer than ever before.

Data Warehouse 72

Data Warehouse Data Collection Architecture Data Storage 72

Knowledge Hut

JANUARY 18, 2024

Data Science has risen to become one of the world's topmost emerging multidisciplinary approaches in technology. Recruiters are hunting for people with data science knowledge and skills these days. Data Scientists collect, analyze, and interpret large amounts of data. Choose data sets.

Data Science 98

Data Science Business Analyst Data Architect ETL Method 98

Monte Carlo

JANUARY 25, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. Data storage and compute are very much the foundation of your data platform. Let’s jump in!

Data Warehouse 40

Data Warehouse Building Data Lake Data Storage 40

Towards Data Science

DECEMBER 15, 2023

For a data scientist, there’s no such thing as too much data. Photo by Trnava University on Unsplash Data Science vs Security/IT: A Battle for the Ages Acquiring and keeping data is the focus of a huge amount of our mental energy as data scientists. If you ask a data scientist “Can we solve this problem?”

Machine Learning 98

Machine Learning Data Science Data Security Data Storage 98

AltexSoft

OCTOBER 30, 2021

Explaining the difference, especially when they both work with something intangible such as data , is difficult. If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. Data science vs data engineering.

Data Engineer 98

Data Engineer Data Engineering Engineering Machine Learning 98