Setting up Data Lake on GCP using Cloud Storage and BigQuery
Analytics Vidhya
FEBRUARY 25, 2023
The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.
From Schemaless Ingest to Smart Schema: Enabling SQL on Raw Data
Rockset
MARCH 27, 2019
You have complex, semi-structured data—nested JSON or XML, for instance, containing mixed types, sparse fields, and null values. It's messy, you don't understand how it's structured, and new fields appear every so often. Without a known schema, it would be difficult to adequately frame the questions you want to ask of the data.
Raw Data 
40
40 Join 37,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Trending Sources
Data Integrity for AI: What’s Old is New Again
Precisely
JANUARY 9, 2025
The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s.
Data Integration 119
119 
Simplifying BI pipelines with Snowflake dynamic tables
ThoughtSpot
MARCH 5, 2024
When created, Snowflake materializes query results into a persistent table structure that refreshes whenever underlying data changes. These tables provide a centralized location to host both your raw data and transformed datasets optimized for AI-powered analytics with ThoughtSpot.
BI 111
111 
Data Vault on Snowflake: Feature Engineering and Business Vault
Snowflake
MARCH 30, 2023
Collecting, cleaning, and organizing data into a coherent form for business users to consume are all standard data modeling and data engineering tasks for loading a data warehouse. Based on Tecton blog So is this similar to data engineering pipelines into a data lake/warehouse?
Engineering 98
98 
Building a SQL Development Environment for Messy, Semi-Structured Data
Rockset
JUNE 13, 2019
Despite the quantity and quality of editors and dashboards available in the SQL community, we realized that using SQL on raw data (e.g. Why ‘reinvent the wheel’ and create our own SQL development environment? nested JSON, Parquet, XML) was a novel concept to our users.
SQL 52
52 
Smart Schema: Enabling SQL Queries on Semi-Structured Data
Rockset
NOVEMBER 19, 2020
In this blog post, we show how Rockset’s Smart Schema feature lets developers use real-time SQL queries to extract meaningful insights from raw semi-structured data ingested without a predefined schema. This is particularly true given the nature of real-world data.
Snowflake PARSE_DOC Meets Snowpark Power
Cloudyard
JANUARY 15, 2025
Apply advanced data cleansing and transformation logic using Python. Automate structured data insertion into Snowflake tables for downstream analytics. Use Case: Extracting Insurance Data from PDFs Imagine a scenario where an insurance company receives thousands of policy documents daily.
Data Cleanse 52
52 
A Guide to Data Pipelines (And How to Design One From Scratch)
Striim
SEPTEMBER 11, 2024
Third-Party Data: External data sources that your company does not collect directly but integrates to enhance insights or support decision-making. These data sources serve as the starting point for the pipeline, providing the raw data that will be ingested, processed, and analyzed.
Accelerate AI Development with Snowflake
Snowflake
NOVEMBER 11, 2024
Deliver multimodal analytics with familiar SQL syntax Database queries are the underlying force that runs the insights across organizations and powers data-driven experiences for users. Traditionally, SQL has been limited to structured data neatly organized in tables.
Snowflake Startup Spotlight: TDAA!
Snowflake
MAY 23, 2024
Right now we’re focused on raw data quality and accuracy because it’s an issue at every organization and so important for any kind of analytics or day-to-day business operation that relies on data — and it’s especially critical to the accuracy of AI solutions, even though it’s often overlooked.
Data Warehouse vs. Data Lake
Precisely
MARCH 9, 2023
We will also address some of the key distinctions between platforms like Hadoop and Snowflake, which have emerged as valuable tools in the quest to process and analyze ever larger volumes of structured, semi-structured, and unstructured data.
Data Lake 59
59 
What is Data Enrichment? Best Practices and Use Cases
Precisely
OCTOBER 5, 2023
According to the 2023 Data Integrity Trends and Insights Report , published in partnership between Precisely and Drexel University’s LeBow College of Business, 77% of data and analytics professionals say data-driven decision-making is the top goal of their data programs. That’s where data enrichment comes in.
Raw Data 69
69 
What Is Data Wrangling? Examples, Benefits, Skills and Tools
Knowledge Hut
JANUARY 29, 2024
In today's data-driven world, where information reigns supreme, businesses rely on data to guide their decisions and strategies. However, the sheer volume and complexity of raw data from various sources can often resemble a chaotic jigsaw puzzle.
Raw Data 52
52 
Understanding Dataform Terminologies And Authentication Flow
Towards Data Science
MAY 14, 2024
Dataform enables the application of software engineering best practices such as testing, environments, version control, dependencies management, orchestration and automated documentation to data pipelines. It is a serverless, SQL workflow orchestration workhorse within GCP.
The Pros and Cons of Leading Data Management and Storage Solutions
The Modern Data Company
MAY 8, 2023
The Data Lake: A Reservoir of Unstructured Potential A data lake is a centralized repository that stores vast amounts of raw data. It can store any type of data — structured, unstructured, and semi-structured — in its native format, providing a highly scalable and adaptable solution for diverse data needs.
The Pros and Cons of Leading Data Management and Storage Solutions
The Modern Data Company
MAY 8, 2023
The Data Lake: A Reservoir of Unstructured Potential A data lake is a centralized repository that stores vast amounts of raw data. It can store any type of data — structured, unstructured, and semi-structured — in its native format, providing a highly scalable and adaptable solution for diverse data needs.
The Pros and Cons of Leading Data Management and Storage Solutions
The Modern Data Company
MAY 8, 2023
The Data Lake: A Reservoir of Unstructured Potential A data lake is a centralized repository that stores vast amounts of raw data. It can store any type of data — structured, unstructured, and semi-structured — in its native format, providing a highly scalable and adaptable solution for diverse data needs.
Data Warehouse vs Data Lake vs Data Lakehouse: Definitions, Similarities, and Differences
Monte Carlo
AUGUST 25, 2023
Understanding data warehouses A data warehouse is a consolidated storage unit and processing hub for your data. Teams using a data warehouse usually leverage SQL queries for analytics use cases. This same structure aids in maintaining data quality and simplifies how users interact with and understand the data.
Data Lake 52
52 
What is Data Extraction? Examples, Tools & Techniques
Knowledge Hut
JANUARY 30, 2024
In today's world, where data rules the roost, data extraction is the key to unlocking its hidden treasures. As someone deeply immersed in the world of data science, I know that raw data is the lifeblood of innovation, decision-making, and business progress. What is data extraction?
ETL Tools 52
52 
Data Lake vs. Data Warehouse: Differences and Similarities
U-Next
SEPTEMBER 7, 2022
Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. Autonomous data warehouse from Oracle. . What is Data Lake? . Essentially, a data lake is a repository of raw data from disparate sources.
Data Lake 52
52 
How to Choose the Right Data Management Solution
The Modern Data Company
MAY 10, 2023
To choose the most suitable data management solution for your organization, consider the following factors: Data types and formats: Do you primarily work with structured, unstructured, or semi-structured data? Consider whether you need a solution that supports one or multiple data formats.
How to Choose the Right Data Management Solution
The Modern Data Company
MAY 10, 2023
To choose the most suitable data management solution for your organization, consider the following factors: Data types and formats: Do you primarily work with structured, unstructured, or semi-structured data? Consider whether you need a solution that supports one or multiple data formats.
How to Choose the Right Data Management Solution
The Modern Data Company
MAY 10, 2023
To choose the most suitable data management solution for your organization, consider the following factors: Data types and formats: Do you primarily work with structured, unstructured, or semi-structured data? Consider whether you need a solution that supports one or multiple data formats.
Differences Between Business Intelligence vs Data Science
Knowledge Hut
APRIL 23, 2024
Data Science is the field that focuses on gathering data from multiple sources using different tools and techniques. Whereas, Business Intelligence is the set of technologies and applications that are helpful in drawing meaningful information from raw data. Business Intelligence only deals with structured data.
Business Intelligence vs. Data Mining: A Comparison
Knowledge Hut
JUNE 28, 2023
Focus Exploration and discovery of hidden patterns and trends in data. Reporting, querying, and analyzing structured data to generate actionable insights. Data Sources Diverse and vast data sources, including structured, unstructured, and semi-structured data.
Data Mining 52
52 
What is data processing analyst?
Edureka
AUGUST 2, 2023
Organisations and businesses are flooded with enormous amounts of data in the digital era. Raw data, however, is frequently disorganised, unstructured, and challenging to work with directly. Data processing analysts can be useful in this situation.
Data Process 52
52 
Data Lake Explained: A Comprehensive Guide to Its Architecture and Use Cases
AltexSoft
AUGUST 29, 2023
The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. This article explains what a data lake is, its architecture, and diverse use cases. Data sources can be broadly classified into three categories.
Data Lake 52
52 
ELT Explained: What You Need to Know
Ascend.io
NOVEMBER 21, 2023
More importantly, we will contextualize ELT in the current scenario, where data is perpetually in motion, and the boundaries of innovation are constantly being redrawn. Extract The initial stage of the ELT process is the extraction of data from various source systems. What Is ELT? So, what exactly is ELT?
Raw Data 52
52 
Mastering the Art of ETL on AWS for Data Management
ProjectPro
FEBRUARY 16, 2023
Data integration with ETL has evolved from structured data stores with high computing costs to natural state storage with read operation alterations thanks to the agility of the cloud. Data integration with ETL has changed in the last three decades.
AWS 52
52 
Power BI Skills in Demand: How to Stand Out in the Job Market
Knowledge Hut
SEPTEMBER 26, 2023
Workspace is the platform where power BI developers create reports, dashboards, data sets, etc. Dataset is the collection of raw data imported from various data sources for the purpose of analysis. DirectQuery and Live Connection: Connecting to data without importing it, ideal for real-time or large datasets.
BI 52
52 
Data Lake vs. Data Warehouse vs. Data Lakehouse
Sync Computing
NOVEMBER 7, 2024
Despite these limitations, data warehouses, introduced in the late 1980s based on ideas developed even earlier, remain in widespread use today for certain business intelligence and data analysis applications. While data warehouses are still in use, they are limited in use-cases as they only support structured data.
Data Lake 52
52 
Business Intelligence vs Artificial Intelligence-Battle of the Brains
ProjectPro
FEBRUARY 16, 2023
Business Intelligence and Artificial Intelligence are popular technologies that help organizations turn raw data into actionable insights. While both BI and AI provide data-driven insights, they differ in how they help businesses gain a competitive edge in the data-driven marketplace.
8 Essential Data Pipeline Design Patterns You Should Know
Monte Carlo
NOVEMBER 21, 2024
You get the structure and performance of a warehouse with the flexibility and scalability of a lake. Want to run SQL queries on your structured data while also keeping raw files for your data scientists to play with? The data lakehouse has got you covered!
Data Science vs Artificial Intelligence [Top 10 Differences]
Knowledge Hut
JANUARY 18, 2024
4 Purpose Utilize the derived findings and insights to make informed decisions The purpose of AI is to provide software capable enough to reason on the input provided and explain the output 5 Types of Data Different types of data can be used as input for the Data Science lifecycle.
Data Science 98
98 
Data Pipeline- Definition, Architecture, Examples, and Use Cases
ProjectPro
DECEMBER 7, 2021
In broader terms, two types of data -- structured and unstructured data -- flow through a data pipeline. The structured data comprises data that can be saved and retrieved in a fixed format, like email addresses, locations, or phone numbers. Step 1- Automating the Lakehouse's data intake.
Difference between Pig and Hive-The Two Key Components of Hadoop Ecosystem
ProjectPro
OCTOBER 15, 2014
Generally data to be stored in the database is categorized into 3 types namely Structured Data, Semi Structured Data and Unstructured Data. 2) Hive Hadoop Component is used for completely structured Data whereas Pig Hadoop Component is used for semi structured data.
Hadoop 52
52 
Data Engineering Zoomcamp – Data Ingestion (Week 2)
Hepta Analytics
FEBRUARY 14, 2022
When the business intelligence needs change, they can go query the raw data again. ELT: source Data Lake vs Data Warehouse Data lake stores raw data. The purpose of the data is not determined. The data is easily accessible and is easy to update.
Data Lakes vs. Data Warehouses
Grouparoo
JANUARY 11, 2022
Difference Between Data Warehouse and Data Lake When looking at the difference between data lake and data warehouse, the following key properties distinguish data lakes vs data warehouses. Data lakes accept and store raw data in any format.
Data Lake 52
52 
How to Design a Modern, Robust Data Ingestion Architecture
Monte Carlo
MAY 28, 2024
Common Tools Data Sources Identification with Apache NiFi : Automates data flow, handling structured and unstructured data. Used for identifying and cataloging data sources. Data Storage with Apache HBase : Provides scalable, high-performance storage for structured and semi-structured data.
Data Lakehouse: Concept, Key Features, and Architecture Layers
AltexSoft
NOVEMBER 10, 2021
The data in this case is checked against the pre-defined schema (internal database format) when being uploaded, which is known as the schema-on-write approach. Purpose-built, data warehouses allow for making complex queries on structured data via SQL (Structured Query Language) and getting results fast for business intelligence.
Architecture 52
52 
Data Warehousing Guide: Fundamentals & Key Concepts
Monte Carlo
FEBRUARY 15, 2023
Cleaning Bad data can derail an entire company, and the foundation of bad data is unclean data. Therefore it’s of immense importance that the data that enters a data warehouse needs to be cleaned. Yes, data warehouses can store unstructured data as a blob datatype. They need to be transformed.
Mythbusting: The Venerable SQL Database and Today’s Real-Time Analytics
Rockset
JANUARY 5, 2022
These streams also continually deliver new fields and columns of data that can be incompatible with existing schemas. Which is why raw data streams cannot be ingested by traditional rigid SQL databases. But some newer SQL databases can ingest streaming data by inspecting the data on the fly.
Database 52
52 
Top ETL Use Cases for BI and Analytics:Real-World Examples
ProjectPro
JANUARY 27, 2023
You have probably heard the saying, "data is the new oil". It is extremely important for businesses to process data correctly since the volume and complexity of raw data are rapidly growing. However, the vast volume of data will overwhelm you if you start looking at historical trends. Well, it surely is!
BI 52
52 
Let's personalize your content