DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Precisely

JUNE 24, 2024

The Precisely team recently had the privilege of hosting a luncheon at the Gartner Data & Analytics Summit in London. It was an engaging gathering of industry leaders from various sectors, who exchanged valuable insights into crucial aspects of data governance, strategy, and innovation.

Food 92

Food Data Analytics Pharmaceutical Consulting 92

Cloudera

AUGUST 20, 2024

Whether it’s rapidly rising costs, an inefficient and outdated data infrastructure, or serious gaps in data governance, there are myriad reasons why organizations are struggling to move past adoption and achieve AI at scale in their enterprises.

Data Lake 83

Data Lake Government Data Governance Data Architecture 83

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. At what point does a lack of an explicit governance policy become a liability?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ? How do we govern all these data products and domains ? The number of subjects to automatize is not short.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding. This approach helps maintain accuracy, relevance, and compliance in generative AI applications.

Data Engineer 67

Data Engineer Data Engineering Pipeline-centric Engineering 67

Precisely

JULY 18, 2024

While data fabric is not a standalone solution, critical capabilities that you can address today to prepare for a data fabric include automated data integration, metadata management, centralized data governance, and self-service access by consumers. Increase metadata maturity.

Data Management 52

Data Management Management Metadata Database-centric 52

RandomTrees

SEPTEMBER 27, 2024

Unified data platform: One Lake provides a unified platform for all data types, including structured, semi-structured, and unstructured data. Data governance and security: One Lake incorporates robust data governance and security features to ensure data quality, compliance, and protection.

Database-centric 52

Database-centric Pipeline-centric IT BI 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

Cloudera

JUNE 6, 2024

To keep pace as banking becomes increasingly digitized in Southeast Asia, OCBC was looking to utilize AI/ML to make more data-driven decisions to improve customer experience and mitigate risks. Lastly, data security is paramount, especially in the finance industry.

Banking 88

Banking Data Lake Finance Data Governance 88

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB. GDPR, HIPAA), and industry standards.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Science Blog: Data Engineering

MAY 20, 2024

It offers full BI-Stack Automation, from source to data warehouse through to frontend. It supports a holistic data model, allowing for rapid prototyping of various models. It also supports a wide range of data warehouses, analytical databases, data lakes, frontends, and pipelines/ETL. pipelines, Azure Data Bricks.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Cloudera

DECEMBER 11, 2020

The Data Security and Governance category, at the annual Data Impact Awards, has never been so important. The sudden rise in remote working, a huge influx in data as the world turned digital, not to mention the never-ending list of regulations businesses need to remain compliant with (how many acronyms can you name in full?

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Cloudera

MARCH 5, 2024

Struggling to access and collect, oftentimes disparate and siloed, data across environments that are required to power AI, many organizations are unable to achieve the business insight and value they had hoped for. Available for cloud and now also for the data center.

Data Lake 91

Data Lake Data Storage Government Kafka 91

Knowledge Hut

JULY 5, 2024

Cloud Era: Cloud platforms like AWS and Azure took center stage, making sophisticated data solutions accessible to all. Modern Landscape: Today, Data Engineering involves slick ETL processes, real-time streaming, and the concept of data lakes, shaping the backbone of our data-driven world.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Cleanse 52

Snowflake

JUNE 29, 2023

government agencies, including the DoD, can now use Snowflake for controlled unclassified information (CUI), including export controlled data, personally identifiable information (PII), and protected health information (PHI), among others. government-designated regions. Ready to learn more?

Education 52

Education Government Data Lake Data Security 52

Cloudera

JUNE 7, 2023

The revised ZTMM is organized by five categories or pillars: identity, devices, networks, applications and workloads, and data, and four levels of maturity: traditional, initial, advanced, and optimal. It operates independently from compute and storage layers, offering integrated security and governance based on metadata.

Metadata 87

Metadata Government Data Lake Data 87

Databand.ai

AUGUST 30, 2023

Data silos: Legacy architectures often result in data being stored and processed in siloed environments, which can limit collaboration and hinder the ability to generate comprehensive insights. Data storage platforms can include traditional relational databases, NoSQL databases, data lakes, or cloud-based storage services.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Monte Carlo

MAY 18, 2023

Whether you’re a data scientist, data engineer, or business analyst, keeping track of your data’s origin, transformation, and movement is crucial for maintaining transparency, enforcing data governance, and ensuring data quality. The Spark data lake lineage example is helpful for context here.

Metadata 64

Metadata Data Lake Government Data Governance 64

Knowledge Hut

NOVEMBER 2, 2023

Azure Data Ingestion Pipeline Create an Azure Data Factory data ingestion pipeline to extract data from a source (e.g., Azure SQL Database, Azure Data Lake Storage). Data Aggregation Working with a sample of big data allows you to investigate real-time data processing, big data project design, and data flow.

Data Engineering 52

Data Engineering Data Engineer Project Coding 52

Monte Carlo

FEBRUARY 21, 2023

These include: Seamless integration A data fabric’s ability to integrate seamlessly with various APIs and software-delivery kits (SDKs) truly differentiates this framework from another type of data management: the ubiquitous data lake. That’s a model worth looking at when it comes to data governance,” says Bob.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

These include: Seamless integration A data fabric’s ability to integrate seamlessly with various APIs and software-delivery kits (SDKs) truly differentiates this framework from another type of data management: the ubiquitous data lake. That’s a model worth looking at when it comes to data governance,” says Bob.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Podcast

SEPTEMBER 23, 2018

Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. In terms of infrastructure, what are the components of a modern data architecture and how has that changed over the years?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Precisely

MARCH 27, 2023

As part of the project, the company implemented a data governance program and rolled out scalable data quality tools to enhance the overall value it could deliver to its go-to-market organization. After several false starts, the company engaged Precisely Strategic Services to help align its people, processes, and technology.

Food 52

Food Consulting Pharmaceutical Data Lake 52

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Knowledge Hut

OCTOBER 30, 2023

Data Transformation and ETL: Handle more complex data transformation and ETL (Extract, Transform, Load) processes, including handling data from multiple sources and dealing with complex data structures. Ensure compliance with data protection regulations. Define data architecture standards and best practices.

BI 52

BI Business Intelligence Data Cleanse Business Analyst 52

Monte Carlo

MAY 2, 2023

Respondents averaged 642 tables across their data lake, lakehouse, or warehouse environments. Respondents reported having an average of 24 dbt models , and 41% reported having 25 or more dbt models.

Data Lake 80

Data Lake Data Software Engineer Software Engineering 80

Precisely

JUNE 6, 2023

The group kicked off the session by exchanging ideas about what it means to have a modern data architecture. Atif Salam noted that as recently as a year ago, the primary focus in many organizations was on ingesting data and building data lakes.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

The Modern Data Company

APRIL 15, 2024

Why Data Products Matter Imagine if managing your company’s data was as straightforward as using your favorite smartphone app. Data Products aim to make this a reality by simplifying how data is accessed, used, and governed across your organization.

Government 52

Government Metadata Data Data Lake 52

Knowledge Hut

OCTOBER 4, 2023

Business analysts and BI professionals use familiar self-service tools like Power Query data to handle the most complex data preparation challenges. Advanced Analytics and AI with Azure: Power BI requirements for dataflows can store data in Azure data lake storage Gen2.

BI 52

BI Systems Raw Data Business Intelligence 52

Cloudera

SEPTEMBER 15, 2022

Mark: The first element in the process is the link between the source data and the entry point into the data platform. At Ramsey International (RI), we refer to that layer in the architecture as the foundation, but others call it a staging area, raw zone, or even a source data lake. See below. .

Data Lake 87

Data Lake Analytics Application Cloud Storage Architecture 87

Scott Logic

MARCH 21, 2024

Can departmental structures and traditional organisational silos be modernised so we could potentially create a public sector Large Language Model (LLM) as well as cross-departmental data lakes? The benefits, if achieved, could free up resources and staff to deliver more valuable frontline tasks as well as reduce costs.

Healthcare 52

Healthcare Government Data Lake Deep Learning 52

Data Engineering Podcast

FEBRUARY 24, 2020

Snowplow takes care of everything from installing your pipeline in a couple of hours to upgrading and autoscaling so you can focus on your exciting data projects. Your team will get the most complete, accurate and ready-to-use behavioral web and mobile data, delivered into your data warehouse, data lake and real-time streams.

IT 100

IT Data Lake Data Pipeline Media 100

Cloudera

DECEMBER 1, 2022

Data observability provides insight into the condition and evolution of the data resources from source through the delivery of the data products. Barr Moses of Monte Carlo presents it as a combination of data flow, data quality, data governance, and data lineage.

Data Lake 89

Data Lake Data Pipeline Analytics Application Data Governance 89