Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 90

Metadata Management Data Governance Government 90

Snowflake

APRIL 2, 2025

The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems. Tune into our webinar Data Engineering Connect: Building Pipelines for Open Lakehouse on April 29, featuring two virtual demos and a hands-on lab.

Data Lake 82

Data Lake Cloud Storage Metadata Data Warehouse 82

Snowflake

NOVEMBER 11, 2024

What if you could streamline your efforts while still building an architecture that best fits your business and technology needs? At BUILD 2024, we announced several enhancements and innovations designed to help you build and manage your data architecture on your terms. Here’s a closer look.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Towards Data Science

JANUARY 27, 2025

How to Build a Data Dashboard Prototype with Generative AI A book reading data visualization withVizro-AI This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com. Its still not complete and can definitely be extended and improved upon.

Building 53

Building Datasets Coding Data 53

Snowflake

DECEMBER 18, 2023

Data clean rooms have emerged as the technology to meet this need, enabling interoperability where multiple parties can collaborate on and analyze sensitive data in a governed way without exposing direct access to the underlying data and business logic. Snowflake’s acquisition of Samooha is subject to customary closing conditions.

Cloud 133

Cloud Building Entertainment Government 133

Precisely

OCTOBER 31, 2024

Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. The focus seemed to shift to: invent something new → build a service for it → ship it.

Software Engineer 310

Software Engineer Software Engineering Architecture Media 310

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 115

Cloud Building Insurance Data Engineering 115

Snowflake

OCTOBER 7, 2024

When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Define an Entity: Define a Feature View: feature_df is a Snowpark DataFrame object containing your feature definition. Producers can create and modify Feature Views.

Management 94

Management Building Datasets Government 94

Towards Data Science

AUGUST 2, 2023

A Step-by-Step Guide to Building an Effective Data Quality Strategy from Scratch How to build an interpretable data quality framework based on user expectations Photo by Rémi Müller on Unsplash As data engineers, we are (or should be) responsible for the quality of the data we provide. How much should we worry about data quality?

Building 98

Building Data Consolidation Data Datasets 98

The Pragmatic Engineer

OCTOBER 31, 2023

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones.”   I asked Google if europe-west9-a and europe-west9-c are in the same building, at least partially. Regional Spanner should have had one replica in each of the three buildings in the region.

AWS 272

AWS Google Cloud Cloud Engineering 272

Towards Data Science

MARCH 9, 2023

Thus, to facilitate our job it is possible to consolidate all the datasets into a single dataframe and create the “ city ” and “ weekday_or_weekend ” features, which definitely will be essential features to the model. Image 2— Starting the Databricks cluster. Source: The author.

Machine Learning 98

Machine Learning Building Datasets Big Data 98

Ascend.io

NOVEMBER 14, 2024

By focusing on these attributes, data engineers can build pipelines that not only meet current demands but are also prepared for future challenges. Each section will provide actionable insights and practical tips to help you build pipelines that are robust, efficient, and ready for whatever the future holds.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Data Engineering Weekly

JANUARY 15, 2025

To safeguard sensitive information, compliance with frameworks like GDPR and HIPAA requires encryption, access control, and anonymization techniques. The AI Data Engineer: A Role Definition AI Data Engineers play a pivotal role in bridging the gap between traditional data engineering and the specialized needs of AI workflows.

Data Engineering 62

Data Engineering Data Engineer Unstructured Data Engineering 62

Snowflake

NOVEMBER 20, 2023

Addressing the challenges of data-intensive apps Using the combined capabilities of Snowflake Native Apps and Snowpark Container Services, you can build sophisticated apps and deploy them to a customer’s account. All these platform functionalities allow for providers to build trust with their consumers when running inside Snowflake.

Utilities 113

Utilities Machine Learning Coding AWS 113

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Precisely

MARCH 20, 2025

The reality is that business has always been defined by rapid change, and change, by definition, is always disruptive to something. This includes accelerating data access and, crucially, enriching internal data with external information. You can feel secure knowing that all data you access has met rigorous criteria on these fronts.

Insurance 69

Insurance Datasets Data Programming 69

Towards Data Science

JULY 13, 2023

Building a maintainable and modular LLM application stack with Hamilton in 13 minutes LLM Applications are dataflows, use a tool specifically designed to express them LLM stacks. Hamilton is great for describing any type of dataflow , which is exactly what you’re doing when building an LLM powered application. Image from pixabay.

Building 97

Building Database-centric Database Coding 97

The Pragmatic Engineer

MAY 25, 2023

Such a log would build confidence that Glassdoor is a neutral platform which is only enforcing its own terms and conditions, and could validate this. However, there’s a definite and ongoing uptick since the mid-2021. Meanwhile, Amazon has announced Bedrock, but more than a month later not even its own developers have access.

Software Engineer 342

Software Engineer Software Engineering AWS Engineering 342

Data Engineering Podcast

FEBRUARY 19, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. We feel your pain. It ends up being anything but that. We feel your pain.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are some of the useful clarifying/scoping questions to address when deciding the path to deployment for different definitions of "AI"?

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Scott Logic

NOVEMBER 6, 2023

Embedded Accessibility is a vision of building accessible products by default. We can consider accessibility embedded when it no longer needs to be prioritised because it is already at the core of the delivery process. Our products will also be more accessible by default. Does this sound familiar?

Accessible 59

Accessible Accessibility Coding Project 59

Knowledge Hut

MAY 23, 2024

As Per the Project Management Institute (PMI) definition, "Project" signifies "a temporary endeavor with a definite beginning and end." While it may look relatively simpler on the outer aspect of determining what outputs a project can have, several stacked deliverables may require definition En route to achieving the final output.

Project 52

Project Management Certification Government 52

Snowflake

JUNE 10, 2024

We’re excited to provide all Snowflake customers with the core building blocks needed to streamline development workflows, aligned with DevOps best practices, paving a seamless path to production. A simple pip install snowflake grants developers access, eliminating the need to juggle between SQL and Python or wrestle with cumbersome syntax.

Python 118

Python Data Pipeline SQL Database 118

Netflix Tech

FEBRUARY 14, 2025

We will explore the challenges we encounter and unveil how we are building a resilient solution that transforms these client-side impressions into a personalized content discovery experience for every Netflixviewer. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

Snowflake

MARCH 12, 2025

Spaulding Ridge: Turning fan 360 from vision to reality Building a fan 360 requires a comprehensive approach. Technology implementation is "a part of," but not the definition of," its approach. Adding to the complexity are evolving data privacy regulations , requiring careful, secure use of fan data.

Media 59

Media Cloud Programming Data Collection 59

Cloudera

AUGUST 16, 2021

Apache Nifi is a powerful tool to build data movement pipelines using a visual flow designer. Ultimately these challenges force NiFi teams to spend a lot of time on managing the cluster infrastructure instead of building new data flows which slows down use case adoption. The need for a cloud-native Apache NiFi service. and later).

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Christophe Blefari

OCTOBER 9, 2023

Gen AI 🤖 OpenAI’s plan to build the "iPhone of artificial intelligence" — Obviously this is one of the main struggle for OpenAI. Introducing Python and Jinja in Cube — Cube, an open source semantic layer, has released a new writing capabilities in Python with Jinja in the YAML definitions.

Python 130

Python Data Database Java 130

Knowledge Hut

MAY 26, 2024

This means that you can always know exactly where your data is stored and how it is accessed. This system is designed to provide a way for applications to access data stored on a remote server without having to copy the data to the local machine. This maintains order and hierarchy when accessing data from the etcd component.

Metadata 52

Metadata Certification Accessible Accessibility 52

Data Engineering Podcast

NOVEMBER 19, 2023

Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Introducing RudderStack Profiles.

Project 147

Project Data Lake SQL High Quality Data 147

Snowflake

SEPTEMBER 19, 2023

Well, more specifically, LLaMA (Large Language Model Meta AI), along with other large language models (LLMs) that have suddenly become more open and accessible for everyday applications. And I would definitely agree that, in my mind at least, this will have a big, big impact on our world, perhaps even bigger than the internet.

Machine Learning 114

Machine Learning Unstructured Data Data Analytics Government 114

Scott Logic

APRIL 2, 2024

There’s just one last hurdle you’ve got to overcome: accessibility testing. Well, there is… and don’t call me Shirley… Make it automatic What if I told you that you could build accessibility testing into your automated test suites so that you can make sure your pages and components are accessible from the start?

Accessible 52

Accessible Accessibility IT Coding 52

Netflix Tech

DECEMBER 3, 2022

This is one way to build trust with our internal user base. Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring. backfill.sch.yaml ??? daily.sch.yaml ???

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Jesse Anderson

AUGUST 28, 2024

Figure 4 - Does the company definition of a team match the book’s definition? The individual contributors must meet the criteria and definitions to represent the job title. Successful data team management involves building high-performing teams, aligning data with business goals, and leveraging modern tools and processes.

Consulting 147

Consulting Data Big Data Data Engineering 147

Cloudera

OCTOBER 19, 2021

This presented challenges for users in building more complex multi-step pipelines that are typical of DE workflows. Multiple steps comprise the overall pipeline, which are stored as pipeline definition files in the CDE resource of the job. We want to ensure these most commonly used ones are easily accessible to the user.

Coding 120

Coding Data Engineering Data Engineer Engineering 120