Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Monte Carlo

SEPTEMBER 17, 2024

It’s because data owners are responsible for ensuring the quality, security, and accessibility of a dataset across the entire organization. And while that all sounds good, like something every business would want, it can only happen if the data owner prods every employee to manage their data properly.

Accessible 52

Accessible Accessibility Government Data 52

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 81

Metadata Management Data Governance Government 81

Snowflake

OCTOBER 19, 2023

Welcome to Snowflake’s Startup Spotlight, where we learn about amazing companies building their businesses on Snowflake. I’m inspired by the opportunity to create a more accessible and fair financial system for underserved communities, alleviating the struggle and ultimately boosting economic growth.

Accessible 90

Accessible Accessibility Banking Data Science 90

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. Snowflake Cortex AI is a fully managed service designed to unlock the potential of the technology for everyone within an organization, regardless of their technical expertise.

Coding 91

Coding Building Management Government 91

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? What do you have planned for the future of Scanner?

Data Lake 147

Data Lake Building High Quality Data AWS 147

ThoughtSpot

OCTOBER 18, 2024

In the fast-paced world of software development, the efficiency of build processes plays a crucial role in maintaining productivity and code quality. At ThoughtSpot , while Gradle has been effective, the growing complexity of our projects demanded a more sophisticated approach to understanding and optimizing our builds.

Building 52

Building Process Pipeline-centric Database-centric 52

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Snowflake

OCTOBER 5, 2023

Learn how Europe’s leading embedded finance provider delivers data faster, decentralizes access and improves lineage controls with Snowflake’s Data Cloud. But while our existing provider offers amazing solutions, we found a few challenges in how we had to manage data access and lineage.”

Accessible 69

Accessible Accessibility Finance Banking 69

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Despite that, it is still a complex set of capabilities.

Building 130

Building Machine Learning SQL Python 130

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. invoke build buck2 build //starlark-rust/starlark 6.

Systems 141

Systems Building Java Programming Language 141

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes.

Building 100

Building Management Data Pipeline Metadata 100

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. Suppose that we have a data lake table that stores real-time statistics about our customers generated by some Spark application.

Data Lake 89

Data Lake Accessible Accessibility SQL 89

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. A few years ago we embarked on a journey to address these shortcomings by rethinking how our data management systems were designed.

Data Management 91

Data Management Management Data SQL 91

Snowflake

MAY 16, 2024

Snowflake joined forces with Merit to provide an identity verification platform and a set of program delivery services that help run large-scale government programs in areas such as licensing regulations, workforce development, emergency management, and educational grants and scholarships. The work we do together is truly meaningful. ”

Education 82

Education Accessible Accessibility Recruitment 82

Engineering at Meta

DECEMBER 6, 2023

Our aim is to ensure that everyone’s personal messages on Messenger can only be accessed by the sender and the intended recipients, and that everyone can be sure the messages they receive are from an authentic sender. Third-party scrutiny E2EE implies confidentiality even if the provider wants to access the contents of a communication.

Building 144

Building Designing Consulting Accessible 144

Snowflake

JUNE 11, 2024

Many enterprises stitch together a complex mix of various MLOps tools to build an end-to-end ML pipeline. The friction of having to set up and manage separate environments for features and models creates operational complexity that can be costly to maintain and difficult to use.

Management 92

Management Government Metadata Python 92

Data Engineering Podcast

JULY 27, 2021

Datastax has recently introduced a new managed offering for Pulsar workloads in the form of Astra Streaming that lowers those barriers and make stremaing workloads accessible to a wider audience. Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here.

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

JANUARY 29, 2023

The team at Omni have taken a new approach by automatically building models based on the queries that are executed. In this episode Chris Merrick shares how they manage integration and automation around the modeling layer and how it improves the organizational experience of business intelligence.

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. Shane Gibson co-founded AgileData to make analytics accessible to companies of all sizes. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping.

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

JANUARY 15, 2023

Unfortunately it has also introduced new overhead to manage the full experience as a single workflow. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations.

Coding 130

Coding Building PostgreSQL Data Lake 130

Scott Logic

JUNE 29, 2023

Every year on Global Accessibility Awareness Day , people come together to raise awareness about the barriers many users still face in the digital world. This May, we, Doro Hinrichs and Oded Sharon, attended an accessibility panel discussion and workshop with UserVision on Creating Accessible Digital Experiences.

Accessible 52

Accessible Accessibility Consulting Designing 52

Edureka

JULY 19, 2024

Access control fundamentally secures systems by controlling and monitoring who can access resources or perform actions. Broken access control , on the other hand, is a vulnerability that is devoid of proper access control; Hence, the users could act or get access to resources to which they really should not have the authorization.

Accessible 40

Accessible Accessibility IT Coding Skills 40

Snowflake

JUNE 28, 2023

This empowers organizations to break free from the burdensome task of spinning up and managing complex standalone systems operating outside of Snowflake’s governance boundary. This is great for building Snowpark apps/integration that requires execution of Snowpark codes that does not have to be persisted.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. Getting Started with the Snowflake Python API Our Quickstart guide makes it easy to see how the Snowflake Python API can manage Snowflake objects.

Data Pipeline 111

Data Pipeline Python Data Engineering Data Engineer 111

Robinhood

APRIL 26, 2023

“Our conviction in the future of Web3 remains strong, and we’ve continued to build new products that put the accessibility and usability of crypto front and center for customers, positioning Robinhood as the lowest cost, easiest to use on-ramp to crypto.” To learn more, visit [link].

Accessible 52

Accessible Accessibility Education Portfolio 52

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 64

Building Metadata Software Engineer Software Engineering 64

Team Data Science

JUNE 6, 2020

I began building out the data ingestion pipeline by launching an EC2 instance. I should note that if you have created an AWS account, but have not yet created an Identity Access Management (IAM) admin role, and are therefore still using root credentials, I am strongly urging you now to set that up before moving forward.

Project 130

Project Management Data Ingestion AWS 130

Netflix Tech

FEBRUARY 16, 2021

However, with our rapid product innovation speed, the whole approach experienced significant challenges: Business Complexity: The existing SKU management solution was designed years ago when the engagement rules were simple?—?three The rules, now encoded in JSON, were easy to generate, manage and modify via automated means.

Building 86

Building Management Metadata Entertainment 86

The Modern Data Company

JULY 29, 2024

Source: “A New Architecture to Manage Data Costs and Complexity,” BCG, February 2023 At the heart of this crisis is a critical lack of agility. Yet, data teams find themselves perpetually playing catch-up, struggling to meet immediate demands rather than building scalable, future-proof solutions.

Data Architecture 52

Data Architecture Architecture Building Pipeline-centric 52

Snowflake

NOVEMBER 1, 2023

And because the Notebook is natively integrated into Snowflake’s role-based access controls (RBAC), it’s easy to securely share and collaborate on your code and results without compromising on any enterprise data. What’s Next? Check out the Snowpark ML demo from Snowday to see the latest launches in action.

Building 109

Building Python SQL Programming Language 109

Precisely

JULY 18, 2024

Key Takeaways Data Fabric is a modern data architecture that facilitates seamless data access, sharing, and management across an organization. Data management recommendations and data products emerge dynamically from the fabric through automation, activation, and AI/ML analysis of metadata.

Data Management 52

Data Management Management Metadata Database-centric 52

Precisely

DECEMBER 28, 2023

Enterprise technology is having a watershed moment; no longer do we access information once a week, or even once a day. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. To build the streaming data pipeline, you’ll initialize the in-memory framework.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 77

Building Transportation Data Lake Metadata 77

Knowledge Hut

NOVEMBER 7, 2023

The roles of 'Product Manager' (PM) and 'Product Owner' (PO) or POMP are typical in organizations implementing agile overview work practices. What do a product manager and a product owner have in common? Product manager and product owner jobs, on the other hand, are distinct. Product managers are tactical thinkers.

Management 98

Management Manufacturing Certification Building 98