Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 128

Pipeline-centric Food Healthcare Unstructured Data 128

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Data Engineering Podcast

MAY 7, 2023

However, that's also something we're re-thinking with our warehouse-centric strategy. How does reverse ETL factor into the enrichment process for profile data? Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform. Let us know if you have opinions there!

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . CDP data lifecycle integration and SDX security and governance. Key features of CDP Data Engineering.

Data Engineering 126

Data Engineering Data Engineer Engineering Pipeline-centric 126

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

ThoughtSpot

OCTOBER 18, 2024

It aims to explain how we transformed our development practices with a data-centric approach and offers recommendations to help your teams address similar challenges in your software development lifecycle. This approach ensured comprehensive data extraction while handling various edge cases and log formats.

Building 52

Building Process Pipeline-centric Database-centric 52

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Engineering Pipeline-centric 67

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Monte Carlo

MAY 30, 2024

It can involve prompt engineering, vector databases like Pinecone , embedding vectors and semantic layers, data modeling, data orchestration, and data pipelines – all tailored for RAG. But when it’s done right, RAG can add an incredible amount of value to AI-powered data products. What is Fine Tuning?

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

Ascend.io

MAY 7, 2024

” Key Partnership Benefits: Cost Optimization and Efficiency : The collaboration is poised to reduce IT and data management costs significantly, including an up to 68% reduction in data stack spend and the ability to build data pipelines 7.5x ABOUT ASCEND.IO Learn more at Ascend.io or follow us @ascend_io.

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Ascend.io

OCTOBER 20, 2023

The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. The Predictability of Pipelines In stark contrast to ad-hoc queries, pipelines are where cost optimization efforts can yield significant dividends.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines.

Pipeline-centric 88

Pipeline-centric Professional Services Datasets BI 88

Monte Carlo

MARCH 28, 2024

In a nutshell, DataOps engineers are responsible not only for designing and building data pipelines, but iterating on them via automation and collaboration as well. So, does this mean you should choose DataOps engineering vs. data engineering when considering your next career move? What does a DataOps engineer do? It depends!

Engineering 75

Engineering Pipeline-centric BI Google Cloud 75

Cloudera

JANUARY 24, 2023

To this end, UBL embarked on a data analytics project that would achieve its goals for an improved data environment. Next, it needed to enhance the company’s customer-centric approach for a needs-based alignment of products and services. Mr. Kashif Riaz, head of data and AI at UBL, shared his thoughts on this project. “To

Banking 80

Banking Data Analytics Pipeline-centric IT 80

DataKitchen

NOVEMBER 5, 2024

When data reaches the Gold layer, it is highly curated and structured, offering a single version of the truth for decision-makers across the organization. We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Data Engineering Podcast

AUGUST 13, 2022

In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process. Data stacks are becoming more and more complex. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Ascend.io

JUNE 21, 2023

Cost Savings : Insights derived from data products can directly lead to cost savings in the business. Risk Reduction : Data-centric decision-making can also contribute to risk reduction. In the case of data products, these are networks of data pipelines, which makes them an integral part of your modern operational machinery.

Pipeline-centric 52

Pipeline-centric Data Pipeline Metadata Data 52

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional responsible for designing, implementing, and managing data solutions using Microsoft's Azure cloud platform. They work with various Azure services and tools to build scalable, efficient, and reliable data pipelines, data storage solutions, and data processing systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. The next problem will be the diversity of these mini data platforms (because of the configuration) and you even go deeper in problems with managing different technologies or version.

Technology 130

Technology Architecture Google Cloud Metadata 130

Knowledge Hut

JUNE 20, 2023

Factors Data Engineer Machine Learning Definition Data engineers create, maintain, and optimize data infrastructure for data. In addition, they are responsible for developing pipelines that turn raw data into formats that data consumers can use easily. Assess the needs and goals of the business.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. dbt, as of today, is the leading framework.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. dbt, as of today, is the leading framework.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 93

Process Pipeline-centric Media Metadata 93

Snowflake

JUNE 28, 2023

Streamlit gives data scientists and Python developers the ability to quickly turn data and models into interactive, enterprise-ready applications. Simplified streaming pipelines in Snowflake We are expanding our streaming capabilities with Dynamic Tables (public preview). We also announced the private preview of Snowflake CLI.

Pipeline-centric 59

Pipeline-centric Programming Language Python Government 59

Databand.ai

JULY 19, 2023

This helps to enhance data quality, facilitate data governance, and enable regulatory compliance. Notable Data Lineage Tools and Solutions Let’s take a look at several notable data lineage tools that can improve the quality and efficiency of your data pipeline.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

The Modern Data Company

DECEMBER 29, 2022

This means moving beyond product-centric thinking to a data-driven customer experience model that’s consistent across all channels. Next, the wealth management industry is also shifting away from a product focus to a client-centric model. DataOS is the world’s first operating system.

Banking 52

Banking Pipeline-centric Building Finance 52

Pipeline Data Engineering

MARCH 2, 2022

Analytics Stacks for Startups Jan Katins, Senior IT Consultant/Data Engineer, kreuzwerker GmbH The stack should be relatively fast to implement (two weeks is possible), so you can quickly reap the benefits of having a data warehouse and BI Tooling in place or upload enriched data back to operational systems.

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

Ripple Engineering

JANUARY 29, 2024

  A lack of a centralized system makes building a single source of high-quality data difficult. The key aspect of any business-centric team in delivering products and features is to make critical decisions on ensuring low latency, high throughput, cost-effective storage, and highly efficient infrastructure.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Snowflake

JUNE 28, 2023

Snowpark is our secure deployment and processing of non-SQL code, consisting of two layers: Familiar Client Side Libraries – Snowpark brings deeply integrated, DataFrame-style programming and OSS compatible APIs to the languages data practitioners like to use. Previously, tasks could be executed as quickly as 1-minute.

Python 52

Python Accessibility Accessible Pipeline-centric 52

Monte Carlo

NOVEMBER 14, 2023

This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. According to one Redditor on r/dataengineering, “Seems like 99/100 data engineering jobs mention Airflow.”

Pipeline-centric 40

Pipeline-centric Google Cloud Python Data Workflow 40

Monte Carlo

JULY 15, 2024

For example, as a data owner in a retail company, your analysis of customer purchasing patterns could inform product development and marketing strategies. Career advancement: As organizations become more data-centric, your role as a data owner offers opportunities for career growth.

Pipeline-centric 52

Pipeline-centric Government Data Governance Retail 52

Dataquest

JANUARY 25, 2017

A data scientist is only as good as the data they have access to. Most companies store their data in variety of formats across databases and text files. This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. Open question: how to seed data in a staging environment? Test system with A/A test. Be adaptable.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

DataKitchen

AUGUST 27, 2021

Without DataOps, companies can employ hundreds of data professionals and still struggle. The data pipelines must contend with a high level of complexity – over seventy data sources and a variety of cadences, including daily/weekly updates and builds. That’s the power of DataOps automation.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Analytics Data Lake 98