Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its im

BI 208

BI SQL Machine Learning Data 208

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently. Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.

Data Engineering 163

Data Engineering Data Engineer Engineering Data Pipeline 163

Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. It is interesting to see the general public having the same amount of excitement as there was a year ago in the LLM space.

Data Engineering 147

Data Engineering Data Engineer Engineering NoSQL 147

KDnuggets

SEPTEMBER 10, 2023

Back to Basics: Understanding key data management principles that data scientists should know.

Data Management 147

Data Management Data Science Management Data 147

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

databricks

SEPTEMBER 12, 2023

Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval.

Machine Learning 145

Machine Learning Engineering 145

Christophe Blefari

SEPTEMBER 15, 2023

Facing the News ( credits ) Hello Data News readers. I'm still struggling to get back into my usual work rhythm. If you add the fact that last week I came up with fewer articles than I expected, this has led me to another blank page. Anyway, after 2 years of work, I have to accept and let go when necessary. But don't worry I don't forget you.

Data Warehouse 130

Data Warehouse Data SQL Python 130

KDnuggets

SEPTEMBER 13, 2023

Getting Started with SQL in 5 Steps • Introduction to Databases in Data Science • Time 100 AI: The Most Influential?

Data Science 142

Data Science Database SQL Data 142

databricks

SEPTEMBER 14, 2023

As part of MLflow 2’s support for LLMOps, we are excited to introduce the latest updates to support prompt engineering in MLflow 2.7. A.

Engineering 134

Engineering Machine Learning 134

Waitingforcode

SEPTEMBER 14, 2023

I won't hide it, I'm still a fresher in the Apache Flink world and despite my past streaming experiences with Apache Spark Structured Streaming and GCP Dataflow, I need to learn. And to learn a new tool or concept, there is nothing better than watching some conference talks!

IT 130

IT 130

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Seattle Data Guy

SEPTEMBER 13, 2023

Success in the data world hinges on team setup. I’ve delved into onboarding and standards in previous articles, but never into the structure of data teams. Typically, there are three configurations: Centralized, Decentralized, and Federated. Most companies I’ve seen use a mix of these. While the newest tech breakthroughs grab headlines, team organization is the… Read more The post How To Set Up Your Data Analytics Team For Success – Centralized vs Decentralized vs Federat

Data Analytics 130

Data Analytics Data 130

KDnuggets

SEPTEMBER 12, 2023

With a shortage of talent and an abundance of opportunity, there’s never been a better time to launch or advance your data science career with the SAS Academy for Data Science. Read on to find out everything you need to become a SAS Certified Data Scientist.

Data Science 142

Data Science Data 142

ArcGIS

SEPTEMBER 14, 2023

Updated data from the Watershed Boundary Dataset (WBD) are added to Living Atlas as new feature services.

Datasets 128

Datasets Data 128

databricks

SEPTEMBER 15, 2023

Special thanks to our partners at Immuta, Alation, and Anomalo for their collaboration on the content and technical assets from this article. The.

Data 117

Data 117

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Snowflake

SEPTEMBER 13, 2023

Building your data product is only the beginning. You’ve considered a wide variety of use cases, and settled on the one you’ll focus on. Maybe you’re going to help hospitals predict emergency room visits and optimize their staffing. Or you’re going to enable restaurants to reduce their food waste. Or maybe you just have some really unique data that you think might be of use to someone.

Food 115

Food Retail Hospitality Data 115

KDnuggets

SEPTEMBER 14, 2023

Mastering the Basics of Linear Regression and Fundamentals of Gradient Descent and Loss Minimization.

Machine Learning 140

Machine Learning 140

ThoughtSpot

SEPTEMBER 14, 2023

We’re thrilled to announce that both ThoughtSpot and Mode ( acquired by ThoughtSpot in July 2023 ) have been recognized as Leaders in Snowflake's recent Modern Marketing Data Stack report! Given the ever-evolving landscape of modern data analytics products, organizations are looking to ThoughtSpot and Mode when seeking innovative solutions—helping them harness the power of their marketing data.

Business Intelligence 111

Business Intelligence Business Analyst BI Data Analytics 111

databricks

SEPTEMBER 12, 2023

As the lakehouse becomes increasingly mission-critical to data-forward organizations, so too grows the risk that unexpected events, outages, and security incidents may derail.

Systems 110

Systems Data 110

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

SEPTEMBER 14, 2023

More than 50% of data leaders recently surveyed by BCG said the complexity of their data architecture is a significant pain point in their enterprise. Companies hampered by legacy data architectures are often plagued by a high total cost of ownership (TCO), an inability to govern data, and a lack of scalability as their data volumes grow. “As a result,” says BCG, “many companies find themselves at a tipping point, at risk of drowning in a deluge of data, overburdened with complexity and costs.

Data Architecture 113

Data Architecture Architecture Hadoop Data Warehouse 113

KDnuggets

SEPTEMBER 12, 2023

High-level exploration of the role of statistics in data science.

Data Science 137

Data Science Data 137

Pinterest Engineering

SEPTEMBER 12, 2023

Raymond Lee | Software Engineer II; Qingxian Lai | Sr. Software Engineer; Karthik Anantha Padmanabhan | Manager II, Engineering; Se Won Jang | Manager II, Engineering Photo by Claudio Schwarz on Unsplash Our mission at Pinterest is to bring everyone the inspiration to create the life they love. Machine Learning plays a crucial role in this mission. It allows us to continuously deliver high-quality inspiration to our 460 million monthly active users, curated from billions of pins on our platform.

Data Process 105

Data Process Process Datasets Software Engineer 105

databricks

SEPTEMBER 15, 2023

Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our s.

Engineering 105

Engineering 105

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

Snowflake

SEPTEMBER 12, 2023

The first annual Data Cloud Industry Day is here! Data Cloud Industry Day 2023 is a free virtual event on September 28, 2023, dedicated to what’s possible for you and your industry in the world of data. From leading-edge innovations to seamless solutions to your toughest industry-specific challenges, Industry Day provides the insights and information you need to drive business value with your data.

Cloud 106

Cloud Manufacturing Retail Healthcare 106

KDnuggets

SEPTEMBER 12, 2023

Understanding Machine Learning: Exposing the Tasks, Algorithms, and Selecting the Best Model.

Machine Learning 136

Machine Learning Algorithm 136

Towards Data Science

SEPTEMBER 14, 2023

Here’s why you should avoid representing paths as strings and use Pathlib instead Continue reading on Towards Data Science »

Python 98

Python Data Science Data Technology 98

databricks

SEPTEMBER 13, 2023

As data enthusiasts, we love uncovering stories in datasets. With Posit’s RStudio Desktop and Databricks Lakehouse, you can analyze data with dplyr, create i.

Datasets 105

Datasets Data 105

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Snowflake

SEPTEMBER 14, 2023

Data is essential to marketing. It’s how we know our audience and measure campaign outcomes. It shows us where to adjust a campaign on the fly, for even better results. But working with data is increasingly complex, and having the right stack of technologies is invaluable. To help marketers understand the rapidly changing world of data technology, we’ve just released our second annual Modern Marketing Data Stack report.

Media 100

Media Technology Datasets Machine Learning 100

KDnuggets

SEPTEMBER 14, 2023

KDnuggets reviews a diverse set of 5 AI tools to help maximize your productivity. Have a look and see what our recommendations include.

135

135

Towards Data Science

SEPTEMBER 10, 2023

Realities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale Image by Cosmin Paduraru As data teams scale up on the cloud, data platform teams need to ensure the workloads they are responsible for are meeting business objectives. At scale with dozens of data engineers building hundreds of production jobs, controlling their performance at scale is untenable for a myriad of reasons from technical to human.

Data Pipeline 98

Data Pipeline Data Coding Utilities 98