Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its im

BI 208

BI SQL Data Machine Learning 208

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently. Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

KDnuggets

SEPTEMBER 10, 2023

Back to Basics: Understanding key data management principles that data scientists should know.

Data Management 153

Data Management Data Science Management Data 153

Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. It is interesting to see the general public having the same amount of excitement as there was a year ago in the LLM space.

Data Engineer 147

Data Engineer Data Engineering Engineering NoSQL 147

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

databricks

SEPTEMBER 12, 2023

Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval.

Machine Learning 145

Machine Learning Engineering 145

Engineering at Meta

SEPTEMBER 12, 2023

Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products. In 2021, we discovered a vulnerability in the Meta Quest 2’s Android-based OS that never made it to production but helped us find new ways to improve the security of Meta Quest products. We’re sharing our journey to get arbitrary native code execution in the privileged VR Runtime service on the Meta Quest 2 by exploiting a memory corruption v

Bytes 138

Bytes Coding Programming Process 138

Christophe Blefari

SEPTEMBER 15, 2023

Facing the News ( credits ) Hello Data News readers. I'm still struggling to get back into my usual work rhythm. If you add the fact that last week I came up with fewer articles than I expected, this has led me to another blank page. Anyway, after 2 years of work, I have to accept and let go when necessary. But don't worry I don't forget you.

Data Warehouse 130

Data Warehouse Data SQL Python 130

databricks

SEPTEMBER 14, 2023

As part of MLflow 2’s support for LLMOps, we are excited to introduce the latest updates to support prompt engineering in MLflow 2.7. A.

Engineering 134

Engineering Machine Learning 134

ArcGIS

SEPTEMBER 14, 2023

Updated data from the Watershed Boundary Dataset (WBD) are added to Living Atlas as new feature services.

Datasets 130

Datasets Data 130

KDnuggets

SEPTEMBER 12, 2023

High-level exploration of the role of statistics in data science.

Data Science 152

Data Science Data 152

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Waitingforcode

SEPTEMBER 14, 2023

I won't hide it, I'm still a fresher in the Apache Flink world and despite my past streaming experiences with Apache Spark Structured Streaming and GCP Dataflow, I need to learn. And to learn a new tool or concept, there is nothing better than watching some conference talks!

IT 130

IT 130

Seattle Data Guy

SEPTEMBER 13, 2023

Success in the data world hinges on team setup. I’ve delved into onboarding and standards in previous articles, but never into the structure of data teams. Typically, there are three configurations: Centralized, Decentralized, and Federated. Most companies I’ve seen use a mix of these. While the newest tech breakthroughs grab headlines, team organization is the… Read more The post How To Set Up Your Data Analytics Team For Success – Centralized vs Decentralized vs Federat

Data Analytics 130

Data Analytics Data 130

databricks

SEPTEMBER 15, 2023

Special thanks to our partners at Immuta, Alation, and Anomalo for their collaboration on the content and technical assets from this article. The.

Data 118

Data 118

KDnuggets

SEPTEMBER 11, 2023

Where do you start in a field as vast as big data? Which tools and techniques to use? We explore this and talk about the most common tools in big data.

Big Data Tools 152

Big Data Tools Big Data Data Data Engineer 152

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Snowflake

SEPTEMBER 13, 2023

Snowflake and Salesforce have built on our existing partnership to unify the full breadth of customer and business data and generate actionable insights for our customers. We are happy to announce the general availability of Bring Your Own Lake (BYOL) Data Sharing with the Snowflake Data Cloud from Salesforce Data Cloud. Organizations can now leverage Salesforce data directly in Snowflake via zero-ETL data sharing to accelerate decision-making and help streamline business processes.

Finance 112

Finance AWS Accessibility Accessible 112

ThoughtSpot

SEPTEMBER 14, 2023

We’re thrilled to announce that both ThoughtSpot and Mode ( acquired by ThoughtSpot in July 2023 ) have been recognized as Leaders in Snowflake's recent Modern Marketing Data Stack report! Given the ever-evolving landscape of modern data analytics products, organizations are looking to ThoughtSpot and Mode when seeking innovative solutions—helping them harness the power of their marketing data.

Business Intelligence 111

Business Intelligence Business Analyst BI Data Analytics 111

databricks

SEPTEMBER 12, 2023

As the lakehouse becomes increasingly mission-critical to data-forward organizations, so too grows the risk that unexpected events, outages, and security incidents may derail.

Systems 110

Systems Data 110

KDnuggets

SEPTEMBER 11, 2023

The not so secret behind becoming a proficient programmer - Math & it’s top 10 concepts.

Programming 150

Programming 150

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

SEPTEMBER 13, 2023

Building your data product is only the beginning. You’ve considered a wide variety of use cases, and settled on the one you’ll focus on. Maybe you’re going to help hospitals predict emergency room visits and optimize their staffing. Or you’re going to enable restaurants to reduce their food waste. Or maybe you just have some really unique data that you think might be of use to someone.

Food 107

Food Retail Hospitality Data 107

Pinterest Engineering

SEPTEMBER 12, 2023

Raymond Lee | Software Engineer II; Qingxian Lai | Sr. Software Engineer; Karthik Anantha Padmanabhan | Manager II, Engineering; Se Won Jang | Manager II, Engineering Photo by Claudio Schwarz on Unsplash Our mission at Pinterest is to bring everyone the inspiration to create the life they love. Machine Learning plays a crucial role in this mission. It allows us to continuously deliver high-quality inspiration to our 460 million monthly active users, curated from billions of pins on our platform.

Data Process 103

Data Process Process Datasets Software Engineer 103

databricks

SEPTEMBER 15, 2023

Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our s.

Engineering 105

Engineering 105

KDnuggets

SEPTEMBER 12, 2023

Understanding Machine Learning: Exposing the Tasks, Algorithms, and Selecting the Best Model.

Machine Learning 149

Machine Learning Algorithm 149

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Snowflake

SEPTEMBER 14, 2023

More than 50% of data leaders recently surveyed by BCG said the complexity of their data architecture is a significant pain point in their enterprise. Companies hampered by legacy data architectures are often plagued by a high total cost of ownership (TCO), an inability to govern data, and a lack of scalability as their data volumes grow. “As a result,” says BCG, “many companies find themselves at a tipping point, at risk of drowning in a deluge of data, overburdened with complexity and costs.

Data Architecture 103

Data Architecture Architecture Hadoop Data Warehouse 103

Booking.com Engineering

SEPTEMBER 14, 2023

source Software development is all about change. And, over the lifespan of our software, the goal is to implement required changes in a reasonable amount of time. Whether the changes are technical in nature, like an urgent security upgrade, or stem from a business need, such as building a new feature to make us more competitive in target markets — how fast we can change is critical.

Coding 93

Coding Architecture Engineering Software Engineer 93

databricks

SEPTEMBER 13, 2023

As data enthusiasts, we love uncovering stories in datasets. With Posit’s RStudio Desktop and Databricks Lakehouse, you can analyze data with dplyr, create i.

Datasets 105

Datasets Data 105

KDnuggets

SEPTEMBER 9, 2023

This comprehensive SQL tutorial covers everything from setting up your SQL environment to mastering advanced concepts like joins, subqueries, and optimizing query performance. With step-by-step examples, this guide is perfect for beginners looking to enhance their data management skills.

SQL 148

SQL Data Management Management Data 148

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Snowflake

SEPTEMBER 12, 2023

The first annual Data Cloud Industry Day is here! Data Cloud Industry Day 2023 is a free virtual event on September 28, 2023, dedicated to what’s possible for you and your industry in the world of data. From leading-edge innovations to seamless solutions to your toughest industry-specific challenges, Industry Day provides the insights and information you need to drive business value with your data.

Cloud 96

Cloud Manufacturing Retail Healthcare 96

Cloudera

SEPTEMBER 13, 2023

I started my current career path with Hortonworks in 2016, back when we still had to tell people what Hadoop was. Once I got to work with all the amazing open-source Apache tools I was hooked. I found Apache NiFi especially interesting. Soon after, I became a huge fan of Apache Kafka. Coupled with amazing technology was an amazing team that only grew and improved with the merger with Cloudera.

Technology 91

Technology Hadoop Kafka Project 91

Workfall

SEPTEMBER 12, 2023

Reading Time: 11 minutes What is Socket.io? Socket.io , a widely-used JavaScript library, offers a framework for facilitating real-time, two-way communication between web clients (like browsers) and servers. It uses WebSockets as the primary communication method but also offers fallback options such as long polling for environments where WebSockets may not be supported.

Building 89

Building MongoDB Programming Language Coding 89

KDnuggets

SEPTEMBER 15, 2023

Explore the top 5 user-friendly platforms that provide free access to large language models, enabling you to experience the latest AI models firsthand.

Accessibility 147

Accessibility Accessible 147

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud