KDnuggets
JULY 17, 2023
The All Things Insights and marketing analytics and data science community completed an extensive survey covering what executives are thinking, how they’re spending and the issues and opportunities they face. Grab your free copy now.
The Pragmatic Engineer
JULY 20, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of five topics from today’s subscriber-only deepdive on Advice on how to sell a startup. To get full issues twice a week, subscribe here.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Waitingforcode
JULY 21, 2023
Starting from Apache Spark 3.2.0 is now possible to load an initial state of the arbitrary stateful pipelines. Even though the feature is easy to implement, it hides some interesting implementation details!
Start Data Engineering
JULY 20, 2023
1. Introduction 2. Sample project 3. Best practices 3.1. Use standard patterns that progressively transform your data 3.2. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5. Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Data Engineering Podcast
JULY 16, 2023
Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. Bob Muglia has had a front-row seat to many of the major shifts driven by technology over his career. In his recent book "Datapreneurs" he reflects on the people and businesses that he has known and worked with and how they relied on data to deliver valuable services and drive meaningful change.
Seattle Data Guy
JULY 16, 2023
The ETL & ELT tool market is experiencing continuous transformation, propelled by fluctuating pricing structures and the advent of inventive alternatives. This industry remains fiercely competitive due to these changing elements and a swiftly growing user base. In the following sections, we will explore four emerging alternatives to Fivetran. Of course, that is if you… Read more The post 4 Alternatives to Fivetran: The Evolving Dynamics of the ETL & ELT Tool Market appeared first
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
KDnuggets
JULY 21, 2023
Let's explore the journeys of SAS users who harnessed the power of SAS to unlock new opportunities and achieve their career goals.
Towards Data Science
JULY 21, 2023
Spinning up a data platform doesn’t have to be complicated. Here are the 5 must-have layers to drive data product adoption at scale. Image courtesy of author. We hope it doesn’t make your eyes water. Like bean dip and ogres , layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline.
databricks
JULY 18, 2023
Today, Meta released their latest state-of-the-art large language model (LLM) Llama 2 to open source for commercial use1. This is a significant development.
Confluent
JULY 18, 2023
Gain the easiest solution for data streaming and increase data flow to your platform through native integrations with Confluent Cloud and 120+ Kafka connectors.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
JULY 20, 2023
Unveiling GPT-4: Deciphering its impact on data science and exploring its strengths and boundaries.
ThoughtSpot
JULY 18, 2023
At ThoughtSpot, we believe making data accessible to every knowledge worker requires human-centered technology—an analytics experience that bridges the “language” barrier between technology and people. AI is the perfect compliment to search because it empowers organizations to analyze, understand, and act on data. In order to achieve this vision, we knew we’d need to work with some of the best, most innovative technology companies across the modern data stack —companies that put their users fir
databricks
JULY 18, 2023
We are excited to announce enhanced monitoring and observability features in Databricks Workflows. This includes a new real-time insights dashboard to see all.
Towards Data Science
JULY 18, 2023
5 Inspiring Learning Resources to Propel Your Skills and Expertise Continue reading on Towards Data Science »
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
KDnuggets
JULY 17, 2023
Putting the world to shame.
ArcGIS
JULY 17, 2023
This blog shares the resource to the ArcGIS Pro Learn Series about SAR satellite imagery.
databricks
JULY 19, 2023
Today, we’re excited to share that we’ve completed our acquisition of MosaicML, a leading platform for creating and customizing generative AI models for you.
Towards Data Science
JULY 17, 2023
Deep Dive Guide for When and How to Use 8 Types of SCD Continue reading on Towards Data Science »
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
KDnuggets
JULY 21, 2023
Don’t have any coding experience? Don’t worry. Check out this drag-and-drop tool that helps you to build your own customized LLM flows. And guess what, you don’t have to be a tech professional!
Pinterest Engineering
JULY 18, 2023
Brian Overstreet | Software Engineer, Observability; Humsheen Geo | Software Engineer, Observability Time series is a critical part of Observability at Pinterest, powering 60,000 alerts and 5,000 dashboards. A time series is an identifier with values where the values are associated with a timestamp. Given the widespread use and critical nature of time series, it’s important to give engineers the ability to adequately express what operations to perform on the time series in a readable, understand
databricks
JULY 21, 2023
Let's say a distributor reached out wanting to understand what factors are driving the sale of carbonated beverages from customers in their convenience.
Towards Data Science
JULY 18, 2023
Where and how to address problems with your data Continue reading on Towards Data Science »
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
KDnuggets
JULY 17, 2023
A guide to using ChatGPT for exploratory data analysis. Use ChatGPT to explore a dataset, generate visualizations, and gain insights.
Monte Carlo
JULY 19, 2023
Building a data stack doesn’t have to be complicated. Here’s what data leaders say are the 5 must-have layers of your data platform to drive data adoption – and ROI – across your business. Like bean dip and ogres , layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline.
databricks
JULY 21, 2023
This is part seven of a multi-part series to share key insights and tactics with Senior Executives leading data and AI transformation initiatives.
ArcGIS
JULY 18, 2023
The purpose is to learn what network diagram storage means and provide guidance to avoid unnecessarily increasing database sizes.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
JULY 19, 2023
This article explores the technical details and implications of Meta's newly released Llama 2, a large language model that promises to revolutionize the field of generative AI. We delve into its capabilities, performance, and potential applications, while also discussing its open-source nature and the company's commitment to safety and transparency.
Towards Data Science
JULY 19, 2023
A deep dive into MapReduce and parallelization Continue reading on Towards Data Science »
databricks
JULY 17, 2023
For over a year now, Databricks and dbt Labs have been working together to realize the vision of simplified real-time analytics engineering, combining.
Engineering at Meta
JULY 17, 2023
Meta has made it possible for people to upload high dynamic range (HDR) videos from their phone’s camera roll to Reels on Facebook and Instagram. To show standard dynamic range (SDR) UI elements and overlays legibly on top of HDR video, we render them at a brightness level comparable to the video itself. We solved various technical challenges to ensure a smooth transition to HDR video across the diverse range of old and new devices that people use to interact with our services every day.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
104 
Let's personalize your content