Analytics Vidhya
JUNE 25, 2023
In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. Imagine being an online shopper who suddenly receives irrelevant recommendations. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.
The Pragmatic Engineer
JUNE 29, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of four topics from today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
JUNE 25, 2023
Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. SQLMesh was designed as a unifying tool that is simple to work with but powerful enough for large-scale transformations and complex projects.
KDnuggets
JUNE 30, 2023
Every job is at risk. Here’s how you can AI-proof your career.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Analytics Vidhya
JUNE 24, 2023
Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases. Properly configured data structures ensure a smoother workflow and prevent data loss or misplacement.
databricks
JUNE 29, 2023
Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
KDnuggets
JUNE 27, 2023
The k-Nearest Neighbors Classifier is a machine learning algorithm that assigns a new data point to the most common class among its k closest neighbors. In this tutorial, you will learn the basic steps of building and applying this classifier in Python.
Analytics Vidhya
JUNE 24, 2023
Introduction We had an amazing opportunity to learn from Mr. Pavan. He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Throughout the conversation, Mr. Pavan shares his journey, inspirations, challenges, and accomplishments. Thus, providing valuable insights into the field of data engineering. As we explore Mr.
Start Data Engineering
JUNE 30, 2023
1. Introduction 2. What is self-serve? 2.1. Components of a self-serve platform 3. Building a self-serve data platform 3.1. Creating dataset(s) 3.1.1. Gather requirements 3.1.2. Get data foundations right 3.2. Accessing data 3.3. Identify and remove dependencies 4. Conclusion 5. Further reading 6. References 1. Introduction Most companies want to build a self-serve data platform.
Waitingforcode
JUNE 30, 2023
Surprised? You shouldn't. I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure. But there is some excitement for learning-from scratch I miss.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Confessions of a Data Guy
JUNE 28, 2023
I’ve been a dog licking my wounds for some time now. Over on my Substack newsletter, I’ve been doing a small series on DSA (Data Structures and Algorithms). I tackled some of the easier stuff first, like Linked Lists, Binary Search, and the like. What’s more, I actually did most of it in Rust, since […] The post Exploring Graphs in Rust.
Christophe Blefari
JUNE 24, 2023
( credits ) Hey, this is the Data News. It's super hard to change habits, but it's how it is, the newsletter is going out on Saturday. I hope this edition finds you well. Summer is coming ☀️ Thank you all because we crossed the 3000 subscribers mark last week. Let's go for the 4000 before the end of the year 🤗 This is a almost-raw edition for this week.
KDnuggets
JUNE 27, 2023
And how approaches to better data management, version control, and experiment tracking can help build reproducible ML pipelines.
databricks
JUNE 27, 2023
Generative AI will have a transformative impact on every business. Databricks has been pioneering AI innovations for a decade, actively collaborating with thousands.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Lyft Engineering
JUNE 28, 2023
Written by Konstantin Gizdarski and Martin Liu at Lyft. In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. On the real-time front, LyftLearn supported real-time inference and input feature validation. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.
LinkedIn Engineering
JUNE 26, 2023
For the last several years, internal infrastructure at LinkedIn has been built around a self-service model, enabling developers to onboard themselves with minimal support. We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. For example, developers can provision Kafka topics, Espresso tables, Venice stores and more via Nuage , our internal cloud-like infra management platform.
KDnuggets
JUNE 29, 2023
This article provides a general overview of Stable Diffusion and focuses on building a basic understanding of how generative artificial intelligence works.
databricks
JUNE 27, 2023
Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Confluent
JUNE 28, 2023
Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.
ThoughtSpot
JUNE 27, 2023
Active listening is an admired and sought after skill in both the professional and personal sphere. After all, who doesn’t love to be heard? But what happens when we apply that mindset to the way our organizations solicit feedback and interact with our customers? We don’t have to make any assumptions to answer this question, because we have the data.
KDnuggets
JUNE 26, 2023
Learn more about the third-party plugins that OpenAI have rolled out to understand ChatGPTs in real-world use.
databricks
JUNE 27, 2023
Data teams face many challenges to quickly access the right data primarily due to data fragmentation, time and cost involved in consolidating data.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Snowflake
JUNE 27, 2023
At Snowflake Summit , we announced the public launch of the Snowflake Performance Index (“SPI”), an aggregate index for measuring real-world improvements in Snowflake performance experienced by customers over time. At Snowflake, our product philosophy revolves around continuously enhancing the performance of our product, particularly the core database engine.
ThoughtSpot
JUNE 26, 2023
At ThoughtSpot, we know how important it is for businesses of every size and industry to empower every knowledge worker with personalized, actionable data-driven insights. These insights are your secret sauce to making better business decisions, growing faster, and delivering customer experiences that keep people coming back for more. But how do you scale self-service analytics to business users without completely overwhelming your data teams?
KDnuggets
JUNE 29, 2023
Large language models are getting released left right and center, and if you want to understand them better you need to know about NLP. Here are 5 Free books to help you.
databricks
JUNE 27, 2023
We are thrilled to announce that materialized views and streaming tables are now publicly available in Databricks SQL on AWS and Azure. Streaming.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Snowflake
JUNE 30, 2023
Since launching this year’s contest in October, receiving hundreds of submissions, and completing three rounds of judging, the wait is over: Maxa is the 2023 Snowflake Startup Challenge grand prize winner! Maxa’s goal is to automate financial and operations ERP insights extremely fast and without requiring special skills. To make that happen, it leverages the breadth of the Snowflake platform to transform raw data from multiple financial and operational systems into a common data model that user
ThoughtSpot
JUNE 30, 2023
Pride is more than just a month-long celebration; it is a powerful movement that reminds us of the importance of equality, acceptance, and love. It is that special time of year for the global queer community to come together to celebrate, commemorate, and continue to push for progress. It’s no different here at ThoughtSpot. We believe in creating an inclusive environment where everyone feels seen, heard, and valued.
KDnuggets
JUNE 26, 2023
This list of the most commonly used machine learning algorithms in Python and R is intended to help novice engineers and enthusiasts get familiar with the most commonly used algorithms.
ArcGIS
JUNE 30, 2023
In June 2023, elevation layers have been updated with high-res datasets of England, New Zealand and USA
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
226 
Let's personalize your content