Waitingforcode

AUGUST 17, 2023

Lessons learned why it's always worth checking the code implementation to avoid surprises later. Even for vendor-supported solutions.

Coding 130

Coding IT 130

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Machine Learning 130

KDnuggets

AUGUST 18, 2023

The improvement you need to get the best result from GPT.

Python 108

Python 108

ArcGIS

AUGUST 15, 2023

Summer internship experience with the Raster Analysis team at Esri- experience the world of GIS with Rakibul Ahasan.

98

98

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Towards Data Science

AUGUST 15, 2023

The modern data stacks allow you to do things differently, not just at a larger scale. Take advantage of it. Photo by Massimo Botturi on Unsplash Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. But instead of pressing it to the wood and pulling the trigger, you turn it sideways and hit the nail just like you would as if it were a hammer.

Cloud 98

Cloud Metadata Machine Learning Media 98

databricks

AUGUST 14, 2023

One of the key challenges that organizations face when adopting the open data lakehouse is selecting the optimal format for their data. Among.

Data 98

Data Engineering 98

Engineering at Meta

AUGUST 15, 2023

Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with asyncio for per-process concurrency.

Python 94

Python Metadata Architecture Process 94

Towards Data Science

AUGUST 17, 2023

Unit testing dbt models has always been one of the most critical missing pieces of the dbt ecosystem.

Data Science 98

Data Science Data Big Data Data Engineering 98

databricks

AUGUST 13, 2023

Today we are announcing the general availability of Delta Live Tables (DLT) on Google Cloud. DLT pipelines empower data engineers to build reliable.

Google Cloud 98

Google Cloud Cloud Data Engineering Data Engineer 98

KDnuggets

AUGUST 17, 2023

Integrating Open Source LLMs and LangChain for Free Generative Question Answering (No API Key required).

108

108

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Knowledge Hut

AUGUST 17, 2023

Staying current with rapidly advancing technology holds significant importance. Obtaining the latest certifications can enhance your professional standing by providing you with sought-after skills, thereby increasing your attractiveness to potential employers. It's noteworthy that AXELOS is a renowned authority in awarding certifications across a diverse spectrum of IT domains.

Certification 90

Certification Portfolio Consulting Business Analyst 90

Monte Carlo

AUGUST 15, 2023

Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. But instead of pressing it to the wood and pulling the trigger, you turn it sideways and hit the nail with the gun as if it were a hammer. You would probably think it’s expensive and not overly effective, while the site’s inspector is going to rightly view it as a safety hazard.

Data Storage 88

Data Storage Cloud Metadata Machine Learning 88

databricks

AUGUST 16, 2023

Thousands of Databricks customers use Databricks Workflows every day to orchestrate business critical workloads on the Databricks Lakehouse Platform. As is often the.

98

98

KDnuggets

AUGUST 18, 2023

"You may also like" is a simple phrase that implies a new era in the way businesses interact and connect with their customers, and graph databases can easily help to build recommendation engines.

Database 108

Database Building Engineering Data Engineering 108

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Lyft Engineering

AUGUST 14, 2023

Where’s My Data — A Unique Encounter with Flink Streaming’s Kinesis Connector For years now, Lyft has not only been a proponent of but also a contributor to Apache Flink. Lyft’s pipelines have evolved drastically over the years , yet, time and time again, we run into unique cases that stretch Flink to its breaking points — this is one of those times.

Data 85

Data Engineering Process Management 85

DoorDash Engineering

AUGUST 15, 2023

As DoorDash experienced rapid growth over the last few years, we began to see the limits of our traditional methods of monitoring. Metrics, logs, and traces provide vital information about our service ecosystem. But these signals almost entirely rely on application-level instrumentation, which can leave gaps or conflicting semantics across different systems.

Bytes 84

Bytes PostgreSQL Coding Database 84

databricks

AUGUST 16, 2023

The Personalization Paradigm: Balancing Business Self-Service and Data Governance Personalization transforms businesses, shaping and reshaping the way brands connect with their audiences. Its.

Data Governance 97

Data Governance Government Data IT 97

KDnuggets

AUGUST 14, 2023

A speedy way to convert a long text to a short PowerPoint Presentation using only ChatGPT.

108

108

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Cloudera

AUGUST 17, 2023

Machine Learning (ML) is at the heart of the boom in AI Applications, revolutionizing various domains. From powering intelligent Large Language Model (LLM) based chatbots like ChatGPT and Bard , to enabling text-to-AI image generators like Stable Diffusion , ML continues to drive innovation. Its transformative impact advances multiple fields from genetics to medicine to finance.

Manufacturing 72

Manufacturing Finance Python Machine Learning 72

Netflix Tech

AUGUST 16, 2023

By Arthur Gonigberg , Argha C Plaintext Past When Zuul was designed and developed , there was an inherent assumption that connections were effectively free, given we weren’t using mutual TLS (mTLS). It’s built on top of Netty , using event loops for non-blocking execution of requests, one loop per core. To reduce contention among event loops, we created connection pools for each, keeping them completely independent.

Algorithm 72

Algorithm Utilities AWS Cloud Computing 72

Monte Carlo

AUGUST 15, 2023

Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. But instead of pressing it to the wood and pulling the trigger, you turn it sideways and hit the nail just like you would as if it were a hammer. You would probably think it’s expensive and not overly effective, while the site’s inspector is going to rightly view it as a safety hazard.

Cloud 69

Cloud Metadata Machine Learning Media 69

KDnuggets

AUGUST 15, 2023

In this free ebook, readers will learn how to employ data cleaning and preprocessing for data science using the Python ecosystem.

Data Science 108

Data Science Python Data 108

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Cloudera

AUGUST 16, 2023

Remember when we announced our redesigned partner program Cloudera Partner Network (CPN) last year? Our goal was to create a more competency-based approach and more comprehensive tools and support to help partners guide their customers adopting modern data strategies based on the Cloudera hybrid data platform. In addition, CPN helps our partners go to market faster, and provides industry-leading incentives and promotions aligned with partner business and sales models.

Telecommunication 65

Telecommunication Insurance Data Architect Manufacturing 65

Netflix Tech

AUGUST 17, 2023

By: Hamid Shahid , Laura Johnson , Tiffany Low Synopsis At Netflix, we have created millions of artwork to represent our titles. Each artwork tells a story about the title it represents. From our testing on promotional assets , we know which of these assets have performed well and which ones haven’t. Through this, our teams have developed an intuition of what visual and thematic artwork characteristics work well for what genres of titles.

Media 71

Media Algorithm Architecture Building 71

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow.

Data Pipeline 62

Data Pipeline Data Raw Data Database 62

KDnuggets

AUGUST 16, 2023

How to Use ChatGPT to Convert Text into a PowerPoint Presentation • Best Python Tools for Building Generative AI Applications Cheat Sheet • Data Scientists Need to Specialize to Survive the Tech Winter • Python Vector Databases and Vector Indexes: Architecting LLM Apps • How To Speed Up SQL Queries Using Indexes [Python Edition]

Python 108

Python Building SQL Database 108

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Ascend.io

AUGUST 18, 2023

Jupyter Notebooks have fundamentally revolutionized how data scientists approach their tasks. They offer an unparalleled environment for experimentation and visualization. Yet, there’s an interest in putting notebooks directly into production environments. While it’s great to take the ideas from notebooks and use them in real-world settings, trying to put the entire notebook directly into production as a code artifact can cause problems.

Data Pipeline 52

Data Pipeline Programming Language Data Science Coding 52

Knowledge Hut

AUGUST 18, 2023

In my early career, I knew that getting certified in AWS would be essential for success. Now that I have obtained multiple AWS certifications, I can vouch for their value to professionals & companies alike. With cloud computing becoming the new norm in today's marketplace, AWS certifications are nothing short of essential. From AWS Certified Solutions Architect to AWS Certified DevOps Engineer, there are many different paths to choose from as per your career goals & skill set.

AWS 52

AWS Certification Amazon Web Services Cloud 52

Confluent

AUGUST 18, 2023

Learn how Australia’s Alex Bank leveraged real-time streaming to create a data-driven, customer-focused banking experience.

Banking 52

Banking Data 52

KDnuggets

AUGUST 14, 2023

Vector databases enable fast similarity search and scale across data points. For LLM apps, vector indexes can simplify architecture over full vector databases by attaching vectors to existing storage. Choosing indexes vs databases depends on specialized needs, existing infrastructure, and broader enterprise requirements.

Database 108

Database Python Architecture Data 108

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer