Confessions of a Data Guy
OCTOBER 7, 2024
This is an interesting one indeed, it’s one that teases and puzzles the brain to no end. Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home? Can we say that after all these decades the Data Warehouse Toolkit […] The post The Death of the Data Warehouse, replaced by the Lake House.
The Pragmatic Engineer
OCTOBER 10, 2024
Automattic, creator of Wordpress, is being sued by one of the largest WordPress hosting providers. The conflict fits into a trend of billion-dollar companies struggling to effectively monetize open source, and are changing tactics to limit their competition and increase their revenue. This article was originally published a week ago, on 3 October 2024, in The Pragmatic Engineer.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Analytics Vidhya
OCTOBER 7, 2024
Introduction Today, data systems evolve quickly, demanding efficient monitoring and response. Real-time change detection is essential to keeping systems stable, preventing failures, and ensuring business continuity. Microsoft’s open-source tool, Drasi, addresses this need by effortlessly detecting, monitoring, and responding to data changes across platforms, including relational and graph databases.
KDnuggets
OCTOBER 8, 2024
Are you a data science beginner looking to build your portfolio? Start working on these projects today.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
databricks
OCTOBER 8, 2024
Summary Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS.
Yelp Engineering
OCTOBER 6, 2024
The Yelp Reservations service (yelp_res) is the service that powers reservations on Yelp. It was acquired along with Seatme in 2013, and is a Django service and webapp. It powers the reservation backend and logic for Yelp Guest Manager, our iPad app for restaurants, and handles diner and partner flows that create reservations. Along with that, it serves a web UI and backend API for our Yelp Reservations app, which has been superseded by Yelp Guest Manager but is still used by many of our restaur
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
KDnuggets
OCTOBER 7, 2024
Read this overview of prompting techniques, challenges, and best practices to help you master this essential AI skill.
databricks
OCTOBER 8, 2024
Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The.
DataKitchen
OCTOBER 10, 2024
Read Our New White Paper: Data Quality The DataOps Way Data quality isn’t just a technical hurdle—it’s a strategic necessity in the data-driven world. Traditional methods fall short, but the DataOps approach to data quality offers a transformative path forward. It empowers individuals to act swiftly, enables continuous improvement, and fosters collaboration across organizational silos.
ArcGIS
OCTOBER 8, 2024
The response digital twin to assist disaster management of North Rhine-Westphalia illustrates how to create and maintain 3D mesh data.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
OCTOBER 11, 2024
Dive into this emerging and powerful LLM-based AI tool for enhancing your business, creative, or daily processes through well-managed conversations.
databricks
OCTOBER 8, 2024
We consistently hear from our customers that one of the headwinds to transitioning Generative AI applications from pilot to production is the accuracy.
Data Engineering Weekly
OCTOBER 11, 2024
The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent. As we grapple with these, another seismic shift is upon us—the rise of Large Language Models (LLMs).
Yelp Engineering
OCTOBER 7, 2024
In 2019, Yelp’s Core Android team led an effort to boost navigation performance in Yelp’s Consumer app. We switched from building screens with multiple separate activities to using fragments inside a single activity. In this blog post, we’ll cover our solution, how we approached the migration and share learnings from along the way as well as performance wins.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
KDnuggets
OCTOBER 9, 2024
Acquire a transversal understanding of high-relevance AI jargon in the time it takes to drink a cup of coffee.
databricks
OCTOBER 10, 2024
Today, we are excited to announce the general availability of Databricks Assistant Autocomplete on all cloud platforms. Assistant Autocomplete provides personalized AI-powered code.
Snowflake
OCTOBER 7, 2024
When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Common issues include redundant efforts between development and production teams, as well as inconsistencies between the features used in training and those in the serving stack, which can lead to decreased performance.
Netflix Tech
OCTOBER 8, 2024
Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital. In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architectu
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
KDnuggets
OCTOBER 8, 2024
Tired of fixing the same deployment issues? Learn how Docker can keep your ML models running smoothly, every time.
databricks
OCTOBER 7, 2024
We are announcing the General Availability of Provider Usage Analytics for Databricks Marketplace providers. This feature lets you analyze lead generation and product.
Robinhood
OCTOBER 9, 2024
Robinhood Retirement has now reached $10 billion in Assets Under Custody across nearly one million funded retirement accounts. We launched Robinhood Retirement in January 2023 with the goal of making investing for the future easy and accessible for all. Importantly, Robinhood introduced the first IRA with a match – democratizing access to retirement by offering a match that does not require a traditional employer.
Towards Data Science
OCTOBER 5, 2024
How to Instantly Detect Data Quality Issues and Identify their Causes Continue reading on Towards Data Science »
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
KDnuggets
OCTOBER 7, 2024
Combining LLM reasoning for text-based models in Scikit-Learn.
databricks
OCTOBER 9, 2024
With the pace of modern business and the competitive need for more and more data, organizations now correctly ask whether their data management.
Cloudera
OCTOBER 8, 2024
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices , part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Background The generative AI landscape is evolving at a rapid pace, marked by explosive growth and widespread adoption across industries.
Towards Data Science
OCTOBER 9, 2024
Get started with Airbyte and Cloud Storage Coding the connectors yourself? Think very carefully Creating and maintaining a data platform is a hard challenge. Not only do you have to make it scalable and useful, but every architectural decision builds up over time. Data connectors are an essential part of such a platform. Of course, how else are we going to get the data?
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
OCTOBER 10, 2024
NotebookLM makes it easy to create study guides from YouTube videos by using AI to summarize and organize key points. Just upload the video link, and the tool helps you turn the content into a structured guide.
databricks
OCTOBER 9, 2024
In industries like finance and retail, vast data is leveraged to generate billions in profits. Yet, in healthcare, the struggle to access critical.
Confluent
OCTOBER 11, 2024
Learn how to leverage event design to make eventual bad data in your event streams easier to repair, and also what to do when you have a contaminated stream.
Pinterest Engineering
OCTOBER 11, 2024
Alex Wang; Software Engineer I | Lei Pan; Software Engineer II | Raymond Lee; Senior Software Engineer | Saurabh Vishwas Joshi; Senior Staff Software Engineer | Chia-Wei Chen; Senior Software Engineer | Introduction In Part 1 of our blog series, we discussed why we chose to use Ray(™) as a last mile data processing framework and how it enabled us to solve critical business problems.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
113 
Let's personalize your content