Waitingforcode
SEPTEMBER 6, 2023
In our social media and marketing-driven era, it's quite hard to get things right. For me there is one common misconception brought by the Modern Data Stack idea that everything should be now ELT. In fact no, it shouldn't but only can.
Data Engineering Podcast
SEPTEMBER 3, 2023
Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. The challenge is that most of those options are complex to operate and exist in their own silo. The dlt project was created to eliminate overhead and bring data integration into your full control as a library component of your overall data system.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
SEPTEMBER 3, 2023
This tutorial covers Python's foundational data structures - lists, tuples, dictionaries, and sets. Learn their characteristics, use cases, and practical examples, all in 5 steps.
Engineering at Meta
SEPTEMBER 7, 2023
Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We wanted to build a decentralized (or federated) app that would enable people to post content that is viewable by anyone on other social apps, and vice versa.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Snowflake
SEPTEMBER 7, 2023
Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Functions or procedures written by users in these languages are executed inside of Snowpark’s secure sandbox environment , which runs on the warehouse.
Seattle Data Guy
SEPTEMBER 2, 2023
SSIS, short for SQL Server Integration Service, is an essential data migration tool for modern businesses. As a key part of Microsoft’s SQL database software, It allows you to easily complete many complex tasks, including data extraction, merging data, loading and transformation, aggregating data, and more. It’s a comprehensive solution to your data management needs.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
ArcGIS
SEPTEMBER 5, 2023
Access a vast collection of satellite imagery with new EarthCache add-In for ArcGIS Pro.
Snowflake
SEPTEMBER 5, 2023
Companies want to train and use large language models (LLMs) with their own proprietary data. Open source generative models such as Meta’s Llama 2 are pivotal in making that possible. The next hurdle is finding a platform to harness the power of LLMs. Snowflake lets you apply near-magical generative AI transformations to your data all in Python, with the protection of its out-of-the-box governance and security features.
Knowledge Hut
SEPTEMBER 6, 2023
As a seasoned software developer with almost a decade of experience in the tech industry, I vividly remember the excitement of taking my first web development course. Back then, I was just starting my journey as a front-end web developer, and that course was a stepping-stone that transformed my career. Today, I am thrilled to share my insights on some of the top software development courses available, hoping to empower aspiring developers like you to find the perfect path to success.
KDnuggets
SEPTEMBER 2, 2023
Want to learn Python? Get started today by learning Python's syntax, supported data types, and control structures.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Confluent
SEPTEMBER 7, 2023
Kafka-on-Windows tutorials are everywhere, but most run Kafka directly on Windows.
Engineering at Meta
SEPTEMBER 7, 2023
We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. Extracting maximum performance from an AI cluster and increasing overall efficiency warrants a multi-input system that accounts for various hardware and software parameters across compute, storage, and network collectively.
databricks
SEPTEMBER 7, 2023
Check out our Solution Accelerator for RFM Segmentation for more details and to download the notebooks. For retail brands, effective customer engagement depends.
KDnuggets
SEPTEMBER 8, 2023
Understand the relevance of databases in data science. Also learn the fundamentals of relational databases, NoSQL database categories, and more.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
ThoughtSpot
SEPTEMBER 6, 2023
In a survey by the Harvard Business Review , 87% of respondents stated their organizations would be more successful if frontline workers were empowered to make important decisions in the moment. And 86% of respondents stated that they needed better technology to enable those in-the-moment decisions. Those coveted insights live at the end of a process lovingly known as the data pipeline.
Engineering at Meta
SEPTEMBER 7, 2023
Meta presents Chakra execution traces , an open graph-based representation of AI/ML workload execution, laying the foundation for benchmarking and network performance optimization. Chakra execution traces represent key operations, such as compute, memory, and communication, data and control dependencies, timing, and resource constraints. In collaboration with MLCommons , we are seeking industry-wide adoption for benchmarking.
databricks
SEPTEMBER 5, 2023
In May this year, we opened our latest development center in Bengaluru, India. We've been busy building out our R&D teams in India.
KDnuggets
SEPTEMBER 5, 2023
This step-by-step tutorial is for beginners to guide them through the process of data cleaning and preprocessing using the powerful Pandas library.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Snowflake
SEPTEMBER 6, 2023
In today’s hyper-connected world of retail and consumer goods, understanding the customer journey is more critical than ever. As digital disruption and evolving customer expectations continue to shape the future of these sectors, organizations are striving to achieve ‘Shopper 360,’ a comprehensive and integrated view of their shoppers that is the retail equivalent of ‘Customer 360.
Pinterest Engineering
SEPTEMBER 5, 2023
Pong Eksombatchai | Principal Engineer; Karthik Anantha Padmanabhan | Manager II, Engineering Image from [link] Pinterest’s mission is to bring everyone the inspiration to create a life they love. We rely on an extensive suite of AI powered products to connect over 460M users to hundreds of billions of Pins, resulting in hundreds of millions of ML inferences per second, hundreds of thousands of ML training jobs per month by just a couple of hundreds of ML engineers.
databricks
SEPTEMBER 4, 2023
We are thrilled to announce great enhancements to onboard more workloads to Unity Catalog clusters in shared access mode, Databricks' highly efficient, secure.
KDnuggets
SEPTEMBER 7, 2023
We have a bad case of generative AI fatigue.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Snowflake
SEPTEMBER 7, 2023
Higher education institutions have a lot of plates to spin, and the University of Birmingham is no exception. Following a tough pandemic, the need to digitally transform had never been more pressing. The university needed to modernize its data capabilities to better serve staff, students and researchers—and it used the Snowflake Data Cloud to do it.
Towards Data Science
SEPTEMBER 8, 2023
Get more data in less time Photo by Wesley Tingey on Unsplash We live in a world of big data. Often, big data is organized as a large collection of small datasets (i.e., one large dataset comprised of multiple files). Obtaining these data is often frustrating because of the download (or acquisition burden). Fortunately, with a little code, there are ways to automate and speed-up file download and acquisition.
Confluent
SEPTEMBER 5, 2023
Want to run Kafka on AWS? Our full tutorial provides expert recommendations on how to deploy, monitor, and manage Kafka clusters on AWS.
KDnuggets
SEPTEMBER 5, 2023
From cleaning data to wowing recruiters - this blog shares 5 killer data science projects to launch your data science career and get hired!
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Snowflake
SEPTEMBER 5, 2023
Snowflake’s fully managed platform helps minimize TCO by achieving faster time to insights and production, decreasing unplanned downtime and operational risks, and reducing business costs through customers paying only for actual usage. Snowflake also eliminates software license fees and recovers storage and server costs. Additionally, Snowflake reduces infrastructure costs, administrative efforts and maintenance so you can reallocate technology resources to higher-value business priorities.
Knowledge Hut
SEPTEMBER 6, 2023
Scrum Alliance training is crucial when it comes to proving competency in project management practices. A good Scrum Alliance certification can imminently help you to excel in your career. It is a versatile Agile Project Management framework suitable for any industry. Scrum Alliance certifications not only help to improve an organization's productivity but are also widely responsible for improving product qualities, risk mitigation, and robust team dynamics.
Towards Data Science
SEPTEMBER 5, 2023
Establishing data pipelines towards automated spatial data science Continue reading on Towards Data Science »
KDnuggets
SEPTEMBER 5, 2023
Build a streaming data pipeline using Formula 1 data, Python, Kafka, RisingWave as the streaming database, and visualize all the real-time data in Grafana.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
228 
Let's personalize your content