The Pragmatic Engineer
FEBRUARY 29, 2024
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of three topics from today’s subscriber-only The Pulse issue. Subscribe to get issues like this in your inbox, every week.
Analytics Vidhya
FEBRUARY 28, 2024
Introduction Data is fuel for the IT industry and the Data Science Project in today’s online world. IT industries rely heavily on real-time insights derived from streaming data sources. Handling and processing the streaming data is the hardest work for Data Analysis. We know that streaming data is data that is emitted at high volume […] The post Kafka to MongoDB: Building a Streamlined Data Pipeline appeared first on Analytics Vidhya.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
FEBRUARY 25, 2024
Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process.
KDnuggets
FEBRUARY 28, 2024
Begin your data professional journey from the basics of statistics to building a production-grade AI application.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
databricks
FEBRUARY 29, 2024
Special thanks to Phillip Jones, Senior Product Manager, and Harshal Brahmbhatt, Systems Engineer from Cloudflare for their contributions to this blog. Organizations across.
Confluent
FEBRUARY 27, 2024
Apache Kafka 3.7 introduces updates to the Consumer rebalance protocol, an official Apache Kafka Docker image, JBOD support in Kraft-based clusters, and more!
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
KDnuggets
MARCH 1, 2024
Want to start your data science journey from home, for free, and work at your own pace? Have a dive into this data science roadmap using the YouTube series.
Seattle Data Guy
MARCH 1, 2024
No matter your industry, you’ll often need to make split-second business decisions in the digital age. Real-time data can help you do just that. It’s information that’s made available as soon as it’s created, meaning you don’t need to wait around for the insights you need. Real-time data processing can satisfy the ever-increasing demand for… Read more The post 5 Real-Time Data Processing and Analytics Technologies – And Where You Can Implement Them appea
Waitingforcode
FEBRUARY 27, 2024
Apache Spark Structured Streaming relies on the micro-batch pattern which evaluates the same query in each execution. That's only a high level vision, though. Under-the-hood, there are many other interesting things that happen.
databricks
FEBRUARY 27, 2024
Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the S.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
FEBRUARY 27, 2024
Want to become a data analyst? This free beginner-friendly data analyst bootcamp is all you need.
Seattle Data Guy
FEBRUARY 26, 2024
SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services. You might seek alternatives to SSIS because you want a more agile… Read more The post Alternatives to SSIS(SQL Server Integration Services) – How To Migrate Away From SSIS appeared first on Seattle Data Guy.
Robinhood
FEBRUARY 29, 2024
As part of the collaboration, Robinhood Wallet announces access to swaps on the Arbitrum network Today at ETHDenver, Robinhood and Arbitrum announced a collaboration that simplifies the path to Layer 2s (L2s) by giving Robinhood Wallet users access to Arbitrum swaps through decentralized exchanges. By opening access to Arbitrum’s advanced scaling solutions, Robinhood Wallet users can now take advantage of low transaction costs and fast transaction speeds on one of the most popular networks in t
Snowflake
FEBRUARY 27, 2024
Snowflake Marketplace is a vibrant resource, with hundreds of providers offering thousands of ready-to-try or ready-to-buy third-party data sets, applications and services. Many of these providers make their products available on Snowflake Marketplace for Snowflake customers to purchase — and they use our integrated Marketplace Monetization capabilities to simplify the process and speed up procurement and sales cycles.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
KDnuggets
FEBRUARY 28, 2024
Exploring the limitless possibilities of AI and making it context-aware.
databricks
FEBRUARY 29, 2024
We are thrilled to announce major improvements to the search capabilities in your Databricks workspace. These enhancements build on DatabricksIQ, the Data Intelligence.
Robinhood
FEBRUARY 27, 2024
Florida State University, Coastal Carolina University, and the University of California, Berkeley will introduce financial education coursework with support from Robinhood Money Drills Robinhood Markets, Inc. is launching Robinhood Money Drills with three new universities, including Florida State University, Coastal Carolina University, and the University of California, Berkeley.
Confessions of a Data Guy
FEBRUARY 25, 2024
I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice. I feel like I have a great burden to bear, a heavy one. I love to write Rust … but I deploy Python. Even when I know I […] The post Why I Love Rust, but Deploy Python appeared first on Confessions of a Data Guy.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
KDnuggets
FEBRUARY 26, 2024
Transform your tech career with one of the best universities in the world!
Snowflake
FEBRUARY 28, 2024
Welcome to Snowflake’s Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. In this edition, find out how Angad Singh, co-founder and CEO of Chabi , is working to give every company the chance to become data-driven with a modern data stack. How would you explain Chabi? Chabi is your all-in-one data stack with state-of-the-art, built-in data warehouse, ETL, data modeling and personalized analytics that are tailored to meet your unique data and BI needs.
databricks
FEBRUARY 28, 2024
This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this.
Engineering at Meta
FEBRUARY 26, 2024
Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig ( @passy ) on the Meta Tech Podcast to discuss the ins and outs of DotSlash , a new open source tool from Meta. DotSlash takes the pain out of distributing binaries and toolchains to developers. Instead of committing large, platform-specific executables to a repository, DotSlash combines a fast Rust program with a JSON manifest prefixed with a #!
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
KDnuggets
FEBRUARY 29, 2024
Want to learn statistics for data science? Check out these free courses to learn essential statistics concepts.
Snowflake
FEBRUARY 29, 2024
Did you know that approximately two thirds of Snowflake customers capture the latitude and longitude of some business entity or event in their account? While latitude and longitude columns can often be used by BI tools and Python libraries to plot points on a map, or shade common administrative boundaries such as states, provinces and countries, companies can do so much more with this valuable geospatial data to perform complex analyses.
ArcGIS
FEBRUARY 29, 2024
You can work with many 3D formats in ArcGIS like i3s and 3D tiles. What is best for your workflow depends on on the 3D capabilities required.
Robinhood
FEBRUARY 28, 2024
Robinhood was founded on the belief that everyone should have access to the financial system. A growing number of people are moving away from the usual 9-5, shifting towards freelancing and side hustles to make a living. But traditional systems haven’t caught up – more than 50% of independent workers don’t feel that they have effective access to retirement and savings plans.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
MARCH 1, 2024
If you are considering transitioning from Microsoft Windows to another operating system that suits your needs, check out these five Linux distributions for data science and machine learning.
Knowledge Hut
FEBRUARY 29, 2024
Being a data scientist means constantly growing, enabling businesses to become more data-propelled, and learning newer trends and tools. There are various excellent resources in data science that can help you to develop your skillset. According to International Data Corporation (IDC), organizations are turning towards digitalization completely. This will help to create more investments, technology development and open various new jobs.
Towards Data Science
FEBRUARY 24, 2024
Best practice and advanced techniques for beginners Continue reading on Towards Data Science »
databricks
FEBRUARY 27, 2024
This blog was written in collaboration with Tim Sedlak, Senior Solutions Architect at Stardog In healthcare and life sciences, accuracy is everything. That's.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
281 
Let's personalize your content