Analytics Vidhya
JULY 25, 2023
In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! In this captivating journey, we’ll explore the distinctive paths these tech titans take […] The post Data Engineer vs Data Scientist: Which Career to Choose?
Confessions of a Data Guy
JULY 22, 2023
Nothing gives me greater joy than rocking the boat. I take pleasure in finding what people love most in tech and trying to poke holes in it. Everything is sacred. Nothing is sacred. I also enjoy doing simple things, things that have a “real-life” feel to them. I suppose I could be like the others […] The post Polars vs Pandas. Inside an AWS Lambda. appeared first on Confessions of a Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Christophe Blefari
JULY 28, 2023
🧜♂️ ( credits ) Hey, this is a mid-2023 edition with some of my favourite articles and the popular articles that have been shared this year in the newsletter. There isn't any fancy calculation on how to find the popular articles. Here how it's done. Every link sent in each newsletter is tracked in 2 ways: when you click on a link it first redirect you to my blog so I know that you've clicked on it it adds ref=blef.fr to the url, so the original articl
Waitingforcode
JULY 25, 2023
You certainly know it, the watermark (aka GC Watermark) is responsible for cleaning state store in Apache Spark Structured Streaming. But you may not know that it's not the single time-based condition. There is a different one involved in the stream-to-stream joins.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Data Engineering Podcast
JULY 23, 2023
Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together.
KDnuggets
JULY 28, 2023
The highly anticipated Python edition of Introduction to Statistical Learning is here. And you can read it for free! Here’s everything you need to know about the book.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
databricks
JULY 28, 2023
For the final part of our Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS series, we'll cover an important.
ArcGIS
JULY 26, 2023
Sure, we can make features look like they are glowing. But how can we make them look like they are casting light on the basemap below?
KDnuggets
JULY 27, 2023
Are you interested in Data Science? This blog will help you kickstart or advance your data science career. You'll learn about the most popular programming languages data scientists use to clean, analyze, visualize, and model data.
Pinterest Engineering
JULY 25, 2023
Soam Acharya | Data Engineering Oversight; Keith Regier | Data Privacy Engineering Manager Background Businesses collect many different types of data. Each dataset needs to be securely stored with minimal access granted to ensure they are used appropriately and can easily be located and disposed of when necessary. As businesses grow, so does the variety of these datasets and the complexity of their handling requirements.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
databricks
JULY 26, 2023
All healthcare is personal. Individuals have different underlying genetic predispositions, environmental exposures, and past medical histories, not to mention different propensities to engage.
Knowledge Hut
JULY 28, 2023
Machine learning for anomaly detection is crucial in identifying unusual patterns or outliers within data. It plays a vital role in cybersecurity, finance, healthcare, and industrial monitoring. By learning from historical data, machine learning algorithms autonomously detect deviations, enabling timely risk mitigation. They excel at identifying subtle anomalies and adapt to changing patterns.
KDnuggets
JULY 25, 2023
This is an overview of the "Textbooks Are All You Need" paper, highlighting the Phi-1 model's success using high-quality synthetic textbook data for AI training.
Towards Data Science
JULY 24, 2023
Build scalable and fast data pipelines with Polars Continue reading on Towards Data Science »
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
databricks
JULY 24, 2023
Large Language Models (LLMs) unlock a wide spectrum of potential use cases to deliver business value, from analyzing the sentiment of text data.
ThoughtSpot
JULY 26, 2023
Today we're excited to officially launch AI Explain on ThoughtSpot for Sheets , the ultimate cheat code for data literacy and exploration. AI Explain integrates Google's PaLM 2 LLM, specifically leveraging the Bison model to automatically generate the top data stories for any visualization created with our Sheets extension. If you're not familiar with ThoughtSpot for Sheets, it's ThoughtSpot’s free app plugin for Google Sheets that lets you explore your Sheets data through in
KDnuggets
JULY 25, 2023
RAPIDS cuDF, with its pandas-like API, enables data scientists and engineers to quickly tap into the immense potential of parallel computing on GPUs–with just a few code line changes. Read on for more.
Confluent
JULY 25, 2023
Confluent obtained the ISO 27701 certification which demonstrates the high standard of Confluent’s privacy program and practices.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
ArcGIS
JULY 27, 2023
Packed circles are a unique visualization technique for representing individual data points within an aggregate symbol.
databricks
JULY 23, 2023
We're thrilled to announce the general availability of Fivetran access in Partner Connect for all users. This innovation makes it 10x easier for.
KDnuggets
JULY 24, 2023
With Generative AI being a hot topic, learn more about these courses provided that can give you a kick start into the wave.
Snowflake
JULY 26, 2023
The nursing profession is in crisis. According to McKinsey, over 30% of surveyed nurses said they may leave their current patient care jobs in the next year, and for inpatient nurses it’s higher at 45%. Meanwhile, the average professional tenure of nurses dropped from 3.6 years to 2.8 years between 2020 and 2023. These alarming trends have healthcare systems on red alert.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Knowledge Hut
JULY 26, 2023
Hybrid project management refers to combining two or more methodologies, thereby allowing a project manager to enjoy the benefits of multiple methodologies. This project management methodology allows you the flexibility to use elements from different methodologies. Organizations that harness hybrid project management methods are more likely to reap the benefits like speed, adaptability, flexibility, etc.
databricks
JULY 27, 2023
Check our Solution Accelerator for Propensity Scoring for more details and to download the notebooks. Consumers increasingly expect to be engaged in a.
KDnuggets
JULY 24, 2023
When working on a data science problem, one of the most important choices to make is selecting the appropriate machine learning algorithm.
Tweag
JULY 26, 2023
The vast majority of the Rust projects are using Cargo as a build tool. Cargo is great when you are developing and packaging a single Rust library or application, but when it comes to a fast-growing and complex workspace, one could be attracted to the idea of using a more flexible and scalable build system. Here is a nice article elaborating on why Cargo should not be considered as a such a build system.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Knowledge Hut
JULY 26, 2023
Amid online activities, digital transactions, social meetups and virtual communication, we are subconsciously dependent on the virtual world where we spend much of our time. There is online bioethics, and regulations that provide safe browsing and working, feature we have incognito browsing, and Tor is one for safe browsing. Tor in cyber security is a security bioethics.
databricks
JULY 23, 2023
Starting today, all users will experience a new and improved navigation experience when using the Databricks UI. The changes will impact three surfaces.
KDnuggets
JULY 26, 2023
Learn from my mistakes and avoid making the same mistakes.
Rock the JVM
JULY 26, 2023
by Herbert Kateu Hey, it’s Daniel here. You’re reading a giant article about a real-life use of the Http4s library. If you want to master the Typelevel Scala libraries (including Http4s) with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. It’s my biggest and most jam-packed course yet. 1. Introduction This article is a continuation of the authentication methods that were covered in part1.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
202 
Let's personalize your content