KDnuggets

APRIL 3, 2024

Here’s how to stand out from the competition, impress employers, and get a job in data analytics.

Project 159

Project Data Analytics Data 159

Snowflake

APRIL 1, 2024

Snowflake and Salesforce are happy to share that bidirectional data sharing between Snowflake, the Data Cloud company and Salesforce Data Cloud is now generally available. In September, we proudly announced that organizations could begin leveraging Salesforce data directly in Snowflake via zero-ETL data sharing to unify their customer and business data, accelerate decision-making and help streamline business processes.

Cloud 138

Cloud Retail Entertainment Media 138

Christophe Blefari

APRIL 5, 2024

Lost between ideas ( credits ) Hey, new Data News edition. I hope you will enjoy this week selection after skipping last week one. I was a bit overwhelmed with the amount of tasks I had on the desk—and I'm still. But here we are. Before jumping to the news, I want to let you know that I have improved the Recommendations page and the weekly emails with the recommendation should arrive soon.

SQL 130

SQL Data Python Datasets 130

Waitingforcode

APRIL 5, 2024

Stream processing is great but it brings some gotchas that are not obvious. Logs are one of them.

Process 130

Process IT 130

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

APRIL 1, 2024

Start your AI journey today with these courses from Google.

157

157

Snowflake

APRIL 4, 2024

Data transformation is the process of converting data from one format to another, the “T” in ELT, or extract, load, transform, which enables organizations to get their data analytics-ready and derive insights and value from it. As companies collect more data, from disparate sources and in disparate formats, building and managing transformations has become exponentially more complex and time-consuming.

Cloud 131

Cloud Management Data Pipeline SQL 131

Data Engineering Podcast

MARCH 31, 2024

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. While there are numerous products available to provide that visibility, they all have different technologies and workflows that they focus on.

Project 130

Project Data Lake High Quality Data Data Workflow 130

KDnuggets

APRIL 1, 2024

This article discusses the psychology of data visualization, including the principles and techniques that underpin the creation of persuasive and effective visuals.

Data 153

Data Data Science 153

Monte Carlo

APRIL 1, 2024

Good Night Moon. Where The Wild Things Are. The Cat in the Hat. And now, from the mind of Barr Moses, comes the historic next children’s literary classic: Mastering Data Quality And Your ABCs. A follow up to 2022’s Data Quality Fundamentals: A Practical Guide to Building Reliable Data Pipelines published by O’Reilly Media , Mastering Data Quality And Your ABCs educates the next generation of data and AI engineers about the importance of highly reliable data.

Media 116

Media Data Pipeline Education Data 116

Cloudera

APRIL 1, 2024

We’re excited to share that Gartner has recognized Cloudera as a Visionary among all vendors evaluated in the 2023 Gartner® Magic Quadrant for Cloud Database Management Systems. This recognition underscores Cloudera’s commitment to continuous customer innovation and validates our ability to foresee future data and AI trends, and our strategy in shaping the future of data management.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

databricks

APRIL 3, 2024

Databricks Unity Catalog ("UC") provides a single unified governance solution for all of a company's data and AI assets across clouds and data.

Government 100

Government Cloud Data 100

KDnuggets

APRIL 2, 2024

Dive into the 50 most popular deep-learning questions to get you ready for your interview.

Deep Learning 149

Deep Learning Machine Learning 149

Knowledge Hut

APRIL 5, 2024

Estimating and planning is an important aspect of the Agile methodology. Every plan will help in building a platform to develop a project and estimation will help in filling the gap and remove the hindrances in the software development process. The Agile Methodology roughly provides an idea of how a project manager can plan and estimate to make project success.

Project 98

Project Management Designing Building 98

Cloudera

APRIL 2, 2024

It’s been said that the Federal Government is one of, if not the largest, producer of data in the United States, and this data is at the heart of mission delivery for agencies across the civilian to DoD spectrum. Data is critical to driving the innovation and decision-making that improves services, streamlines operations and strengthens national security.

Designing 101

Designing Process Government Metadata 101

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Towards Data Science

APRIL 5, 2024

A comparative overview Continue reading on Towards Data Science »

Data Science 98

Data Science Data Data Engineering Data Engineer 98

KDnuggets

APRIL 4, 2024

These GitHub repositories provide valuable resources for mastering computer science, including comprehensive roadmaps, free books and courses, tutorials, and hands-on coding exercises to help you gain the skills and knowledge necessary to thrive in the ever-evolving field of technology.

Computer Science 148

Computer Science Coding Technology Programming 148

Confluent

APRIL 2, 2024

Use Confluent data streaming platform to enable real-time pharmaceutical approvals – with healthcare compliance, improved patient safety, and automation for greater efficiency and cost savings.

Pharmaceutical 84

Pharmaceutical Healthcare Data 84

databricks

APRIL 3, 2024

In the era of rapid data growth and increasing pressure on financial institutions to utilize data for AI or genAI models, data governance.

Data Governance 83

Data Governance Government Utilities Data 83

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Towards Data Science

APRIL 2, 2024

A set of strategies and guiding principles to effectively scale your data platform while maximizing its business impact.

Data Science 98

Data Science Data IT Data Architecture 98

KDnuggets

APRIL 5, 2024

The C-suite of business, technology, and data executives sees a new addition – the CAIO (Chief AI Officer). But what does this role mean for the organizations? Let’s find out!

Technology 137

Technology Data 137

Confluent

APRIL 4, 2024

Our latest Confluent Champion post explores how Technical Architect Bergur Ziska helps customers win with data streaming.

Data 64

Data 64

databricks

APRIL 4, 2024

Innovation in the Power and Utilities industry is all but a necessary step to move forward with the evolution of the national power.

Utilities 81

Utilities 81

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. Intuit wrote an exciting article about what it learned from rolling out the internal GenAI tool. My key highlight is that Excellent data documentation and “clean data” improve results.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

KDnuggets

APRIL 2, 2024

Explore some of Python’s sharp corners by coding your way through simple yet helpful examples.

Python 135

Python Coding 135

Confluent

APRIL 3, 2024

Learn why Confluent was named a Leader in the analytic stream processing and event brokering software markets. We believe we innovate every industry with real-time stream processing and analytics, cloud-native Apache Kafka®, and robust developer tooling.

Kafka 64

Kafka Cloud Process 64

ThoughtSpot

APRIL 2, 2024

Over the past couple of years, ThoughtSpot and Google have collaborated on a series of seamless user experiences—enabling deployments on Google Cloud Platform, creating the ability to live query entire Google BigQuery analytics catalogs, and integrating key Looker Modeling functionality just to name a few. This type of co-innovation helps mutual customers get the most value out of their data.

Google Cloud 59

Google Cloud Cloud Accessible Accessibility 59

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Precisely

APRIL 3, 2024

The Precisely Women in Technology (PWIT) network was first established to bring the women of Precisely together to create more opportunities for learning and engagement. Throughout the years, the program has grown, and it now provides mentorship opportunities, a book club, networking events, and more. Each month, a woman from the program is featured to share more about her experience as a woman in tech, her career journey, and the advice she has for other women navigating the same industry.

Technology 59

Technology Programming Project Engineering 59

KDnuggets

APRIL 4, 2024

Do you want to know how to run LLMs on your computer without installing a lot of dependencies or writing code? Well, you're in luck! By the end of this tutorial, you will have successfully run an LLM using llamafile and interacted with it through a user-friendly interface.

Coding 129

Coding IT 129

Scott Logic

APRIL 5, 2024

Introduction This series of blog posts discusses the methods of estimating carbon emissions of end-user devices. Specifically, this looks at web user interfaces, such as websites and web applications, and the devices we use to access them. After intending to write a single blog post, the research journey prompted me to reconsider how to present this to an audience.

Bytes 52

Bytes Systems Designing Data Storage 52

Knowledge Hut

APRIL 5, 2024

In today's competitive software development environment, DevOps enables smooth interaction and cooperation between development and operations teams. The two groups collaborate in DevOps, sharing responsibilities to achieve their primary objective: frequent & faster delivery of rising software that meets customers changing needs. DevOps practices in collaboration with relevant tools and techniques, motivate organizations to complete tasks as effectively as possible.

Coding 52

Coding Java Project Programming 52

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer