KDnuggets

JANUARY 22, 2025

With millions of client-server comms occurring every second across networks, the ability to maintain optimal performance is crucial to avoiding downtime, latency, and inefficiencies that could cost a business thousands or even millions of dollars.

111

111

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. This allows us to verify that our users everyday interactions are protected across our family of apps, such as their religious views in the Facebook Dating app, the example well walk through in this

Data Warehouse 110

Data Warehouse SQL Programming Language Data 110

dbt Developer Hub

JANUARY 22, 2025

Ever since dbt Labs acquired SDF Labs last week , I've been head-down diving into their technology and making sense of it all. The main thing I knew going in was "SDF understands SQL". It's a nice pithy quote, but the specifics are fascinating. For the next era of Analytics Engineering to be as transformative as the last, dbt needs to move beyond being a string preprocessor and into fully comprehending SQL.

SQL 78

SQL Database Coding Technology 78

Simon Späti

JANUARY 22, 2025

To be proficient as a data engineer, you need to know various toolkitsfrom fundamental Linux commands to different virtual environments and optimizing efficiency as a data engineer. This article focuses on the building blocks of data engineering work, such as operating systems, development environments, and essential tools. We’ll start from the ground upexploring crucial Linux commands, containerization with Docker, and the development environments that make modern data engineering possibl

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

KDnuggets

JANUARY 22, 2025

Let's have a look at the different functions included within the statistics module, and point to more in-depth tutorials on each of them individually.

112

112

databricks

JANUARY 22, 2025

Databricks has been recognized as one of the winners of the annual Glassdoor Employees Choice Awards, a list of the Best Places to.

75

75

Towards Data Science

JANUARY 22, 2025

How to set an automated check to improve your dbt documentation Image by the author (generated withchatgpt) In large dbt projects, maintaining consistent and up-to-date documentation can be a challenge. Although dbts {{ doc() }} function allows you to store and reuse descriptions for the columns of your models, ensuring its usage remains quite a manual process and prone to mistakes, and therefore can easily lead to incomplete or outdated documentation.

Python 40

Python Project Coding Data Science 40

Snowflake

JANUARY 22, 2025

Silos complicate effective governance and discovery With the advent of generative AI and large language models (LLMs), enterprises are racing to unlock as much business value as possible from their data assets, including apps and models. Unfortunately, these data assets are often locked away in silos across multiple cloud service providers and solutions, as well as across different partner, customer and vendor ecosystems.

Government 91

Government Data Governance Unstructured Data Cloud 91

Towards Data Science

JANUARY 22, 2025

How to set an automated check to improve your dbt documentation Image by the author (AI generated) In large dbt projects, maintaining consistent and up-to-date documentation can be a challenge. Although dbts {{ doc() }} function allows you to store and reuse descriptions for the columns of your models, ensuring its usage remains quite a manual process and prone to mistakes, and therefore can easily lead to incomplete or outdated documentation.

Python 83

Python Project Coding Data Science 83

Cloudera

JANUARY 22, 2025

Security Operations Centers (SOCs) are the backbone of organizational cybersecurity, responsible for detecting, investigating, and responding to threats in real-time. Yet, the increasing complexity and volume of cyber threats present significant challenges. SOC teams often grapple with alert fatigue, skill shortages, and time-consuming processes. Generative AI (GenAI), coupled with Agentic AI, offers a revolutionary approach to addressing these pain points.

Data Security 70

Data Security Algorithm Machine Learning Coding 70

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Tweag

JANUARY 22, 2025

In software development, especially with microservices, ensuring seamless integration between components is crucial for delivering high-quality applications. One approach I really like, to tame this complexity, is contract testing. Contract testing is a powerful technique that focuses on verifying interactions between software components early and in a controlled environment.

Python 57

Python Database BI Metadata 57

Monte Carlo

JANUARY 22, 2025

As a core technology provider to over 130 global marketplaces, 2,200 financial institutions, and over 6,000 corporations, and over 30 exchanges in North America and the Nordics, Nasdaq is one of the largest market operators in the world today. That means that when data goes bad, the impact may well reach around the globe as well. With millions of stakeholders in a highly regulated industry and data products feeding a web of international financial services, trustworthy data isnt just a nice-to-h

Amazon Web Services 52

Amazon Web Services Data Lake AWS Data 52

WeCloudData

JANUARY 22, 2025

Prompt engineering has become a key discipline for optimizing AI systems. Among the various prompt engineering techniques, role-playing prompting stands out for its creativity and effectiveness. In this last blog of WeCloudDatas Prompt Engineering Series, well dive deep into role-playing prompting, explore its applications, and share actionable tips to master this innovative approach.

Engineering 52

Engineering Systems IT 52

Scott Logic

JANUARY 22, 2025

Last week the UK government launched their 50-point AI Opportunities Action Plan. Im going to ignore the marketing hyperbole (theyre going to mainline AI into the veins of this enterprising nation - seriously?!) and concentrate more on the substance and there is a lot of it. The plan is ambitious, but it is something of a mixed bag. Some sizeable and worthwhile investments, alongside others which are quite questionable.

Data Mining 52

Data Mining Government Medical Datasets 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing