KDnuggets

JANUARY 7, 2022

This article will show you what SQL concepts you should know as an experienced professional.

SQL 160

SQL 160

Confluent

JANUARY 5, 2022

Chances are your business is migrating to the cloud. But if you operate business applications in an on-premises datacenter, you know firsthand that the journey to the cloud is fraught […].

Cloud 124

Cloud Building Data 124

Data Engineering Podcast

JANUARY 7, 2022

Summary Data observability is a set of technical and organizational capabilities related to understanding how your data is being processed and used so that you can proactively identify and fix errors in your workflows. In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices.

Data Warehouse 100

Data Warehouse BI Metadata SQL 100

DataKitchen

JANUARY 7, 2022

The post Trend-Setting Products in Data and Information Management for 2022 first appeared on DataKitchen.

Management 98

Management Data 98

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

JANUARY 7, 2022

How to present yourself as a strong candidate in interview presentations.

Data Science 160

Data Science Data 160

Confluent

JANUARY 6, 2022

While Self-Balancing Clusters (SBC) perform effectively in balancing Apache Kafka® clusters, one of the common themes we hear from our users is that they would love some visibility into the […].

Kafka 105

Kafka 105

DataKitchen

JANUARY 3, 2022

The post Data Science and AI Predictions for 2022 first appeared on DataKitchen.

Data Science 98

Data Science Data 98

KDnuggets

JANUARY 6, 2022

Integrate Excel with Word to generate automated reports seamlessly.

Python 160

Python 160

ProjectPro

JANUARY 6, 2022

Can you believe that the human brain takes only 13 milliseconds to process an image? Humans crave stories, and visualizations allow us to create one from data. The majority of data that data scientists and machine learning engineers work with is in a structured or unstructured format that is challenging for humans to analyze and comprehend. Understanding data requires the use of data visualizations, and this is because visuals are processed 60,000 times faster than text inside the human brain.

Python 52

Python Datasets Programming Language Data Science 52

Data Engineering Podcast

JANUARY 1, 2022

Summary This has been an active year for the data ecosystem, with a number of new product categories and substantial growth in existing areas. In an attempt to capture the zeitgeist Maura Church, David Wallace, Benn Stancil, and Gleb Mezhanskiy join the show to reflect on the past year and share their thought son the year to come. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to

Data Warehouse 100

Data Warehouse SQL Hadoop Data Lake 100

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

DataKitchen

JANUARY 3, 2022

Business analysts often find themselves in a no-win situation with constraints imposed from all sides. Their business unit colleagues ask an endless stream of urgent questions that require analytic insights. Business analysts must rapidly deliver value and simultaneously manage fragile and error-prone analytics production pipelines. Data tables from IT and other data sources require a large amount of repetitive, manual work to be used in analytics.

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

KDnuggets

JANUARY 4, 2022

Here are 15 neural network projects you can take on in 2022 to build your skills, your know-how, and your portfolio.

Deep Learning 160

Deep Learning Project Building Portfolio 160

Teradata

JANUARY 6, 2022

Learn more about the pressures and some of the potential responses for banks in the rapidly evolving area of climate risk.

Banking 52

Banking Data 52

Data Engineering Podcast

JANUARY 1, 2022

Summary Communication and shared context are the hardest part of any data system. In recent years the focus has been on data catalogs as the means for documenting data assets, but those introduce a secondary system of record in order to find the necessary information. In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Engineering 100

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Rockset

JANUARY 5, 2022

Rockset is the real-time analytics database in the cloud for modern data teams. Get faster analytics on fresher data, at lower costs, by exploiting indexing over brute-force scanning. It's not your father’s Oracle cluster, but better.* We all know the lightning pace of software innovation. Show me a technology or platform that’s been around for a decade, and I’ll show you an outmoded relic that’s been leapfrogged by faster, more efficient competitors.

Database 52

Database SQL NoSQL Raw Data 52

KDnuggets

JANUARY 4, 2022

To support the creation of new and exciting ML and artificial intelligence (AI) applications, developers need a robust programming language. That's where the Python programming language comes in.

Machine Learning 160

Machine Learning Python Programming Language Project 160

Monte Carlo

JANUARY 5, 2022

When it comes to trusting your data, Monte Carlo, the leading data observability platform and dbt Core are better together. “Why didn’t my job run?” “What happened to this dashboard?” “Why is this column missing?” “What went wrong with my data?!” If you’ve been on the receiving end of a broken data pipeline, these questions probably look familiar to you.

Retail 52

Retail Metadata Data Pipeline Software Engineer 52

Teradata

JANUARY 4, 2022

From supply chain to inflation, our top retail industry consultants weigh in on what the retail & CPG industry will experience in 2022 and beyond.

Retail 52

Retail Consulting 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

phData: Data Engineering

JANUARY 3, 2022

DataOps: What Is It, Core Principles, and Tools For Implementation Nick Goble January 3, 2022 When building a successful company, it’s critical to have a strategy around how you build and scale your business from a technology and data perspective. Your business likely has competitors that are trying to beat you to market, technology is constantly evolving, and so are your customers.

IT 52

IT AWS Software Engineering Software Engineer 52

KDnuggets

JANUARY 3, 2022

Over a year ago, I lost my job due to the COVID-19 pandemic. During this this, I taught myself data science and tripled my income.

Data Science 160

Data Science Data 160

RudderStack

JANUARY 7, 2022

Leveraging RudderStack with Braze, you can effortlessly sync data in and out of the customer engagement platform.

Data 40

Data 40

Hepta Analytics

JANUARY 4, 2022

Today my first LinkedIn Learning course on securing fintech solutions went live! Securing fintech solutions from Security in Fintech Essential Training by Emmanuel Chebukati It was an exciting surprise to wake up to the notifications of the course’s release, and to see the initial reactions it elicited. This demonstrative course covers the essentials that fintech providers and professionals in the industry ought to implement to arrive at a baseline security posture.

Media 40

Media Designing Building Cloud Computing 40

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

phData: Data Engineering

JANUARY 3, 2022

What is Data Engineering? Everything You Need to Know in 2022 Nick Goble January 4, 2022 It’s easy to overlook the amount of data that’s being generated every day — from your smartphone, your Zoom calls, to your Wi-Fi-connected dishwasher. It is estimated that the world will have created and stored 200 Zettabytes of data by the year 2025.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

KDnuggets

JANUARY 3, 2022

This is part 3 of my hands-on course on reinforcement learning, which takes you from zero to HERO. Today we will learn about SARSA, a powerful RL algorithm.

Algorithm 159

Algorithm 159

RudderStack

JANUARY 6, 2022

This post details our engineering team's decision making process for optimizing our Javascript SDK and highlights the results of their work.

Engineering 40

Engineering Process 40

Grouparoo

JANUARY 5, 2022

At Grouparoo, we use a lot of TypeScript. We are always striving to enhance our usage of strong TypeScript types to make better software, and to make it easier to develop Grouparoo. Strong types make it easy for team members to get quick validation about new code, and see hints and tips in their IDEs - a double win! Recently, I found myself repeating a lot of metadata when defining a new API endpoint as I was working to enable noImplicitAny within the @grouparoo/core project.

Utilities 52

Utilities Metadata Coding Building 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

ProjectPro

JANUARY 4, 2022

Whether it is predicting the likelihood of having a heart attack based on weight and workout routine or predicting the probability of email being spam based on the country of origin and word count -logistic regression is widely used because of its remarkable results. It is a machine learning method to solve a classification problem by differentiating one class from another in a given dataset.

Building 52

Building Datasets Machine Learning Algorithm 52

KDnuggets

JANUARY 6, 2022

Semantic segmentation is a computer vision problem that entails putting related elements of an image into the same class. Read on to discover more, including the difficulties associated with annotation.

158

158

RudderStack

JANUARY 5, 2022

In 2021, we wrote about trends we saw emerging in data engineering and made a few predictions. Here, we revisit those predictions and make a few for 2022.

Data Engineering 40

Data Engineering Data Engineer Engineering Data 40

KDnuggets

JANUARY 5, 2022

During transfer learning, the knowledge leveraged and rapid progress from a source task is used to improve the learning and development to a new target task. Read on for a deeper dive on the subject.

Machine Learning 155

Machine Learning 155

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer