KDnuggets

NOVEMBER 26, 2021

Learn to use custom Named Entity Recognition and Relation Extraction models.

Building 160

Building 160

Azure Data Engineering

NOVEMBER 21, 2021

During some scenarios in Azure Data Factory, we may want to intentionally stop the execution of the pipeline. An example could be when we want to check the existence of a file or folder using Get Metadata activity. We may want to fail the pipeline if the file/folder does not exist. To achieve this, we could use the Fail Activity. Invoking the Fail Activity ensures that the pipeline execution will be stopped.

Metadata 130

Metadata Data Utilities Coding 130

Teradata

NOVEMBER 22, 2021

Teradata's Martin Willcox recently passed 17 years at Teradata and a quarter of a century in the industry. Here are the ten things he's learned about data analytics in those 20-odd years.

Data Analytics 111

Data Analytics Data 111

Data Engineering Podcast

NOVEMBER 20, 2021

Summary The technology for scaling storage and processing of data has gone through massive evolution over the past decade, leaving us with the ability to work with massive datasets at the cost of massive complexity. Nick Schrock created the Dagster framework to help tame that complexity and scale the organizational capacity for working with data. In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a f

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

NOVEMBER 23, 2021

Sure, we all make mistakes -- which can be a bit more painful when we are trying to get hired -- so check out these typical errors applicants make while answering SQL questions during data science interviews.

Data Science 160

Data Science SQL Data 160

Cloudera

NOVEMBER 25, 2021

Advances in the performance and capability of Artificial Intelligence (AI) algorithms has led to a significant increase in adoption in recent years. In a February 2021 report by IDC, they estimate that world-wide revenues from AI will grow by 16.4% in 2021 to USD $327 billion. Furthermore, AI adoption is becoming increasingly widespread and not just concentrated within a small number of organisations.

Government 98

Government Insurance Algorithm Machine Learning 98

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. With the improvements in streaming engines it is now possible to perform all of your data integration in near real time, but it can be challenging to understand the proper processing patterns to make that performant. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

KDnuggets

NOVEMBER 24, 2021

Maintaining a centralized data repository can simplify your business intelligence initiatives. Here are four data integration tools that can make data more valuable for modern enterprises.

Data Integration 160

Data Integration Business Intelligence Data Data Preparation 160

Cloudera

NOVEMBER 22, 2021

Introduction. In legacy analytical systems such as enterprise data warehouses, the scalability challenges of a system were primarily associated with computational scalability, i.e., the ability of a data platform to handle larger volumes of data in an agile and cost-efficient way. Open source frameworks such as Apache Impala, Apache Hive and Apache Spark offer a highly scalable programming model that is capable of processing massive volumes of structured and unstructured data by means of paralle

Hadoop 94

Hadoop Government Data Security Cloud 94

AltexSoft

NOVEMBER 23, 2021

Nowadays, all organizations need real-time data to make instant business decisions and bring value to their customers faster. But this data is all over the place: It lives in the cloud, on social media platforms, in operational systems, and on websites, to name a few. Not to mention that additional sources are constantly being added through new initiatives like big data analytics , cloud-first, and legacy app modernization.

Process 69

Process Data Lake Metadata Data Warehouse 69

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Rockset

NOVEMBER 23, 2021

We built Rockset with the mission to make real-time analytics easy and affordable in the cloud. We put our users first and obsess about helping our users achieve speed, scale and simplicity in their modern real-time data stack (some of which I discuss in depth below). But we, as a team, still take performance benchmarks seriously. Because they help us communicate that performance is one of the core product values at Rockset.

MongoDB 59

MongoDB Data Ingestion SQL PostgreSQL 59

KDnuggets

NOVEMBER 23, 2021

Notes from Fluent Python by Luciano Ramalho.

Python 160

Python Programming 160

Cloudera

NOVEMBER 23, 2021

Concepts. What is Cloudera Operational Database (COD)? Operational Database is a relational and non-relational database built on Apache HBase and is designed to support OLTP applications, which use big data. The operational database in Cloudera Data Platform has the following components: . Apache Phoenix provides a relational model facilitating massive scalability.

Database 91

Database Non-relational Database NoSQL Government 91

Teradata

NOVEMBER 22, 2021

Teradata's Martin Willcox recently passed 17 years at Teradata and a quarter of a century in the industry. Here are the ten things he's learned about data analytics in those 20-odd years.

Data Analytics 59

Data Analytics Data 59

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

ProjectPro

NOVEMBER 25, 2021

Although businesses have an inclination towards structured data for insight generation and decision-making, text data is one of the vital information generated from digital platforms. However, it is not straightforward to extract or derive insights from a colossal amount of text data. To mitigate this challenge, organizations are now leveraging natural language processing and machine learning techniques to extract meaningful insights from unstructured text data.

Machine Learning 52

Machine Learning Algorithm Datasets Google Cloud 52

KDnuggets

NOVEMBER 23, 2021

Also: How I Redesigned over 100 ETL into ELT Data Pipelines; Where NLP is heading; Don’t Waste Time Building Your Data Science Network; Data Scientists: How to Sell Your Project and Yourself.

Data Science 160

Data Science Project Data Pipeline Data 160

Cloudera

NOVEMBER 25, 2021

As data continues to grow at an exponential rate, our customers are increasingly looking to advance and scale operations through digital transformation and the cloud. These modern digital businesses are also dealing with unprecedented rates of data volume, which is exploding from terabytes to petabytes and even exabytes which could prove difficult to manage.

Amazon Web Services 90

Amazon Web Services Cloud AWS Machine Learning 90

Pipeline Data Engineering

NOVEMBER 24, 2021

Most data professionals realise very early in their journey that accessing the knowledge that they really need to solve data engineering problems is hard to come by. The other thing they don’t necessarily see is how short-sighted a lot of courses are, and how most of the technical content they provide is going to be rendered useless in a year or two.

Data Engineer 52

Data Engineer Data Engineering Engineering Education 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

ProjectPro

NOVEMBER 24, 2021

“Is data science hard to learn?”, “Is data science a hard job?”, “Is it hard to get a data science job?” Are you a data science enthusiast who believes data science is hard and keeps thinking about such questions? Allow us to challenge your thoughts and read this blog as we will help you answer all those questions.

Data Science 52

Data Science Consulting Machine Learning Software Engineering 52

KDnuggets

NOVEMBER 22, 2021

PyTorch and TensorFlow are the two leading AI/ML Frameworks. In this article, we take a look at their on-device counterparts PyTorch Mobile and TensorFlow Lite and examine them more deeply from the perspective of someone who wishes to develop and deploy models for use on mobile platforms.

Deep Learning 160

Deep Learning 160

Cloudera

NOVEMBER 24, 2021

For underserved youth, the lack of educational opportunity can seriously hinder their development and future career prospects. Many are deprived of early childhood chances at experiencing the professional world, so a career in science, finance, IT, or marketing is a pipe dream. . Unless someone shows them it’s possible. At the Middle Tennessee and Peninsula chapters of the Boys & Girls Clubs, high school students are receiving an introduction into a new world of possibilities.

Finance 76

Finance Education Programming Designing 76

Preset

NOVEMBER 23, 2021

In this tutorial, we'll learn how to build a metrics dashboard with Apache Superset, a modern and open-source data exploration and visualization platform. We'll also use Cube, an open-source metrics store, as the data source for Superset.

Building 52

Building Data 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

ProjectPro

NOVEMBER 23, 2021

Towards the end of the 2000s, complex neural networks and model-based deep learning saw a huge upsurge in demand with revolutionary results in the fields of computer vision and natural language processing. While reinforcement learning has been around the corner from the same time, it was overshadowed by its counterparts for decades. It first became the talk of the town when in 2016, Google Deepmind’s AlphaGo defeated the World Champion in the Chinese game of Go.

Project 52

Project Python Algorithm AWS 52

KDnuggets

NOVEMBER 22, 2021

Read some of the key things the author has learned during the infamous job seeking stage.

Data 159

Data Data Science 159

RudderStack

NOVEMBER 23, 2021

In this update, we cover our new high-performance JavaScript SDK, announce a new destination integration, and highlight our Event Stream pricing promotion.

40

40

Preset

NOVEMBER 22, 2021

This tutorial post will cover all of the steps needed to make your first code contribution to the Apache Superset project.

Coding 52

Coding Project 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

ProjectPro

NOVEMBER 22, 2021

PySpark has exploded in popularity in recent years, and many businesses are capitalizing on its advantages by producing plenty of employment opportunities for PySpark professionals. According to the Businesswire report , the worldwide big data as a service market is estimated to grow at a CAGR of 36.9% from 2019 to 2026, reaching $61.42 billion by 2026.

Hadoop 52

Hadoop Python Datasets Metadata 52

KDnuggets

NOVEMBER 25, 2021

You can call Mito into your Jupyter Environment and each edit you make will generate the equivalent Python in the code cell below.

Python 159

Python Coding Programming 159

Rock the JVM

NOVEMBER 20, 2021

Discover how Akka Streams implements backpressure, a key component of the Reactive Streams specification, in this detailed demonstration

40

40

KDnuggets

NOVEMBER 25, 2021

A classic KDnuggets Thanksgiving cartoon examines the predicament of one group of fowl Data Scientists.

Data Science 158

Data Science Data 158

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer