KDnuggets

JUNE 13, 2022

Gain a beginner's perspective on artificial neural networks and deep learning with this set of 14 straight-to-the-point related key concept definitions.

Deep Learning 160

Deep Learning Machine Learning 160

Simon Späti

JUNE 14, 2022

Data consumers, such as data analysts, and business users, care mostly about the production of data assets. On the other hand, data engineers have historically focused on modeling the dependencies between tasks (instead of data assets) with an orchestrator tool. How can we reconcile both worlds? This article reviews open-source data orchestration tools (Airflow, Prefect, Dagster) and discusses how data orchestration tools introduce data assets as first-class objects.

Data Pipeline 130

Data Pipeline Data Data Engineering Data Engineer 130

Azure Data Engineering

JUNE 12, 2022

Self-hosted integration runtime in the context of Azure data factory is a gateway that connects the on-prem data sources to datastores in the cloud. To know more about Integration runtimes, please refer to the previous post. We have discussed how to check whether Integration Runtime is online or offline using PowerShell command in a previous post. In today’s post, lets have a look at how to monitor self-hosted integration runtime metrics such as CPU utilization, Available memory, number of concu

Utilities 130

Utilities Cloud Data Process 130

Data Engineering Podcast

JUNE 12, 2022

Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information. Aparavi was created to tame the sprawl of information across machines, datacenters, and clouds so that you can reduce the amount of duplicate data and save time an

Unstructured Data 130

Unstructured Data MongoDB MySQL Scala 130

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

KDnuggets

JUNE 13, 2022

Learn essential Git commands for versioning and collaborating on data science projects.

Data Science 159

Data Science Data Project Programming 159

Teradata

JUNE 14, 2022

With the Teradata QueryGrid Google BigQuery Connector, we’re enabling our customers to natively join data between Vantage and BigQuery in real-time, at scale.

Data 98

Data 98

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth. In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

KDnuggets

JUNE 17, 2022

In this tutorial, we are going to list some of the most common algorithms that are used in supervised learning along with a practical tutorial on such algorithms.

Algorithm 159

Algorithm Machine Learning 159

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection. A key requirement for these use cases is the ability to not only actively pull data from source systems but to receive data that is being pushed from various sources to the central distribution service. .

Data Pipeline 94

Data Pipeline Building Kafka Java 94

AltexSoft

JUNE 16, 2022

Choosing the machine learning path when developing your software is half the success. Yes, it’s an advanced way of doing things. Yes, it brings automation, so widely discussed machine intelligence, and other awesome perks. But just because you put it there doesn’t guarantee your project will do well and pay off. So, how would you measure the success of a machine learning model?

Machine Learning 91

Machine Learning Hospitality Retail Medical 91

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Confluent

JUNE 16, 2022

How real-time integrations between modern and legacy systems benefit communication service providers with autonomous network features, enhanced customer experiences, and more.

Media 86

Media Engineering Systems Architecture 86

KDnuggets

JUNE 15, 2022

An introduction to the generative adversarial network model DoppelGANger, and how you can use a new open-source PyTorch implementation of it to create high-quality synthetic time-series data.

Data 155

Data IT Data Science 155

Cloudera

JUNE 16, 2022

Every large enterprise organization is attempting to accelerate their digital transformation strategies to engage with their customers in a more personalized, relevant, and dynamic way. The ability to perform analytics on data as it is created and collected (a.k.a. real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for organizations. .

Kafka 88

Kafka Manufacturing Data Lake SQL 88

Netflix Tech

JUNE 15, 2022

By Alex Hutter , Falguni Jhaveri , and Senthil Sayeebaba In a previous post , we described the indexing architecture of Studio Search and how we scaled the architecture by building a config-driven self-service platform that allowed teams in Content Engineering to spin up search indices easily. This post will discuss how Studio Search supports querying the data available in these indices.

Engineering 76

Engineering Accessible Accessibility Architecture 76

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Confluent

JUNE 14, 2022

Growth marketing and sales isn’t only for growing revenue, but your sales people. Here’s how our Director of Sales ensured success for our SDRs through mentorship and upward mobility.

72

72

KDnuggets

JUNE 14, 2022

The guide to writing production-ready Python code for machine learning projects.

Machine Learning 153

Machine Learning Python Coding Project 153

Cloudera

JUNE 15, 2022

Cloudera’s June Volunteer Spotlight is Michael Billau, customer operations engineer from Raleigh, North Carolina! Michael volunteers with the Food Bank of Central and Eastern North Carolina. The Food Bank of Central and Eastern North Carolina provides food daily to the over 200,000 people facing food insecurity and hunger in the Raleigh area, while simultaneously building solutions to end hunger permanently in communities across North Carolina. .

Food 82

Food Banking Engineering Systems 82

ProjectPro

JUNE 16, 2022

An AWS data pipeline helps businesses move and unify their data to support several data-driven initiatives. Generally, it consists of three key elements: a source, processing step(s), and destination to streamline movement across digital platforms. It enables flow from a data lake to an analytics database or an application to a data warehouse. Amazon Web Services (AWS) offers an AWS Data Pipeline solution that helps businesses automate the transformation and movement of data.

Data Pipeline 59

Data Pipeline AWS Amazon Web Services Data Consolidation 59

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineer

Confluent

JUNE 17, 2022

IDC shares takeaways from Kafka Summit London, how data streaming maximizes real-time data connections, revenue growth, and the ability to win in a digital-first world.

Kafka 59

Kafka Data 59

KDnuggets

JUNE 16, 2022

In this article, we outline 15 books on topics ranging from the technical to the non-technical, to help you improve your understanding of end-to-end best practices related to data.

Data 149

Data Data Science 149

Cloudera

JUNE 13, 2022

We are excited to announce that Cloudera is named as a 2022 Gartner Peer Insights Customers’ Choice for Cloud Database Management Systems (DBMS). Peer Insights is a user review site, the technology professional’s “go-to” destination for information on customer experience. Gartner Peer Insights collects anonymous customer reviews on select product categories.

Hadoop 73

Hadoop Manufacturing Finance Media 73

Monte Carlo

JUNE 16, 2022

We surveyed over 200 data leaders on the show floor at Snowflake Summit 2022 about their experiences driving data adoption in the cloud. Hint: while progress has been made to migrate data to Snowflake, gaps remain when it comes to quality, documentation, and democratization. Over the past several years, Snowflake has taken the data engineering world by storm, with a rapidly growing list of new product features, a dizzying ecosystem of technology partners, and, most importantly, an impressive col

Cloud 52

Cloud Recruitment Retail Healthcare 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

know.bi

JUNE 15, 2022

Why would you upgrade your Pentaho projects to Apache Hop? Before going into the details of how you should upgrade to Apache Hop , let's have a look at a couple of reasons why upgrading to Apache Hop is a good idea. We'll look at why it helps you to work with a platform that is actively innovating, is truly open source, and has an active community. Work with an innovative platform Since Apache Hop started as an Incubating project at the Apache Software Foundation back in 2020 and graduated in la

Metadata 52

Metadata Data Integration Cloud Storage Project 52

KDnuggets

JUNE 16, 2022

Although dashboards have become quite an integral part of performance tracking in organizations, implementing them can be tricky even for the most experienced analysts. This guide walks you through the steps that will allow you to create easily updatable, automated and scalable Power BI / Tableau dashboards.

BI 119

BI Data Data Science 119

Rockset

JUNE 14, 2022

In this 30 minute video overview, CTO and Rockset Co-founder Dhruba Borthakur discusses Rockset's ALT architecture , how data is ingested, stored and queried in Rockset, and why Rockset is simple to use, incredibly fast, and capable of the highly efficient execution of complex distributed queries across diverse data sets. Embedded content: [link] We'll be doing more videos like this in the future, so sign up for notices from our blog and join our community so you don't miss them.

Architecture 52

Architecture Lambda Architecture Hadoop Database 52

Monte Carlo

JUNE 14, 2022

Conferences typically follow a bell curve. A few people trickle in on day one, a bit more at the welcome event. Then you peak at the keynote. After Day One, these events slowly lose steam until only the most fanatical conference warriors are roaming exhibitor booths late Thursday morning. Snowflake Summit 2022 has been different – and I mean this in the best way possible.

Cloud 52

Cloud Python Data Ingestion Government 52

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineering

Eventbrite Engineering

JUNE 13, 2022

As Eventbrite engineering leans into team-owned infrastructure, or DevOps, we’re obviously learning a lot of new technologies in order to stand up our infrastructure, but owning the infrastructure also means it’s up to us to make sure that infrastructure is stable as we continue to release software. Obviously, the answer is that we need to … Continue reading "Monitoring Your System" The post Monitoring Your System appeared first on Engineering Blog.

Systems 52

Systems Engineering Technology IT 52

KDnuggets

JUNE 15, 2022

Here are some data science related podcasts to help you either grow your interest in the field, increase your current knowledge, or help you develop yourself.

Data Science 119

Data Science Data 119

Propel Data

JUNE 14, 2022

Today, we are thrilled to announce Propel Data – an API Platform for developers to easily build in-product analytics with large-scale data.

Data Analytics 40

Data Analytics Data Building 40

Monte Carlo

JUNE 17, 2022

As more companies invest in more data tools, initiatives, and teams, the appetite to become a “data-driven organization” continues to grow. But if stakeholders, consumers, and leaders across the company don’t trust that the data flowing through your pipelines and populating your products is useful and reliable, all that investment is for naught. So how can a team build a culture of data trust—especially within a complex environment?

Building 52

Building Portfolio Data Project 52

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud