KDnuggets

MAY 20, 2022

Let's look at six must-have tools every data scientist should use.

Machine Learning 160

Machine Learning Python Data 160

Start Data Engineering

MAY 18, 2022

1. Introduction 2. Set up 3. Reproducibility 3.1. Docker 3.2. Docker Compose 4. Developer ergonomics 4.1. Formatting and testing 4.2. Makefile 5. Conclusion 6. Further reading 7. References 1. Introduction Data systems usually involve multiple systems, which makes local development challenging.

Project 147

Project Python Data Systems 147

Confluent

MAY 17, 2022

I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. The 3.2.0 release contains many new features and improvements. This blog will highlight […].

Kafka 139

Kafka 139

Azure Data Engineering

MAY 15, 2022

When it comes to transforming structured data, (e.g., applying business logic, standardization etc.) stored in a database, SQL is the most convenient and fit-to-purpose option. Stored procedures provide a way to store the transformation logic as a set of SQL statements that can be re-executed as pre-compiled code. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.

SQL 130

SQL Database Structured Data Data 130

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

KDnuggets

MAY 19, 2022

Read the best books on Programming, Statistics, Data Engineering, Web Scraping, Data Analytics, Business Intelligence, Data Applications, Data Management, Big Data, and Cloud Architecture.

Data Science 160

Data Science Business Intelligence Big Data Architecture 160

Cloudera

MAY 18, 2022

Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain a better understanding of where it is and where it needs to go.

Manufacturing 100

Manufacturing High Quality Data Retail Algorithm 100

Teradata

MAY 19, 2022

Faced with persistent supply chain disruption automotive companies need a new approach to planning. Find out more.

Management 94

Management 94

KDnuggets

MAY 16, 2022

This guide will help aspiring data scientists and machine learning engineers gain better knowledge and experience. I will list different types of machine learning algorithms, which can be used with both Python and R.

Machine Learning 160

Machine Learning Algorithm Python Engineering 160

Cloudera

MAY 19, 2022

From fashion to data flow, in this #ClouderaLife Spotlight Margot talks about her career transition from fashion design to cloud computing and her co-founding of Cloudera’s Asian American and Pacific Islander community Employee Resource Group amid the racial tensions of 2021. . It started with feeling stuck and ended with a brand-new career (BTW, lots of hard work in the middle).

Software Engineer 96

Software Engineer Software Engineering Engineering Cloud Computing 96

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience.

Data Lake 100

Data Lake Building Architecture BI 100

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Confluent

MAY 18, 2022

There’s an increasing need for businesses to act intelligently and in real time to win in today’s digital-first world. To achieve this, forward-thinking companies are modernizing their data infrastructure with […].

Data 75

Data 75

KDnuggets

MAY 16, 2022

This post provides a concise overview of 18 natural language processing terms, intended as an entry point for the beginner looking for some orientation on the topic.

Process 157

Process 157

dbt Developer Hub

MAY 18, 2022

If you're reading this article, it looks like you're wondering how you can better optimize your Redshift queries - and you're probably wondering how you can do that in conjunction with dbt. In order to properly optimize, we need to understand why we might be seeing issues with our performance and how we can fix these with dbt sort and dist configurations.

Raw Data 59

Raw Data Consulting Datasets IT 59

Emeritus

MAY 19, 2022

With businesses increasingly relying on data for their day-to-day operations, the role of a data engineer has emerged as one of the most sought-after professions in the industry. But what does a data engineer do exactly? And why is it in demand? According to McKinsey, by 2025, smart workflows and seamless interactions between humans and… The post What Does a Data Engineer do?

Data Engineer 52

Data Engineer Data Engineering Engineering Data 52

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Confluent

MAY 18, 2022

With more data being produced in real time by many systems and devices than ever before, it is critical to be able to process it in real time and get […].

Process 59

Process Systems IT Data 59

KDnuggets

MAY 19, 2022

What should you be looking for in an IDE? Find out here.

Programming 157

Programming 157

Big Data Tools

MAY 19, 2022

Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.

Data Engineer 52

Data Engineer Data Engineering Engineering Big Data Tools 52

Emeritus

MAY 19, 2022

Data is so ubiquitous and valuable that it is touted as the new currency. From data analytics to data engineering, everything is data-centric. As Carly Fiorina, the former Chief Executive Officer of Hewlett Packard, said, “The goal is to turn data into information, and information into insight.” Data allows leaders to make informed decisions that… The post What Does a Data Engineer Do and How Can You Become One?

Data Engineer 52

Data Engineer Data Engineering Engineering Data Analytics 52

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Rockset

MAY 17, 2022

This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Posts published so far in the series: Why Mutability Is Essential for Real-Time Data Analytics Handling Out-of-Order Data in Real-Time Analytics Applications Handling Bursty Traffic in Real-Time Analytics Applications SQL and Complex Queries

SQL 52

SQL NoSQL Hadoop MongoDB 52

KDnuggets

MAY 18, 2022

Complete guide and blog post series on IT Operations Management with AIOps. Using AI and Machine Learning to manage IT complexity to deliver world class IT service while keeping the lights on.

IT 131

IT Management Machine Learning 131

Big Data Tools

MAY 19, 2022

Long time no see! Sorry about the silence, but luckily we’re back. Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see.

Data Engineer 52

Data Engineer Data Engineering Engineering Big Data Tools 52

Rock the JVM

MAY 16, 2022

Scala's general type projections are considered unsound and were removed in Scala 3: discover what this means and how it affects your code

Scala 52

Scala Project Coding IT 52

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Monte Carlo

MAY 16, 2022

Data lineage isn’t new, but automation has finally made it accessible and scalable—to a certain extent. In the old days (way back in the mid-2010s), lineage happened through a lot of manual work. This involved identifying data assets, tracking them to their ingestion sources, documenting those sources, mapping the path of data as it moved through various pipelines and stages of transformation, and pinpointing where the data was served up in dashboards and reports.

Metadata 52

Metadata Data Data Pipeline Big Data 52

KDnuggets

MAY 18, 2022

Here’s how you can use your data skills to generate side income from home.

Data Science 131

Data Science Data 131

dbt Developer Hub

MAY 16, 2022

Analytics engineers (AEs) are constantly navigating through the names of the models in their project, so naming is important for maintainability in your project in the way you access it and work within it. By default, dbt will use your model file name as the view or table name in the database. But this means the name has a life outside of dbt and supports the many end users who will potentially never know about dbt and where this data came from, but still access the database objects in the datab

BI 52

BI Data Warehouse Database SQL 52

KDnuggets

MAY 20, 2022

Most companies haven’t seen ROI from machine learning since the benefit is only realized when the models are in production. Here’s how to make sure your ML project works.

Machine Learning 122

Machine Learning Project 122

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

KDnuggets

MAY 16, 2022

A simple guide to reinforcement learning for a complete beginner. The blog includes definitions with examples, real-life applications, key concepts, and various types of learning resources.

Machine Learning 122

Machine Learning 122

KDnuggets

MAY 20, 2022

We give a taxonomy of the trustworthy GNNs in privacy, robustness, fairness, and explainability. For each aspect, we categorize existing works into various categories, give general frameworks in each category, and more.

Machine Learning 116

Machine Learning 116

KDnuggets

MAY 17, 2022

Is the data warehouse broken? Is the "immutable data warehouse" the right path for your data team? Learn more here.

Data Warehouse 116

Data Warehouse Data Data Science 116

KDnuggets

MAY 18, 2022

5 Free Hosting Platform For Machine Learning Applications; Data Mesh Architecture: Reimagining Data Management; Popular Machine Learning Algorithms; Reinforcement Learning for Newbies ; Deep Learning For Compliance Checks: What's New?

Machine Learning 108

Machine Learning Architecture Data Management Deep Learning 108

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud