Start Data Engineering
MAY 11, 2022
1. Introduction 2. Objective 3. Design 4. Setup 4.1 Prerequisite 4.2 AWS Infrastructure costs 4.3 Data lake structure 5. Code walkthrough 5.1 Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Design considerations 8.
KDnuggets
MAY 13, 2022
This article is the first in a series of articles looking at the different aspects of k-means clustering, beginning with a discussion on centroid initialization.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confluent
MAY 12, 2022
Internet connectivity is something we sometimes take for granted. For many, most places we visit, work, or reside have some form of connectivity whether it be cellular, Wi-Fi, fiber, etc. […].
AltexSoft
MAY 12, 2022
We live in the world of sounds: Pleasant and annoying, low and high, quiet and loud, they impact our mood and our decisions. Our brains are constantly processing sounds to give us important information about our environment. But acoustic signals can tell us even more if analyze them using modern technologies. Today, we have AI and machine learning to extract insights, inaudible to human beings, from speech, voices, snoring, music, industrial and traffic noise, and other types of acoustic signals
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Cloudera
MAY 9, 2022
Tuning Hive on Tez queries can never be done in a one-size-fits-all approach. The performance on queries depends on the size of the data, file types, query design, and query patterns. During performance testing, evaluate and validate configuration parameters and any SQL modifications. It is advisable to make one change at a time during performance testing of the workload, and would be best to assess the impact of tuning changes in your development and QA environments before using them in product
KDnuggets
MAY 12, 2022
Learn about the free and easy-to-deploy hosting platform for your machine learning projects.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
MAY 8, 2022
Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. These connections are best represented and analyzed as graphs to provide efficient and accurate analysis of their relationships. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning.
Cloudera
MAY 13, 2022
Cloudera Data Platform (CDP) unifies the technologies from Cloudera Enterprise Data Hub (CDH) and Hortonworks Data Platform (HDP). A few functionalities that existed in the legacy platforms (HDP and CDH) are substituted by other alternatives based on a detailed and careful analysis. CDH users would have used Fair Scheduler (FS), and HDP users would have used Capacity Scheduler (CS).
KDnuggets
MAY 9, 2022
Read this overview of 12 important machine learning concepts, presented in a no frills, straightforward definition style.
Teradata
MAY 12, 2022
The next generation of customers expects newer technologies & advanced self-service capabilities as the airline business becomes more competitive. How can airlines meet these expectations?
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Data Engineering Podcast
MAY 8, 2022
Summary Dan Delorey helped to build the core technologies of Google’s cloud data services for many years before embarking on his latest adventure as the VP of Data at SoFi. From being an early engineer on the Dremel project, to helping launch and manage BigQuery, on to helping enterprises adopt Google’s data products he learned all of the critical details of how to run services used by data platform teams.
Rockset
MAY 12, 2022
This is the third post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Posts published so far in the series: Why Mutability Is Essential for Real-Time Data Analytics Handling Out-of-Order Data in Real-Time Analytics Applications Handling Bursty Traffic in Real-Time Analytics Applications SQL and Complex Queries A
KDnuggets
MAY 10, 2022
This is a list of FREE data science resources and notes that are available online, some of which are provided by universities.
Rock the JVM
MAY 11, 2022
Scala generics are a breeze for Java developers, but what about those coming from Python or JavaScript?
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Monte Carlo
MAY 11, 2022
If you’ve been a data analyst, BI analyst, or general business user of dashboards and reports, you’ve probably asked these questions (and more) before: What’s the most reliable field to use? When was the last time this table was updated? Should there be this many null entries in this column? Who can I reach out to figure out if this data is expected?
Palantir
MAY 11, 2022
(Scroll down for English translation below) Avec l’adoption des accords de Paris en 2016, les institutions du secteur public et privé ont considérablement renforcé leurs ambitions en matière de décarbonation. Plus particulièrement, la capacité des organisations à s’adapter et améliorer leur prise de décision va devenir un élément clé de différenciation et de compétitivité.
KDnuggets
MAY 12, 2022
By implementing the different NLP techniques into the production processes, compliance departments can maintain detailed checks and keep up with regulator demands.
Rockset
MAY 10, 2022
DynamoDB is a popular NoSQL database available in AWS. It is a managed service with minimal setup and pay-as-you-go costing. Developers can quickly create databases that store complex objects with flexible schemas that can mutate over time. DynamoDB is resilient and scalable due to the use of sharding techniques. This seamless, horizontal scaling is a huge advantage that allows developers to move from a proof of concept into a productionized service very quickly.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
dbt Developer Hub
MAY 9, 2022
❓ Who is this for: This is for advanced users of dbt Cloud that are interested in expanding their knowledge of the dbt API via an interactive Postman Collection. We only suggest diving into this once you have a strong knowledge of dbt + dbt Cloud. You have a couple of options to review the collection: get a live version of the collection via. check out the collection documentation to learn how to use it.
KDnuggets
MAY 11, 2022
Save time and effort with this guide, which will show you how to do data join operations in Tableau.
KDnuggets
MAY 9, 2022
Check out this list of free resources, courses, and more to help you become a Data Scientist for free.
KDnuggets
MAY 13, 2022
Predict the performance of your model - before the ground truth is available.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
MAY 10, 2022
While it is true that Machine Learning today isn’t ready for prime time in many business cases that revolve around Document Analysis, there are indeed scenarios where a pure ML approach can be considered.
KDnuggets
MAY 11, 2022
If you aren't familiar with Kaggle, you should be. Hear why from two expert Kagglers in this article.
KDnuggets
MAY 11, 2022
The objective of data mesh is to establish coherence between data coming from different domains across an enterprise. The domains are handled autonomously to eliminate the challenges of data availability and accessibility for cross-functional teams.
KDnuggets
MAY 10, 2022
How To Tutorials with SAS data scientists and analytics instructors.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
KDnuggets
MAY 13, 2022
In this article, we will build a beginner-friendly machine learning model using TensorFlow.
KDnuggets
MAY 11, 2022
SQL Notes for Professionals: The Free eBook Review; How To Structure a Data Science Project: A Step-by-Step Guide; Everything You Need to Know About Tensors; Free University Data Science Resources; Image Classification with Convolutional Neural Networks (CNNs).
KDnuggets
MAY 10, 2022
Good planning and preparation will not only improve productivity, but it will help avoid potential pitfalls and roadblocks that could be encountered during project execution.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Expert insights. Personalized for you.
246 
Let's personalize your content