Data Engineering Podcast

JUNE 30, 2024

Summary This episode features an insightful conversation with Petr Janda, the CEO and founder of Synq. Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

KDnuggets

JULY 4, 2024

So you want to become a software developer? Start coding your way through these free certifications today.

Certification 144

Certification Coding Programming 144

databricks

JULY 2, 2024

Databricks announced the public preview of Mosaic AI Agent Framework & Agent Evaluation alongside our Generative AI Cookbook at the Data + AI.

Data 142

Data Data Science 142

Robinhood

JULY 1, 2024

Robinhood Markets, Inc. is excited to announce the acquisition of Pluto Capital Inc., an artificial intelligence (AI) powered investment research platform that delivers highly-customized investment strategies based on customer needs and financial goals. With this strategic acquisition, investors can look forward to a new era of intelligent, data-driven investing at Robinhood.

Portfolio 135

Portfolio Finance Retail Algorithm 135

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Seattle Data Guy

JULY 2, 2024

Running a successful data team is hard. Data teams are expected to juggle a combination of ad-hoc requests, big bet projects, migrations, etc. All while keeping up with the latest changes in technology. In the past few years I have gotten to work with dozens of teams and see how various directors and managers deal… Read more The post 9 Habits Of Effective Data Managers – Running A Data Team appeared first on Seattle Data Guy.

Data Management 130

Data Management Management Data Project 130

KDnuggets

JULY 5, 2024

Learn essential techniques for tuning hyperparameters to enhance the performance of your neural networks.

Machine Learning 139

Machine Learning 139

Start Data Engineering

JULY 1, 2024

1. Introduction 2. Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Chose coding interface for people who will maintain the pipeline 3.2.1. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

ArcGIS

JULY 5, 2024

Learn more about the collection of data models, workflows, and planning tools tailored for flooding available in ArcGIS Pro 3.3.

Data 113

Data Management 113

KDnuggets

JULY 2, 2024

Learn SQL, Python, statistics, mathematics, and data analysis—everything you need to learn before you start the journey of becoming a professional data scientist.

Data Science 137

Data Science SQL Data Analysis Python 137

Snowflake

JULY 1, 2024

How insurance leaders can use the power of data and AI to transform the industry, from claims analytics to risk selection and beyond There is a growing recognition that insurers can introduce data, analytics and AI into virtually all of the important insurance functions and workflows, including product development, pricing and risk selection, underwriting, claims management, contact center optimization, distribution management, reinsurance, and understanding and shaping customer journeys.

Insurance 111

Insurance Media Machine Learning Unstructured Data 111

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

databricks

JULY 1, 2024

Mixture-of-Experts (MoE) has emerged as a promising LLM architecture for efficient training and inference. MoE models like DBRX , which use multiple expert.

Architecture 105

Architecture 105

ArcGIS

JUNE 30, 2024

In June 2024, elevation layers have been updated with lidar derived DTM’s of Slovakia, Belgium, San Mateo County (USA) along with USGS 3DEP.

Data 105

Data 105

KDnuggets

JULY 2, 2024

In today's data science landscape, how does one set themselves apart from the competition? Let’s take a look at seven of the best certifications out there.

Data Science 134

Data Science Certification Data 134

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Ozge Demirci, Jonas Hannane & Xinrong Zhu: Who Is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms The economic impact of Gen AI is widely speculated, and we see few signs of impact.

Data Engineer 102

Data Engineer Data Engineering Engineering Data Pipeline 102

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Knowledge Hut

JULY 2, 2024

Cloud computing is changing faster than we ever imagined. Every day, new features and capabilities have been released that change how we think about, use, and administer cloud services. Thus, the cloud computing future looks pretty bright and stable. There is no doubt that the cloud has disrupted the traditional IT landscape, and the momentum of cloud computing shows no signs of abating.

Cloud Computing 98

Cloud Computing Cloud Healthcare Education 98

Scott Logic

JULY 4, 2024

More and more I am of the opinion that putting points against stories is a waste of time. I’ve spent many hours, as I’m sure have you, sitting in meetings of various shapes and sizes guessing numbers and looking back I’m starting to question if it was really worth it. I’ll say upfront, I’m going to be fairly critical of story pointing here, I’m not just being a grumpy old Yorkshireman!

IT 97

IT Process Building Project 97

KDnuggets

JULY 3, 2024

Should I bother pursuing a career in data science in 2024?

Data Science 134

Data Science IT Data 134

Towards Data Science

JULY 4, 2024

From CSVs to databases: loading data into PySpark DataFrames Continue reading on Towards Data Science »

Data Science 75

Data Science Database Data Programming 75

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Tweag

JULY 3, 2024

Making great animated graphics on mobile apps has always been challenging. While react-native-svg has served React Native developers well for basic vector graphics, it often falls short when it comes to replicating the more complex effects seen in web applications. We’ll be integrating Skia for rendering sharp, efficient 2D graphics and Reanimated for creating fluid, responsive animations.

Building 63

Building Coding Project Engineering 63

Precisely

JULY 3, 2024

Although technology has historically been a male-dominated industry, more women are continuing to enter the field. With this, more resources and programs have emerged which help girls learn about tech hobbies and career possibilities. Precisely supports the growth of women in the industry and as a result, established the Precisely Women in Technology (PWIT) Program which supports women at the company.

Technology 59

Technology Programming Education Project 59

KDnuggets

JULY 3, 2024

Explore the simplicity and flexibility of duck typing in Python — where code adapts based on behavior, not rigid types!

Coding 133

Coding Python 133

RandomTrees

JUNE 29, 2024

In today’s data-driven world, organizations are increasingly turning to advanced analytics to gain deeper insights and make informed decisions. Power BI, Microsoft’s powerful business analytics tool, offers a robust platform for harnessing the full potential of data. Here are some best practices to maximize the impact of advanced analytics in Power BI: 1.

BI 59

BI Machine Learning Government Data Analysis 59

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Knowledge Hut

JULY 5, 2024

In my experience, data silos have emerged as a significant challenge for organizations. Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Hadoop 52

Scott Logic

JULY 5, 2024

The impending Digital Operational Resilience Act (DORA) aims to strengthen the IT security of financial entities such as banks, insurance companies and investment firms across Europe. While the regulations will standardise ICT risk management, business continuity, and incident response, they won’t recommend best practice resilience strategies that banks should adopt.

Amazon Web Services 52

Amazon Web Services Banking Google Cloud Insurance 52

KDnuggets

JULY 5, 2024

In this blog, we will define Pandas and provide an example of how you can vectorize your Python code to optimize dataset analysis using Pandas to speed up your code over 300x times faster.

Python 126

Python Datasets Coding 126

Edureka

JULY 5, 2024

Amazon Machine Image (AMI) is an image in the public or private cloud storage that stores information relating to virtual machines known as instances in Amazon’s Elastic Compute Cloud (EC2). In the following article, you will learn more about in addition to how and the details of Amazon AMI Image and some of the subclasses in the virtualization of Amazon Linux AMIs.

AWS 52

AWS Amazon Web Services Cloud Cloud Storage 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Knowledge Hut

JULY 5, 2024

The DevOps market is expected to reach USD 12,215.54 million by 2026 at a compound annual growth rate of 18.95% according to reports published by the Global DevOps Market Research Report (2021 to 2026). The DevOps market is rapidly eliminating conflicts between the operations team and the development team, which was one of the biggest challenges faced by companies so far.

Certification 52

Certification Java Cloud Education 52

Hevo

JULY 5, 2024

Data practitioners often need manual intervention to load large volumes of data into Snowflake in near real-time. Traditional batch loading can be slow and intensive and may lead to latency and increased operational costs. Enter Snowflake Snowpipe.

Data 52

Data Process Data Integration 52

KDnuggets

JULY 1, 2024

Let's take a look at how to navigate the Unix/Linux filesystem using bash.

Programming 125

Programming 125

Edureka

JULY 4, 2024

Amazon Elastic File System (EFS) is a service that Amazon Web Services ( AWS ) provides. It is intended to deliver serverless, fully-elastic file storage that enables you to share data independently of capacity and performance. This article aims to explain what is AWS Elastic File System and the features that make it stand out, the available choices of backups, how to create an EFS file system, and providing you with helpful FAQs about this tool and how to gain maximum from it successfully.

AWS 52

AWS Systems Amazon Web Services Cloud Storage 52

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer