The Pragmatic Engineer

SEPTEMBER 19, 2024

This is a follow-up to the article Paying down tech debt , written by industry veteran Lou Franco. Lou has been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app. Later this year, he’s publishing a book on tech debt.

Recruitment 297

Recruitment Java Coding Project 297

Start Data Engineering

SEPTEMBER 18, 2024

1. Introduction 2. Setup 3. Parts of data engineering 3.1. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4. Define checks to ensure the output dataset is usable 3.2. Identify what tool to use to process data 3.3. Data flow architecture 3.

Project 240

Project Building Datasets Architecture 240

Jesse Anderson

SEPTEMBER 18, 2024

Survey Changes Over Time Between 2020 and 2024 (see 2020, 2023, and 2024 for each year’s information), I’ve been conducting a data teams survey. I wanted to dedicate an entire post to examining the change in data teams over time. Total Value Creation The most important question I ask each year concerns data team value creation. I break the question into two parts: “How successful would the business say your projects are?

Data Warehouse 162

Data Warehouse Data Big Data Project 162

KDnuggets

SEPTEMBER 12, 2024

Develop unique yet standing-out data science projects to improve your data portfolio.

Data Science 156

Data Science Project Portfolio Data 156

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge. Picture this: thus, unlike many other extensions that require deep setup and constant coding, […] The post Setup Mage AI with Postgres to Build and Manage Your Data Pipeline appeared first on Analytics Vidhy

Data Pipeline 147

Data Pipeline Building Management Data 147

Confluent

SEPTEMBER 9, 2024

Confluent has acquired WarpStream, an innovative Kafka-compatible streaming solution. Read the full statement by Jay Kreps, co-founder and CEO of Confluent.

Cloud 142

Cloud Kafka 142

Start Data Engineering

SEPTEMBER 23, 2024

1. Introduction 2. Steps to decide on a data project to build 2.1. Objective 2.2. Research 2.2.1. Job description 2.2.2. Potential referral/hiring manager research 2.2.3. Company research 2.3. Data 2.3.1. Dataset Search 2.3.2. Generate fake data 2.4. Outcome 2.4.1. Visualization 2.5. Presentation 3. Conclusion 4. Read these 1.

Portfolio 130

Portfolio Project Datasets Data 130

Seattle Data Guy

SEPTEMBER 20, 2024

We are still in the early days of data and the value it can add to companies. You’ll read plenty of statistics about how much value data can drive and how far behind companies that aren’t using data are. And as a data consultant, I have helped companies find that value in their data. It… Read more The post How To Modernize Your Data Strategy And Infrastructure For 2025 appeared first on Seattle Data Guy.

Consulting 130

Consulting Data IT Data Warehouse 130

KDnuggets

SEPTEMBER 26, 2024

Are you an aspiring data scientist or early in your data science career? If so, you know that you should use your programming, statistics, and machine learning skills—coupled with domain expertise—to use data to answer business questions. To succeed as a data scientist, therefore, becoming proficient in coding is essential. Especially for handling and analyzing.

Data Science 154

Data Science Coding Machine Learning Data 154

ArcGIS

SEPTEMBER 18, 2024

Learn the difference between ArcGIS Data Interoperability and FME technology and how they relate to one another.

Data 127

Data Technology Data Management Management 127

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Confluent

SEPTEMBER 5, 2024

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 139

Kafka Data 139

databricks

SEPTEMBER 19, 2024

Mosaic AI Model Training now supports fine-tuning up to 131K context length for Llama 3.1 models. More efficient training at long sequence lengths is made possible by several optimizations highlighted in this post.

138

138

Start Data Engineering

SEPTEMBER 4, 2024

1. Introduction 2. Key parts of data systems: 2.1. Requirements 2.2. Data flow design 2.3. Orchestrator and scheduler 2.4. Data processing design 2.5. Code organization 2.6. Data storage design 2.7. Monitoring & Alerting 2.9. Infrastructure 3. Conclusion 1. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Seattle Data Guy

SEPTEMBER 3, 2024

One of the holy grails that many data teams seem to chase is real-time data analytics. After all, if you can have real-time analytics, you can make better decisions faster. However, there often is a conflation between real-time data analytics and stream processing. These are two different concepts that are crucial to understanding how to… Read more The post Real-time Analytics Vs Stream Processing – What Is The Difference?

Process 130

Process Data Analytics Data Big Data 130

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 151

Python Data Engineering Data Engineer Engineering 151

Engineering at Meta

SEPTEMBER 10, 2024

We’re sharing more about the role that reinforcement learning plays in helping us optimize our data centers’ environmental controls. Our reinforcement learning-based approach has helped us reduce energy consumption and water usage across various weather conditions. Meta is revamping its new data center design to optimize for artificial intelligence and the same methodology will be applicable for future data center optimizations as well.

Data 122

Data Designing Systems Building 122

Precisely

SEPTEMBER 23, 2024

Are mainframes still relevant today? You bet! The following ten statistics paint a picture that shows mainframes are still going strong, with no signs of slowing. 1. The Mainframe Turns 60: A Milestone in Computing History. 60 years can really fly by! On April 7, 2024 , the Mainframe turned 60. At this milestone, we should all reflect on what the mainframe has done to the computing industry.

Banking 116

Banking Insurance Finance Systems 116

databricks

SEPTEMBER 5, 2024

An improved answer-correctness judge in Agent Evaluation Agent Evaluation enables Databricks customers to define, measure, and understand how to improve the quality of.

Data Science 137

Data Science Data 137

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Confessions of a Data Guy

SEPTEMBER 24, 2024

I recently wrote on my Substack (Data Engineering Central) about how I used the new OpenAI o1 model to do some basic Data Engineering tasks surrounding PostgreSQL. It did ok. I’ve also been using CoPilot and ChatGPT for over a year now to assist me with my daily code that I have to write for […] The post AI (LLMs) and Software Engineering (Writing Code) appeared first on Confessions of a Data Guy.

Software Engineer 113

Software Engineer Software Engineering Coding Engineering 113

ArcGIS

SEPTEMBER 24, 2024

Interested in 3D GIS but not sure where to start? Learn the proper method to transform pre-existing 2D footprint polygons into a 3D buildings.

Building 113

Building 113

KDnuggets

SEPTEMBER 12, 2024

The GitHub repository includes up-to-date learning resources, research papers, guides, popular tools, tutorials, projects, and datasets.

Datasets 149

Datasets Project 149

Engineering at Meta

SEPTEMBER 4, 2024

We are working in partnership with others to scale inclusive solutions that support the transition to a zero-carbon economy and help create a healthier planet for all.

Engineering 117

Engineering Data 117

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Confluent

SEPTEMBER 24, 2024

Learn how your data goes from a producing client all the way to disk on a broker—along the way traversing buffers, threads, queues and more.

Kafka 111

Kafka Data 111

databricks

SEPTEMBER 25, 2024

We are excited to partner with Meta to launch the latest models in the Llama 3 series on the Databricks Data Intelligence Platform.

Data 135

Data Data Science 135

Confessions of a Data Guy

SEPTEMBER 19, 2024

It is a Brave New World out there these days. The new tools and features come out faster than your mom on Sunday morning getting you ready for church. The same goes for the context and advice being produced on a myriad of platforms, the ole’ Like and Subscribe, and all that bit. It does […] The post Data Modeling in the Brave New Lakehouse World appeared first on Confessions of a Data Guy.

Data 113

Data IT Data Engineering Data Engineer 113

ArcGIS

SEPTEMBER 25, 2024

To publish different views of the same source data, alter map layer settings before you publish each web feature layer.

Data 109

Data Data Management Management 109

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

KDnuggets

SEPTEMBER 11, 2024

Kickstart your data analyst career with all these free courses.

Data Analytics 149

Data Analytics Data Data Science 149

Engineering at Meta

SEPTEMBER 17, 2024

This episode of the Meta Tech Podcast is all about Bento , Meta’s internal distribution of Jupyter Notebooks, an open-source web-based computing platform. Bento allows our engineers to mix code, text, and multimedia in a single document and serves a wide range of use cases at Meta from prototyping to complex machine learning workflows. Pascal Hartig ( @passy ) is joined by Steve, whose team has built several features on top of Jupyter, including scheduled notebooks , sharing with colleagues, and

Machine Learning 109

Machine Learning Engineering Coding Data 109

Snowflake

SEPTEMBER 24, 2024

Today, Snowflake is officially launching the AI Data Cloud for Travel and Hospitality. Snowflake’s newest AI Data Cloud offers a unified and secure platform that streamlines AI and ML development to support the growth of travel and hospitality businesses, empowering organizations to harness their data’s full potential. With Snowflake and its ecosystem of partners, travel and hospitality businesses can integrate and analyze valuable third-party data to deliver top-notch customer experiences and m

Hospitality 104

Hospitality Cloud Database-centric Data 104

databricks

SEPTEMBER 23, 2024

We are excited to announce that Databricks now supports Amazon EC2 G6 instances powered by NVIDIA L4 Tensor Core GPUs. This addition marks.

Data Science 131

Data Science Engineering Data 131

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering