Start Data Engineering

JANUARY 20, 2025

1. Introduction 2. Guide the interviewer through the process 2.1. [Requirements gathering] Make sure you clearly understand the requirements & business use case 2.2. [Understand source data] Know what you have to work with 2.3. [Model your data] Define data models for historical analytics 2.4. [Pipeline design] Design data pipelines to populate your data models 2.5.

Designing 130

Designing Systems Data Engineering Data Engineer 130

KDnuggets

JANUARY 20, 2025

Whether youre starting from scratch or building on existing skills, this hands-on program teaches you how to import, clean, and visualize data from day one using libraries like pandas, Seaborn, and Matplotlib. Plus, earn an industry-recognized certification to showcase your expertise and stand out in the job market.

Data Science 120

Data Science Python Certification Programming 120

Towards Data Science

JANUARY 20, 2025

Here’s how your data retains its business relevance as it travels through your enterprise Continue reading on Towards Data Science

Engineering 83

Engineering Data Science Data IT 83

KDnuggets

JANUARY 20, 2025

Our latest infographic breaks down 10 of the most common and enduring myths about data science, offering clarity on the misconceptions that often surround this rapidly evolving field.

Data Science 119

Data Science Data 119

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Towards Data Science

JANUARY 20, 2025

From AI Agent to Human-In-The-Loop — Master 12 critical data concepts and turn them into simple projects to stay ahead in IT.

Data Science 83

Data Science Data Project IT 83

KDnuggets

JANUARY 20, 2025

Check out these 10 ways to leverage efficient distributed dataset processing combining the strengths of Spark and Python libraries for data science.

Big Data 108

Big Data Data Process Process Datasets 108

KDnuggets

JANUARY 20, 2025

Unsure what to learn, where to start, and which order to follow to master LLM engineering concepts and skills? This comprehensive roadmap with clear milestones and stages is here to help!

Engineering 103

Engineering 103

Towards Data Science

JANUARY 20, 2025

Here’s how your data retains its business relevance as it travels through your enterprise Continue reading on Towards Data Science

Engineering 40

Engineering Data Science Data IT 40

Zalando Engineering

JANUARY 20, 2025

Enhancing the Security of Our Customer Identity Platform Through Automated Key Rotation Static secrets are evil. Whether secret keys hard-coded in source code, tokens without expiry or plaintext API keys referenced in configuration files, static secrets are ticking time bombs. The same is true for cryptographic key material in the context of JSON Web Tokens (JWTs) and OpenID Connect (OIDC).

Accessible 104

Accessible Accessibility Coding Process 104

Snowflake

JANUARY 20, 2025

Imagine a world where AI recommends an optimal product, speeds up medical breakthroughs and predicts financial trends this isn't the future; it's happening now across retail, healthcare and financial services. Having worked through major market transitions before such as the emergence of cloud computing and the rise of AI/ML I understand the importance of taking a thoughtful approach to technology and prioritizing real-world impact.

Retail 97

Retail Healthcare Medical Insurance 97

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Data Engineering Weekly

JANUARY 20, 2025

Try Fully Managed Apache Airflow for FREE Astro is the fully-managed DataOps platform powered by Apache Airflow. With Astro, you can build, run, and observe your data pipelines in one place, ensuring your mission critical data is delivered on time. Try Astro Free → Julia Wiesinger, Patrick Marlow, and Vladimir Vuskovic: Agents The combination of reasoning, logic, and access to external information that are all connected to a Generative AI model invokes the concept of an agent.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Snowflake

JANUARY 20, 2025

Security has been an integral capability of Snowflake since the company was founded. Through the customer-configurable security capabilities of the Snowflake Horizon Catalog , we empower security admins and chief information security officers (CISOs) to better protect their environments and centralize threat monitoring and role-based access controls across clouds.

Database-centric 97

Database-centric Cloud Accessible Accessibility 97

Zalando Engineering

JANUARY 20, 2025

OpenTelemetry is a vendor-neutral, flexible standard that supports traces, metrics, and logs all in one place. Organizations who adopted older tracing solutions like OpenTracing or custom legacy tracer libraries to instrument their applications are faced with a migration task. Today, were excited to announce the Lightstep Receiver for OpenTelemetry Collector, a component capable to receive tracing traffic from legacy Lightstep tracers, convert it to OpenTelemetry and propagate via OpenTelemetry

Raw Data 66

Raw Data AWS Python Systems 66

dbt Developer Hub

JANUARY 20, 2025

When my wife and I renovated our home, we chose to take on the role of owner-builder. It was a bold (and mostly naive) decision, but we wanted control over every aspect of the project. What we didnt realize was just how complex and exhausting managing so many moving parts would be. My wife pondering our sanity We had to coordinate multiple elements: The architects , who designed the layout, interior, and exterior.

Data Pipeline 52

Data Pipeline Architecture Cloud Data Workflow 52

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Striim

JANUARY 20, 2025

Manaen Schlabach, Data Administrator at The Motley Fool, shares how Snowflake and Striim enable reliable, scalable, and cost-effective data delivery to support smarter investing tools like Fool IQ. By integrating Snowflake and Striim, The Motley Fool achieved a 10x improvement in the reliability and timeliness of their replication processes. The unified solution, deployed in less than 20 days, tracks membership and campaign activity, allowing timely adjustments to increase value for members.

Process 52

Process Data 52