Start Data Engineering
APRIL 22, 2024
1. Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with. Most data projects use Docker to set up the data infra locally (and often in production).
KDnuggets
APRIL 22, 2024
Are you an aspiring data scientist? If so, these free data science courses from Stanford will help you move forward in your data science journey!
Start Data Engineering
APRIL 22, 2024
1. Introduction 2. Ensure the code’s logic is working as expected with tests 2.1. Test types for data pipelines 2.2. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4. Define items to be shared among tests with conftest.
KDnuggets
APRIL 22, 2024
The secret recipe to excel in your career in AI.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
ArcGIS
APRIL 22, 2024
A drawing alert notification system was added in ArcGIS Pro 3.2 as a method for resolving drawing issues in your ArcGIS Pro projects.
KDnuggets
APRIL 22, 2024
Leverage the latest technology to improve our search engine capabilities.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Scott Logic
APRIL 22, 2024
Generative AI has been grabbing headlines, but many businesses are starting to feel left-behind. Large-model AI is becoming more and more influential in the market, and with the well-known tech giants starting to introduce easy-access AI stacks, a lot of businesses are left feeling that although there may be a use for AI in their business, they’re unable to see what use cases it might help them with.
Zalando Engineering
APRIL 22, 2024
Introduction Our team is responsible for sending out communications to all our customers at Zalando - e.g. confirming a placed order, informing about new content from a favourite brand or announcing sales campaigns. During the preparation of those messages as well during sending those out via different service providers we have to deal with limited resources.
Snowflake
APRIL 22, 2024
Magnite stands as the largest independent sell-side advertising platform, providing an essential bridge between publishers and advertisers. At its core, Magnite streamlines the advertising process, facilitating the buying and selling of advertising space across various channels, including connected TV (CTV), mobile, and desktop environments. By leveraging advanced technology and data analytics, Magnite offers a comprehensive suite of tools and services designed to maximize ad revenue for publish
Pinterest Engineering
APRIL 22, 2024
Michelle Vu | Web Performance Engineer Detecting, preventing, and resolving performance regressions has been a standard at Pinterest for many years. Over the years, we have seen many examples showing significant business metric movements resulting from performance optimizations and regressions. These concrete examples motivate us to optimize and maintain performance.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Ascend.io
APRIL 22, 2024
At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Ascend has always been at the forefront of innovation, and with Project Inception, we’re setting a new standard.
Striim
APRIL 22, 2024
Discovery Health, originating in South Africa, has transcended borders to extend its services to over 40 million customers across more than 40 global markets, encompassing regions in Asia, EMEA, and the Americas. Since its inception in 1992, the company has remained steadfast in its core purpose: “to make people healthier and to enhance and protect their lives.” As a multifaceted financial services organization, Discovery Health operates in various sectors including healthcare, life
Scott Logic
APRIL 22, 2024
In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. This time round I’ll look at the messages themselves which over the last few years I’ve found to be a more complex and nuanced topic than expected. To set the scene see the diagram below of an imaginary financial trading application: There’s lots of data flying around varying from real time pricing data to instructions to execute trades.
Expert insights. Personalized for you.
225 
Let's personalize your content