Start Data Engineering
APRIL 22, 2024
1. Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with. Most data projects use Docker to set up the data infra locally (and often in production).
KDnuggets
APRIL 22, 2024
Are you an aspiring data scientist? If so, these free data science courses from Stanford will help you move forward in your data science journey!
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
APRIL 22, 2024
1. Introduction 2. Ensure the code’s logic is working as expected with tests 2.1. Test types for data pipelines 2.2. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4. Define items to be shared among tests with conftest.
KDnuggets
APRIL 22, 2024
The secret recipe to excel in your career in AI.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
ArcGIS
APRIL 22, 2024
A drawing alert notification system was added in ArcGIS Pro 3.2 as a method for resolving drawing issues in your ArcGIS Pro projects.
KDnuggets
APRIL 22, 2024
Leverage the latest technology to improve our search engine capabilities.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Knowledge Hut
APRIL 22, 2024
Anyone aspiring to be a data scientist, machine learning engineer, or software developer must have thought about learning Python. The popularity of this programming language has grown exponentially in the past ten years. Even those unfamiliar with coding have probably heard about it. As per the developer survey by Stack Overflow in 2021 , approximately 68% of software developers or data scientists who have worked on developing software using Python have expressed that they will continue doing so
Scott Logic
APRIL 22, 2024
Generative AI has been grabbing headlines, but many businesses are starting to feel left-behind. Large-model AI is becoming more and more influential in the market, and with the well-known tech giants starting to introduce easy-access AI stacks, a lot of businesses are left feeling that although there may be a use for AI in their business, they’re unable to see what use cases it might help them with.
Zalando Engineering
APRIL 22, 2024
Introduction Our team is responsible for sending out communications to all our customers at Zalando - e.g. confirming a placed order, informing about new content from a favourite brand or announcing sales campaigns. During the preparation of those messages as well during sending those out via different service providers we have to deal with limited resources.
Pinterest Engineering
APRIL 22, 2024
Michelle Vu | Web Performance Engineer Detecting, preventing, and resolving performance regressions has been a standard at Pinterest for many years. Over the years, we have seen many examples showing significant business metric movements resulting from performance optimizations and regressions. These concrete examples motivate us to optimize and maintain performance.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Ascend.io
APRIL 22, 2024
At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Ascend has always been at the forefront of innovation, and with Project Inception, we’re setting a new standard.
Striim
APRIL 22, 2024
Discovery Health, originating in South Africa, has transcended borders to extend its services to over 40 million customers across more than 40 global markets, encompassing regions in Asia, EMEA, and the Americas. Since its inception in 1992, the company has remained steadfast in its core purpose: “to make people healthier and to enhance and protect their lives.” As a multifaceted financial services organization, Discovery Health operates in various sectors including healthcare, life
Scott Logic
APRIL 22, 2024
In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. This time round I’ll look at the messages themselves which over the last few years I’ve found to be a more complex and nuanced topic than expected. To set the scene see the diagram below of an imaginary financial trading application: There’s lots of data flying around varying from real time pricing data to instructions to execute trades.
Expert insights. Personalized for you.
225 
Let's personalize your content