This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookie Settings
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookie Settings
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Strictly Necessary: Used for the proper function of the website
Performance/Analytics: Used for monitoring website traffic and interactions
Summary The information about how data is acquired and processed is often as important as the data itself. For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. These systems are frequently cumbersome and difficult to maintain, so Octopai was founded to alleviate that burden.
Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis. This analysis powers our services and enables the delivery of more seamless and reliable user … The post Scaling Uber’s Apache Hadoop Distributed File System for Growth appeared first on Uber Engineering Blog.
A product manager's insights on customization Personalization is a common term with digital products. But what does it actually mean, why do we do it, and how does it affect the product manager? To illustrate, let me tell a personal story. I have gone to the same hairdresser for 10 years. He has seen a big part of my life, with big changes and evolutions.
[This blog by Claudia Juech, Executive Director of the Cloudera Foundation, highlights how increased collaboration between different philanthropic organizations can result in better funding for critical social issues. By adopting technologies like machine learning and analytics, these organizations can optimize how they spend funds for social good. She highlights how this technology can impact society.].
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
News on Hadoop - March 2018 Kyvos Insights to Host Session "BI on Big Data - With Instant Response Times" at the Gartner Data and Analytics Summit 2018.PRNewswire.com, March 5, 2018 The big data analytics company Kyos Insights announced that it will host a session “BI on Big Data - With Instant Response Times” at the Gartner Data and Analytics Summit 2018 conference in Grapevine, Texas from March 5-8, 2018.
Summary Business Intelligence software is often cumbersome and requires specialized knowledge of the tools and data to be able to ask and answer questions about the state of the organization. Metabase is a tool built with the goal of making the act of discovering information and asking questions of an organizations data easy and self-service for non-technical users.
Summary The rate of change in the data engineering industry is alternately exciting and exhausting. Joe Crobak found his way into the work of data management by accident as so many of us do. After being engrossed with researching the details of distributed systems and big data management for his work he began sharing his findings with friends. This led to his creation of the Hadoop Weekly newsletter, which he recently rebranded as the Data Engineering Weekly newsletter.
Summary The rate of change in the data engineering industry is alternately exciting and exhausting. Joe Crobak found his way into the work of data management by accident as so many of us do. After being engrossed with researching the details of distributed systems and big data management for his work he began sharing his findings with friends. This led to his creation of the Hadoop Weekly newsletter, which he recently rebranded as the Data Engineering Weekly newsletter.
Summary Managing an analytics project can be difficult due to the number of systems involved and the need to ensure that new information can be delivered quickly and reliably. That challenge can be met by adopting practices and principles from lean manufacturing and agile software development, and the cross-functional collaboration, feedback loops, and focus on automation in the DevOps movement.
Summary Cloud computing and ubiquitous virtualization have changed the ways that our applications are built and deployed. This new environment requires a new way of tracking and addressing the security of our systems. ThreatStack is a platform that collects all of the data that your servers generate and monitors for unexpected anomalies in behavior that would indicate a breach and notifies you in near-realtime.
As the Vice President of Engineering for Uber’s Core Infrastructure group, Matthew Mengerink faces a daunting task. He oversees 350 engineers across four teams tasked with not only maintaining the platform on which 3,500 microservices run, but also figuring out … The post Scaling for Growth: A Q&A with Uber’s VP of Core Infrastructure, Matthew Mengerink appeared first on Uber Engineering Blog.
How stepping out of our comfort zone led to a hackathon victory Zalando Tech doesn't just put on hackathons , we love to attend them too! Here, we catch up with software engineers, Lisa Knolle and Izabela Bratovic about their time at #picturepunk. At the end of last year we took part in a hackathon. We came to this decision for the sake of exposing ourselves to new experiences, new people, and new technologies.
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
How we migrated the Zalando Logistics Operating Services to Java 8 “Never touch working code!” goes the old saying. How often do you disregard this message and touch a big monolithic system? This article tells you why you should ignore common wisdom and, in fact, do it even more often. Preface Various kinds of migration are a natural part of software development.
Using an API to drive marketing profitability: a gift card study Gift cards are becoming increasingly popular in the US and Europe. For time-pressed consumers trying to find a convenient gift for friends and family, gift cards are an easy solution, and it shows: gift cards are projected to grow at a 24% Compound Annual Growth Rate (CAGR) until 2023, According to Allied Market Research.
Using Akka cluster-sharding and Akka HTTP on Kubernetes This article captures the implementation of an application serving data over HTTP which is stored in cluster-sharded actors and deployed on Kubernetes. Use case: An application, serving data over HTTP and with a high request rate, and the latency of order of 10ms with limited database IOPS available.
Our experience of The Sprint About two years ago, Jake Knapp , John Zeratsky and Braden Kowitz from Google Ventures published “ The Sprint.” They describe a methodology that helps you answer critical business questions, develop ideas, or tackle problems in just five days, and last year Jake Knapp shared his insights in a fireside chat at Zalando. Last week, we had the chance to see it in action.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
How data science is becoming available ‘for the good of all’ businesses In his 2010 Ted Talk “ When Ideas Have Sex ,” Matt Ridley posits that human prosperity was caused by one thing and one thing only; our unique human ability to specialise and exchange ideas and tools. Ridley’s example of the invention of the reading light illustrates how far we’ve come.
We organize all of the trending information in your field so you don't have to. Join 37,000+ users and stay up to date on the latest articles your peers are reading.
You know about us, now we want to get to know you!
Let's personalize your content
Let's get even more personalized
We recognize your account from another site in our network, please click 'Send Email' below to continue with verifying your account and setting a password.
113 
Let's personalize your content