Uber Engineering
JULY 22, 2021
Introduction. As Uber’s business grew, we scaled our Apache Hadoop (referred to as ‘Hadoop’ in this article) deployment to 21000+ hosts in 5 years, to support the various analytical and machine learning use cases. We built a team with varied … The post Containerizing Apache Hadoop Infrastructure at Uber appeared first on Uber Engineering Blog.
Confluent
JULY 20, 2021
We’re pleased to announce ksqlDB 0.19.0! This release includes a new NULLIF function and a major upgrade to ksqlDB’s data modeling capabilities—foreign-key joins. We’re excited to share this highly requested […].
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
JULY 21, 2021
Introduction Using Native Python Using Pydantic Pydantic Caveats Conclusion Further reading References Introduction Data type issues are one of the biggest concerns when processing data in python. If you are wondering how to Make sure that a column is of a specific data type ( e.g.
Cloudera
JULY 22, 2021
My name is Shanmukha Kota and I am a recent graduate from University at Buffalo. I interned with Cloudera last summer and joined Cloudera as a software engineer a couple of weeks ago and this is my first experience with CDP and CDP Operational Database. For a new hire college graduate in the industry with only academic experience with HBase, I can only say it is very simple and easy to set up and work with CDP Operational Database.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Data Engineering Podcast
JULY 22, 2021
Summary Collecting and cleaning data is only useful if someone can make sense of it afterward. The latest evolution in the data ecosystem is the introduction of a dedicated metrics layer to help address the challenge of adding context and semantics to raw information. In this episode Nick Handel shares the story behind Transform, a new platform that provides a managed metrics layer for your data platform.
DataKitchen
JULY 21, 2021
The post Does Your Organization Need a Chief Data Officer? Probably first appeared on DataKitchen.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Cloudera
JULY 19, 2021
Meet Veda Kadam. She’s relatively new to the Cloudera family. She started her journey here in June of 2020 when she joined our first ever fully virtual intern program. Now she’s a full time employee working as a Software Engineer on our Data In Motion team. From an early age, Veda knew she wanted to work in the technology industry. Her father worked in pharmaceuticals and her mother worked in accounting.
Data Engineering Podcast
JULY 19, 2021
Summary Data quality is a concern that has been gaining attention alongside the rising importance of analytics for business success. Many solutions rely on hand-coded rules for catching known bugs, or statistical analysis of records to detect anomalies retroactively. While those are useful tools, it is far better to prevent data errors before they become an outsized issue.
Rockset
JULY 22, 2021
Social gaming is on the rise. During COVID-19, 29% of consumers reported playing games on a weekly basis and the goal for many players was to connect with friends and family ( Deloitte: Games and Streaming Services Fight it Out During Pandemic from VentureBeat ). One of the challenges that gaming companies face is rapidly building features that can strengthen network effects.
Teradata
JULY 22, 2021
Learn about the techniques and frameworks needed to build a more resilient, cost-effective, and efficient data & analytic decisioning support capability for the post-pandemic supply chain.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Cloudera
JULY 20, 2021
Most businesses, whether you are in Retail, Manufacturing, Specialty Chemicals, Telecommunications, consider a 10% market capitalization increase from 2020 to 2021 outstanding. But what would you say to your shareholders when they found out your competitors’ market capitalization grew 35%? A recent McKinsey report dove into the divergence between retail’s laggards and winners and found if there is one message in the retail sector’s stock market performance since the pandemic’s start, it is
DataKitchen
JULY 22, 2021
Learn about four data architectures patterns for agility - DataOps, Data Fabric, Data Mesh & Functional Data Engineering - & an example combining all four. The post DataOps: The Foundation for Your Agile Data Architecture first appeared on DataKitchen.
ProjectPro
JULY 21, 2021
As a big data architect or a big data developer, when working with Microservices-based systems, you might often end up in a dilemma whether to use Apache Kafka or RabbitMQ for messaging. Rabbit MQ vs. Kafka - Which one is a better message broker? You might find some articles across the web that conclude that Apache Kafka is better than RabbitMQ and few others that mention RabbitMQ to be more reliable than Kafka.
A Cloud Guru: Data Engineering
JULY 20, 2021
In this post, (based on my session from the recent ACG Community Summit) I’m going to lay out what I view as the four pillars of Azure, trends we’re seeing around these, where I think they’re heading, and how you might plan your cloud career around these areas. What are the pillars of Azure? Before […] The post Pillars of Azure: 4 trends to watch in your cloud career appeared first on A Cloud Guru.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Cloudera
JULY 21, 2021
In the build-up to this year’s Data Impact Awards, we’re looking back at last year’s winners. We are reflecting on their accomplishments, finding out about further developments, and giving you a taste of what it takes to get the judges’ attention. Last year’s awards saw OVO crowned as Data Champions. This is the category for Cloudera customers whose IT administration provides the agility business requires, without putting organizations at risk, and who are embracing a pattern of technology adopt
DataKitchen
JULY 22, 2021
Chris Bergh chats with author Randy Bean about his book, Fail Fast, Learn Faster: Lessons in Data-Driven Leadership in an Age of Disruption, Big Data & AI. The post A Chat with Randy Bean on His Book, Fail Fast, Learn Faster first appeared on DataKitchen.
ProjectPro
JULY 21, 2021
In this blog, explore a diverse list of interesting NLP projects ideas, from simple NLP projects for beginners to advanced NLP projects for professionals that will help master NLP skills. As per the Future of Jobs Report released by the World Economic Forum in October 2020, humans and machines will be spending an equal amount of time on current tasks in the companies, by 2025.
Teradata
JULY 20, 2021
Many Retailers & CPGs are missing huge opportunities to improve their margins & further enhance their customer experience due to broad brush data that lack insight. Read more.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Grouparoo
JULY 18, 2021
Update (January 2022) The Grouparoo community is continually working to improve the developer experience for Reverse ETL. Here's our guide to Getting Started with Grouparoo to lead you through installation, configuration, running, and deploying projects. Grouparoo's recommend way to configure the application is through UI Config. An important enhancement to the workflow is the addition of Models.
Preset
JULY 21, 2021
Why Preset is the best alternative to Chart.io. Learn how to avoid lock-in with Preset, built on top of Apache Superset.
ProjectPro
JULY 23, 2021
Machine Learning and Deep Learning have experienced unusual tours from bust to boom from the last decade. Simmering in research labs, these two verticals of artificial intelligence became a savior for many companies. As there is a famous saying, "the larger, the better." But when it comes to large data sets, determining insights from them through deep learning algorithms and mining them becomes tricky.
Teradata
JULY 18, 2021
Read more about Teradata's “Sleep Prediction” Hackathon, based on Apple Watch data, to capture different stages of sleep based on heart rate and activity count.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
RudderStack
JULY 21, 2021
Here’s an overview of the specific ways engineering teams support marketing from the data layer with RudderStack.
Rockset
JULY 19, 2021
Apache Druid is a distributed real-time analytics database commonly used with user activity streams, clickstream analytics, and Internet of things (IoT) device analytics. Druid is often helpful in use cases that prioritize real-time ingestion and fast queries. Druid’s list of features includes individually compressed and indexed columns, various stream ingestion connectors and time-based partitioning.
ProjectPro
JULY 19, 2021
Artificial Intelligence tools and technologies are moving at a rapid pace of innovation, so not to be surprised by the constant emergence of novel artificial intelligence and machine learning job roles like NLP Engineer , Computer Vision Engineer, Machine Learning Engineer, AI Software Engineer, AI Research Engineer, Artificial Intelligence Engineer , Machine Learning Scientist , Data Scientist , and many more to mention.
Scribd Technology
JULY 19, 2021
Delta Lake is integral to our data platform which is why we have invested heavily in delta-rs to support our non-JVM Delta Lake needs. This year I had the opportunity to share the progress of delta-rs at Data and AI Summit. Delta-rs was originally started by my colleague QP just over a year ago and it has now grown to now a multi-company project with numerous contributors, and downstream projects such as kafka-delta-ingest.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
ProjectPro
JULY 17, 2021
Want to become an AI Engineer? Check out this detailed AI Engineer salary guide to understand how much can you make as an AI engineer based on various factors- experience level, companies, and location. Artificial Intelligence (AI) market will be worth 190 Billion USD by 2025. As of June 2022, there are 18,380 open vacancies for AI Engineers in the United States, while India has 2,740 openings for the role of an AI Engineer.
Scribd Technology
JULY 20, 2021
Extracting metadata from our documents is an important part of our discovery and recommendation pipeline, but discerning useful and relevant details from text-heavy user-uploaded documents can be challenging. This is part 2 in a series of blog posts describing a multi-component machine learning system the Applied Research team built to extract metadata from our documents in order to enrich downstream discovery models.
Expert insights. Personalized for you.
145 
Let's personalize your content