Uber Engineering
JULY 22, 2021
Introduction. As Uber’s business grew, we scaled our Apache Hadoop (referred to as ‘Hadoop’ in this article) deployment to 21000+ hosts in 5 years, to support the various analytical and machine learning use cases. We built a team with varied … The post Containerizing Apache Hadoop Infrastructure at Uber appeared first on Uber Engineering Blog.
Confluent
JULY 20, 2021
We’re pleased to announce ksqlDB 0.19.0! This release includes a new NULLIF function and a major upgrade to ksqlDB’s data modeling capabilities—foreign-key joins. We’re excited to share this highly requested […].
Start Data Engineering
JULY 21, 2021
Introduction Using Native Python Using Pydantic Pydantic Caveats Conclusion Further reading References Introduction Data type issues are one of the biggest concerns when processing data in python. If you are wondering how to Make sure that a column is of a specific data type ( e.g.
Cloudera
JULY 22, 2021
My name is Shanmukha Kota and I am a recent graduate from University at Buffalo. I interned with Cloudera last summer and joined Cloudera as a software engineer a couple of weeks ago and this is my first experience with CDP and CDP Operational Database. For a new hire college graduate in the industry with only academic experience with HBase, I can only say it is very simple and easy to set up and work with CDP Operational Database.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Data Engineering Podcast
JULY 22, 2021
Summary Collecting and cleaning data is only useful if someone can make sense of it afterward. The latest evolution in the data ecosystem is the introduction of a dedicated metrics layer to help address the challenge of adding context and semantics to raw information. In this episode Nick Handel shares the story behind Transform, a new platform that provides a managed metrics layer for your data platform.
DataKitchen
JULY 21, 2021
The post Does Your Organization Need a Chief Data Officer? Probably first appeared on DataKitchen.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Cloudera
JULY 19, 2021
Meet Veda Kadam. She’s relatively new to the Cloudera family. She started her journey here in June of 2020 when she joined our first ever fully virtual intern program. Now she’s a full time employee working as a Software Engineer on our Data In Motion team. From an early age, Veda knew she wanted to work in the technology industry. Her father worked in pharmaceuticals and her mother worked in accounting.
Data Engineering Podcast
JULY 19, 2021
Summary Data quality is a concern that has been gaining attention alongside the rising importance of analytics for business success. Many solutions rely on hand-coded rules for catching known bugs, or statistical analysis of records to detect anomalies retroactively. While those are useful tools, it is far better to prevent data errors before they become an outsized issue.
Teradata
JULY 22, 2021
Learn about the techniques and frameworks needed to build a more resilient, cost-effective, and efficient data & analytic decisioning support capability for the post-pandemic supply chain.
DataKitchen
JULY 22, 2021
Learn about four data architectures patterns for agility - DataOps, Data Fabric, Data Mesh & Functional Data Engineering - & an example combining all four. The post DataOps: The Foundation for Your Agile Data Architecture first appeared on DataKitchen.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Cloudera
JULY 21, 2021
In the build-up to this year’s Data Impact Awards, we’re looking back at last year’s winners. We are reflecting on their accomplishments, finding out about further developments, and giving you a taste of what it takes to get the judges’ attention. Last year’s awards saw OVO crowned as Data Champions. This is the category for Cloudera customers whose IT administration provides the agility business requires, without putting organizations at risk, and who are embracing a pattern of technology adopt
ProjectPro
JULY 21, 2021
As a big data architect or a big data developer, when working with Microservices-based systems, you might often end up in a dilemma whether to use Apache Kafka or RabbitMQ for messaging. Rabbit MQ vs. Kafka - Which one is a better message broker? You might find some articles across the web that conclude that Apache Kafka is better than RabbitMQ and few others that mention RabbitMQ to be more reliable than Kafka.
Rockset
JULY 22, 2021
Social gaming is on the rise. During COVID-19, 29% of consumers reported playing games on a weekly basis and the goal for many players was to connect with friends and family ( Deloitte: Games and Streaming Services Fight it Out During Pandemic from VentureBeat ). One of the challenges that gaming companies face is rapidly building features that can strengthen network effects.
DataKitchen
JULY 22, 2021
Chris Bergh chats with author Randy Bean about his book, Fail Fast, Learn Faster: Lessons in Data-Driven Leadership in an Age of Disruption, Big Data & AI. The post A Chat with Randy Bean on His Book, Fail Fast, Learn Faster first appeared on DataKitchen.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
A Cloud Guru: Data Engineering
JULY 20, 2021
In this post, (based on my session from the recent ACG Community Summit) I’m going to lay out what I view as the four pillars of Azure, trends we’re seeing around these, where I think they’re heading, and how you might plan your cloud career around these areas. What are the pillars of Azure? Before […] The post Pillars of Azure: 4 trends to watch in your cloud career appeared first on A Cloud Guru.
ProjectPro
JULY 21, 2021
In this blog, explore a diverse list of interesting NLP projects ideas, from simple NLP projects for beginners to advanced NLP projects for professionals that will help master NLP skills. As per the Future of Jobs Report released by the World Economic Forum in October 2020, humans and machines will be spending an equal amount of time on current tasks in the companies, by 2025.
Teradata
JULY 20, 2021
Many Retailers & CPGs are missing huge opportunities to improve their margins & further enhance their customer experience due to broad brush data that lack insight. Read more.
Grouparoo
JULY 18, 2021
Update (January 2022) The Grouparoo community is continually working to improve the developer experience for Reverse ETL. Here's our guide to Getting Started with Grouparoo to lead you through installation, configuration, running, and deploying projects. Grouparoo's recommend way to configure the application is through UI Config. An important enhancement to the workflow is the addition of Models.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Preset
JULY 21, 2021
Why Preset is the best alternative to Chart.io. Learn how to avoid lock-in with Preset, built on top of Apache Superset.
ProjectPro
JULY 23, 2021
Machine Learning and Deep Learning have experienced unusual tours from bust to boom from the last decade. Simmering in research labs, these two verticals of artificial intelligence became a savior for many companies. As there is a famous saying, "the larger, the better." But when it comes to large data sets, determining insights from them through deep learning algorithms and mining them becomes tricky.
Teradata
JULY 18, 2021
Read more about Teradata's “Sleep Prediction” Hackathon, based on Apple Watch data, to capture different stages of sleep based on heart rate and activity count.
RudderStack
JULY 21, 2021
Here’s an overview of the specific ways engineering teams support marketing from the data layer with RudderStack.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Scribd Technology
JULY 19, 2021
Delta Lake is integral to our data platform which is why we have invested heavily in delta-rs to support our non-JVM Delta Lake needs. This year I had the opportunity to share the progress of delta-rs at Data and AI Summit. Delta-rs was originally started by my colleague QP just over a year ago and it has now grown to now a multi-company project with numerous contributors, and downstream projects such as kafka-delta-ingest.
ProjectPro
JULY 19, 2021
Artificial Intelligence tools and technologies are moving at a rapid pace of innovation, so not to be surprised by the constant emergence of novel artificial intelligence and machine learning job roles like NLP Engineer , Computer Vision Engineer, Machine Learning Engineer, AI Software Engineer, AI Research Engineer, Artificial Intelligence Engineer , Machine Learning Scientist , Data Scientist , and many more to mention.
Rockset
JULY 19, 2021
Apache Druid is a distributed real-time analytics database commonly used with user activity streams, clickstream analytics, and Internet of things (IoT) device analytics. Druid is often helpful in use cases that prioritize real-time ingestion and fast queries. Druid’s list of features includes individually compressed and indexed columns, various stream ingestion connectors and time-based partitioning.
Cloudera
JULY 20, 2021
Most businesses, whether you are in Retail, Manufacturing, Specialty Chemicals, Telecommunications, consider a 10% market capitalization increase from 2020 to 2021 outstanding. But what would you say to your shareholders when they found out your competitors’ market capitalization grew 35%? A recent McKinsey report dove into the divergence between retail’s laggards and winners and found if there is one message in the retail sector’s stock market performance since the pandemic’s start, it is
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Scribd Technology
JULY 20, 2021
Extracting metadata from our documents is an important part of our discovery and recommendation pipeline, but discerning useful and relevant details from text-heavy user-uploaded documents can be challenging. This is part 2 in a series of blog posts describing a multi-component machine learning system the Applied Research team built to extract metadata from our documents in order to enrich downstream discovery models.
ProjectPro
JULY 17, 2021
Want to become an AI Engineer? Check out this detailed AI Engineer salary guide to understand how much can you make as an AI engineer based on various factors- experience level, companies, and location. Artificial Intelligence (AI) market will be worth 190 Billion USD by 2025. As of June 2022, there are 18,380 open vacancies for AI Engineers in the United States, while India has 2,740 openings for the role of an AI Engineer.
Expert insights. Personalized for you.
145 
Let's personalize your content