Start Data Engineering
OCTOBER 5, 2021
1. Introduction 2. What is a staging area 3. The advantages of having a staging area 5. Conclusion 6. Further reading 1. Introduction Working with data pipelines, you might have noticed a staging area in most data pipelines. If you work in the data space and have questions like Why is there a staging area? Can’t we just load data into the destination tables?
Confluent
OCTOBER 6, 2021
Today, an organization’s strategic objective is to deliver innovations for a connected life and to improve the quality of life worldwide. With connected devices comes data, and with data comes […].
Cloudera
OCTOBER 7, 2021
October sees the launch of Partner Appreciation Month and during the next few weeks we will be sharing success stories, updates and interviews with our valued partners across the world. . We’re on a mission to make data and analytics easy and accessible, for everyone, and the hybrid data cloud is how we’ll get there. Today’s world is a hybrid world—there’s hybrid data, hybrid infrastructure, hybrid work—and leading businesses are embracing these changes, unafraid to transform their processes and
Data Engineering Podcast
OCTOBER 8, 2021
Summary The key to making data valuable to business users is the ability to calculate meaningful metrics and explore them along useful dimensions. Business intelligence tools have provided this capability for years, but they don’t offer a means of exposing those metrics to other systems. Metriql is an open source project that provides a headless BI system where you can define your metrics and share them with all of your other processes.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Start Data Engineering
OCTOBER 3, 2021
1. Introduction 2. Business requirements: dashboards and analytics 3. What is a data warehouse 4. OLTP vs OLAP based data warehouses 5. Conclusion 6. Further reading 7. References 1. Introduction If you are a student, analyst, engineer, or anyone in the data space, it’s important to understand what a data warehouse is. If you are wondering What is a data warehouse?
DataKitchen
OCTOBER 8, 2021
The post DataOps Lowers The Cost Of Asking Analytic Questions first appeared on DataKitchen.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
OCTOBER 5, 2021
Summary Transactions are a necessary feature for ensuring that a set of actions are all performed as a single unit of work. In streaming systems this is necessary to ensure that a set of messages or transformations are all executed together across different queues. In this episode Denis Rystsov explains how he added support for transactions to the Redpanda streaming engine.
Teradata
OCTOBER 4, 2021
An interdisciplinary team from Volkswagen, AWS and Teradata have created an intelligent solution that enables greater transparency and efficiency in car body construction. Find out more.
DataKitchen
OCTOBER 7, 2021
The post How Predictive and Prescriptive Analytics Improve the Call Center Experience first appeared on DataKitchen.
Netflix Tech
OCTOBER 7, 2021
Martin Tingley with Wenjing Zheng , Simon Ejdemyr , Stephanie Lane , and Colin McFarland This is the third post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making at Netflix) and Part 2 (What is an A/B Test?). Subsequent posts will go into more details on experimentation across Netflix, how Netflix has invested in infrastructure to support and scale experimentation, and the i
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Data Engineering Podcast
OCTOBER 2, 2021
Summary Aerospike is a database engine that is designed to provide millisecond response times for queries across terabytes or petabytes. In this episode Chief Strategy Officer, Lenley Hensarling, explains how the ability to process these large volumes of information in real-time allows businesses to unlock entirely new capabilities. He also discusses the technical implementation that allows for such extreme performance and how the data model contributes to the scalability of the system.
Cloudera
OCTOBER 8, 2021
Introduction. Apache Impala is a massively parallel in-memory SQL engine supported by Cloudera designed for Analytics and ad hoc queries against data stored in Apache Hive, Apache HBase and Apache Kudu tables. Supporting powerful queries and high levels of concurrency Impala can use significant amounts of cluster resources. In multi-tenant environments this can inadvertently impact adjacent services such as YARN, HBase, and even HDFS.
DataKitchen
OCTOBER 6, 2021
The post Why Enterprise AI Needs Human Intervention first appeared on DataKitchen.
Netflix Tech
OCTOBER 7, 2021
By Minal Mishra Quality of a client application is of paramount importance to global digital products, as it is the primary way customers interact with a brand. At Netflix, we have significant investments in ensuring new versions of our applications are well tested. However, Netflix is available for streaming on thousands of types of devices and it is powered by hundreds of micro-services which are deployed independently, making it extremely challenging to comprehensively test internally.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Pipeline Data Engineering
OCTOBER 8, 2021
In this interview series we’ll share some of the stories that Daniel and I get to watch unfold at Pipeline Academy. Check out what our graduates have to say about the course, how they’ve tackled its challenges and what they are doing now with their new data engineering superpowers. Peter: Michele, it's great to see you again. Thanks for taking the time to have a chat with me.
Cloudera
OCTOBER 5, 2021
If your organization is using multi-tenant big data clusters (and everyone should be), do you know the usage and cost efficiency of resources in the cluster by tenants? A chargeback or showback model allows IT to determine costs and resource usage by the actual analytic users in the multi-tenant cluster, instead of attributing those to the platform (“overhead’) or IT department.
DataKitchen
OCTOBER 5, 2021
A DataOps Engineer owns the assembly line that’s used to build a data and analytic product. Data operations (or data production) is a series of pipeline procedures that take raw data, progress through a series of processing and transformation steps, and output finished products in the form of dashboards, predictions, data warehouses or whatever the business requires.
ProjectPro
OCTOBER 8, 2021
Retail is one of the first industries that started leveraging the power of machine learning and artificial intelligence. There are machine learning projects for almost every retail use case - right from inventory management to customer satisfaction. Machine learning projects in retail directly convert into profits and increase an organization’s market share with better customer acquisition and satisfaction.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
AltexSoft
OCTOBER 8, 2021
To drive deeper business insights and greater revenues, organizations — whether they are big or small — need quality data. But more often than not data is scattered across a myriad of disparate platforms, databases, and file systems. What’s more, that data comes in different forms and its volumes keep growing rapidly every day — hence the name of Big Data.
Monte Carlo
OCTOBER 7, 2021
I first “joined” Monte Carlo exactly a year ago, as a data science intern. I met Lior , our co-founder, on Zoom in August of 2020. I had cast a volley of solicitous emails into my network — “(sort of) Stanford C.S. student looking to be (sort of) hired and avoid school for a while” — and one opportunity had come back from Oren and Glenn , former colleagues and now mentors of mine at GGV.
A Cloud Guru: Data Engineering
OCTOBER 6, 2021
In this post, we’ll discuss the difference between ETL vs ELT and when you might choose ETL or ELT. We’ll also include a flowchart to help walk you through the ETL vs ELT decision-making process. The difference between ETL vs ELT What’s the difference between ETL and ELT? The short answer is it’s all about […] The post ETL vs ELT flowchart: When to use each appeared first on A Cloud Guru.
ProjectPro
OCTOBER 8, 2021
Microsoft Power BI is the most used business intelligence tool according to peer ratings on Gartner. Companies like Adobe, Heathrow, PharmD have shown their trust in Power BI and continue to do so year after year. For 14 consecutive years, Power BI has bagged the first position in Magic Quadrant. As Power BI customers keep increasing, companies look for Power BI developers and analysts who can drive their business analytics and intelligence tasks.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Grouparoo
OCTOBER 6, 2021
Last month, we decided that we should all read a book and talk about it as a company. It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This was the first book I have read in this series and I liked the format. It is made up of 97 small vignettes that are 2-3 pages each. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, complianc
Teradata
OCTOBER 6, 2021
This how-to guide will help you connect Teradata Vantage with the AWS Data Exchange service. Read more for step-by-step instructions.
Big Data Tools
OCTOBER 5, 2021
In most countries, students start learning in September. As data engineers, let’s follow their lead and learn something new, too! I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of developments and highlight ideas from the wider community. If you think I missed something worthwhile, ping me on Twitter and suggest a topic, link, or anything else.
ProjectPro
OCTOBER 6, 2021
With 67 zones, 140 edge locations, over 90 services, and 940163 organizations using GCP across 200 countries - GCP is slowly garnering the attention of cloud users in the market. Flexera’s State of Cloud report highlighted that 41% of the survey respondents showed the most interest in using Google Cloud Platform for their future cloud computing projects.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Datakin
OCTOBER 5, 2021
Last week, Matt Turck and John Wu published the latest annual report on the state of data, the 2021 Machine Learning, AI and Data (MAD) Landscape. If you haven’t read it yet, we recommend it as a comprehensive snapshot of the intricate world of AI, machine learning, and data science & engineering. Our team enjoyed reading it. We represent several of the pixels on this chart (hey, cool!
Preset
OCTOBER 4, 2021
We discuss how to use dbt transformation (data build tool) to convert JSON data from GitHub into clean, tidy data for visualization.
Big Data Tools
OCTOBER 5, 2021
In most countries, students start learning in September. As data engineers, let’s follow their lead and learn something new, too! I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of developments and highlight ideas from the wider community. If you think I missed something worthwhile, ping me on Twitter and suggest a topic, link, or anything else.
ProjectPro
OCTOBER 6, 2021
This blog is a step-by-step guide for a beginner in NLP. If you are someone who wants to know what is the best way to learn NLP from scratch, then please go through our blog till the end. We assure you will build the confidence and gear up yourself to make a career transition into data science as an NLP Engineer. We will first begin with what are the essential subjects one must be aware of, to prepare them for diving into the world of Natural Language Processing.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Expert insights. Personalized for you.
130 
Let's personalize your content