François Nguyen

FEBRUARY 28, 2021

If there is one only book to read about lean manufacturing, this is the one. This is the kind of book you can read again and again and still learn something about your current context. It is also a book you can read whatever your industry, you will always find situations covered by this book. Today, we are going to apply these principles to the data pipelines. “The right process will deliver the right results” – Totoya way (section II) In the 14 Toyota way principles, you have

Data Pipeline 130

Data Pipeline Building Manufacturing BI 130

Start Data Engineering

FEBRUARY 28, 2021

Introduction Pre-requisites Setting up the data-ops pipeline Snowflake Local development environment dbt cloud Connect to Snowflake Link to github repository Setup deployment(release/prod) environment Setup CI PR -> CI -> merge cycle Schedule jobs Host data documentation Conclusion and next steps Further reading References Introduction With companies realizing the importance of having correct data, there has been a lot of attention on the data-ops side of things.

Cloud 130

Cloud Data 130

Team Data Science

FEBRUARY 23, 2021

Companies can now make data useful to elevate decision making and to optimise products and processes. But what organizational capabilities are necessary and how to get started? It's currently easy to acquire data strategically. First, consider that smartphones function like questionnaires that customers are frequently filling out in a passive or active manner [ , 1 ].

Business Intelligence 130

Business Intelligence Food Unstructured Data Relational Database 130

Confluent

FEBRUARY 8, 2021

One of the great things about using an Apache Kafka® based architecture is that it naturally decouples systems and allows you to use the best tool for the job. While […].

Kafka 145

Kafka Architecture Systems IT 145

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government

Data Engineering Podcast

FEBRUARY 22, 2021

Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. There are a number of managed platforms available, but the list of options for an open source system that supports a large variety of sources and destinations is still embarrasingly short. The team at Airbyte is adding a new entry to that list with the goal of making robust and easy to use data integration more accessible to teams who want or need to maintain full control of thei

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Teradata

FEBRUARY 9, 2021

Everyone knows that data is vital for success in retail. But without a clear data strategy, retailers often eat up resources fighting small-scale battles, whilst gradually losing the war.

Retail 112

Retail Data 112

Start Data Engineering

FEBRUARY 13, 2021

Why data exploration Apache Superset architecture Setup Prerequisites Seed data Using Apache Superset 1. Connecting to a data warehouse 2. Querying data in SQL Lab 3. Creating a chart 4. Creating a dashboard Pros and Cons Pros Cons Conclusion Why data exploration In most companies the end users of a data warehouse are analysts, data scientists and business people.

Data Warehouse 100

Data Warehouse SQL Architecture Data 100

Cloudera

FEBRUARY 23, 2021

Meet Kevin Smith, a Staff Customer Operations Engineer within the US Public Sector support team. He sums up his day-to-day by saying he works directly with clients on technical cases and provides support and guidance as they troubleshoot unexpected behavior. He also serves as a member of several project teams focusing on upgrade experiences, internal tools, product testing, training, and documentation.

Engineering 111

Engineering Education Technology Project 111

Confluent

FEBRUARY 22, 2021

Apache Kafka® is at the heart of the data transportation layer at Pinterest. The amount of data that runs through Kafka has constantly grown over the years. This growth sometimes […].

Kafka 145

Kafka Transportation Data 145

Data Engineering Podcast

FEBRUARY 15, 2021

Summary Every business aims to be data driven, but not all of them succeed in that effort. In order to be able to truly derive insights from the data that an organization collects, there are certain foundational capabilities that they need to have capacity for. In order to help more businesses build those foundations, Tarush Aggarwal created 5xData, offering collaborative workshops to assist in setting up the technical and organizational systems that are necessary to succeed.

Building 100

Building Data Warehouse BI Data Pipeline 100

Speaker: Christophe Louvion, Chief Product & Technology Officer of NRC Health and Tony Karrer, CTO at Aggregage

Christophe Louvion, Chief Product & Technology Officer of NRC Health, is here to take us through how he guided his company's recent experience of getting from concept to launch and sales of products within 90 days. In this exclusive webinar, Christophe will cover key aspects of his journey, including: LLM Development & Quick Wins 🤖 Understand how LLMs differ from traditional software, identifying opportunities for rapid development and deployment.

Designing

Netflix Tech

FEBRUARY 3, 2021

By Paul Bakker and Kavitha Srinivasan , Images by David Simmer , Edited by Greg Burrell Netflix has developed a Domain Graph Service (DGS) framework and it is now open source. The DGS framework simplifies the implementation of GraphQL, both for standalone and federated GraphQL services. Our framework is battle-hardened by our use at scale. By open-sourcing the project, we hope to contribute to the Java and GraphQL communities and learn from and collaborate with everyone who will be using the fra

Java 98

Java Architecture Coding Designing 98

François Nguyen

FEBRUARY 14, 2021

Let’s start with maybe the best definition you can find on Devops (credit to AWS ) : “DevOps is the combination of cultural philosophies , practices , and tools that increases an organization’s ability to deliver applications and services at high velocity : evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.

AWS 130

AWS Programming Process Building 130

Start Data Engineering

FEBRUARY 7, 2021

Introduction What is an SCD2 table and why use it? Application table Dimension table Setup Joining fact and SCD2 tables high_spenders user_items Educating end users Conclusion Further reading Introduction If you are using a data warehouse, you would have heard of fact and dimension tables. Simply put, fact tables are used to record a business event and dimension tables are used to record the attributes of business items(eg user, item tables in an e-commerce app).

Data Warehouse 100

Data Warehouse Education IT Data 100

Cloudera

FEBRUARY 8, 2021

The digital revolution is making a deep impact on the automotive industry, offering practically unlimited possibilities for more efficient, convenient, and safe driving and travel experiences in connected vehicles. This revolution is just beginning to accelerate – in fact, according to a recent Applied Market Research study, the global connected car market was valued at $63.03 billion in 2019, and is projected to reach $225.16 billion by 2027, registering a CAGR of 17.1% from 2020 to 2027.

Manufacturing 110

Manufacturing Machine Learning Data Ingestion Electronics 110

Speaker: Jamie Bernard

Success in product management goes beyond delivering great features - it’s about achieving measurable financial outcomes that resonate across the organization. By connecting your product’s journey with the company’s financial success, you’ll ensure that every feature, release, and innovation contributes to the bottom line, driving both customer satisfaction and business growth.

Management

Confluent

FEBRUARY 18, 2021

Soon, Apache Kafka® will no longer need ZooKeeper! With KIP-500, Kafka will include its own built-in consensus layer, removing the ZooKeeper dependency altogether. The next big milestone in this effort […].

Kafka 145

Kafka IT 145

Data Engineering Podcast

FEBRUARY 8, 2021

Summary With all of the tools and services available for building a data platform it can be difficult to separate the signal from the noise. One of the best ways to get a true understanding of how a technology works in practice is to hear from people who are running it in production. In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify.

Data Warehouse 100

Data Warehouse Building BI SQL 100

DataKitchen

FEBRUARY 1, 2021

Every DataOps initiative starts with a pilot project. How do you choose a project that matters to people? DataOps addresses a broad set of use cases because it applies workflow process automation to the end-to-end data-analytics lifecycle. DataOps reduces errors, shortens cycle time, eliminates unplanned work, increases innovation, improves teamwork, and more.

Project 98

Project Raw Data Consulting Data Science 98

François Nguyen

FEBRUARY 8, 2021

Or Robert C Martin, this uncle you should pay a visit more often. Where was I 20 years ago at that time when these 17 brillant folks were in a ski station for the Agile Manifesto ? I was part of a small team with great individuals and in fact we were an alternative to IT unable to deliver what we wanted. So we are going to do it ourselves. Without knowing it, we were totally in that agile mindset : valuing interactions, working software, our collaborations with the users and be able to change be

Architecture 130

Architecture Project IT Process 130

Advertisement

Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.

IT

Netflix Tech

FEBRUARY 25, 2021

By Cyril Concolato Introduction In previous blog posts, our colleagues at Netflix have explained how 4K video streams are optimized , how even legacy video streams are improved and more recently how new audio codecs can provide better aural experiences to our members. In all these cases, prior to being delivered through our content delivery network Open Connect , our award-winning TV shows, movies and documentaries like The Crown need to be packaged to enable crucial features for our members.

Technology 73

Technology Bytes Media Entertainment 73

Cloudera

FEBRUARY 18, 2021

Almost a year into the pandemic, the accelerated digital transformation has begun to feel less abrupt and more sustained. 2021 looks likely to be defined by a new phase: Thriving on digital transformation, rather than just surviving through it. . We’ve written about the changes forced on the traditionally risk-averse insurance industry by COVID-19. In 2021, with the crisis hopefully fading, insurance will have time to evaluate the changes made in 2020, assessing what worked and what didn’t

Insurance 106

Insurance Cloud Cloud Computing Data 106

Confluent

FEBRUARY 10, 2021

We are pleased to announce the release of Confluent Platform 6.1. With this release, we are further simplifying management tasks for Apache Kafka® operators and providing even higher availability for […].

Kafka 142

Kafka Management Cloud 142

Teradata

FEBRUARY 7, 2021

In order to survive, the auto industry needs to leverage 'digital threads’ that connect data from customers to dealers to products, & link R&D to production line & the aftermarket.

Data 69

Data 69

Advertisement

The complexity of financial data, the need for real-time insight, and the demand for user-friendly visualizations can seem daunting when it comes to analytics - but there is an easier way. With Logi Symphony, we aim to turn these challenges into opportunities. Our platform empowers you to seamlessly integrate advanced data analytics, generative AI, data visualization, and pixel-perfect reporting into your applications, transforming raw data into actionable insights.

Raw Data

Retail Insight

FEBRUARY 26, 2021

When I left school to start a professional football career, I understood very little about data – I did keep a note of the goals I scored, the assists I made and, most likely, the keepie-ups I could perform, but that was about it.

Retail 52

Retail Data IT 52

Grouparoo

FEBRUARY 25, 2021

Developers have been using the Grouparoo UI to set up automated data movement from their databases to Mailchimp, Marketo, Salesforce, and more. While having these integrations already written for them saved plenty of time, there was something they missed: their normal developer workflow. Grouparoo now supports declarative data models and integrations to continuously sync your data to all of your cloud-based tools.

Data 52

Data Database Cloud Engineering 52

A Cloud Guru: Data Engineering

FEBRUARY 23, 2021

How do you get started with an Azure database? As a database novice or someone new to Microsoft Azure, there are so many options it can be hard to know where to begin. Which is right for you as you get started on the path to becoming a data engineer? Let’s turn the question around […] The post Intro to databases on Azure: Basics for aspiring data engineers appeared first on A Cloud Guru.

Database 52

Database Data Engineer Data Engineering Engineering 52

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. And, check out the documentation here. . In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 104

Database Java Data Ingestion SQL 104

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Confluent

FEBRUARY 16, 2021

One of the most common relational database systems that connects to Apache Kafka® is Oracle, which often holds highly critical enterprise transaction workloads. While Oracle Database (DB) excels at many […].

Relational Database 124

Relational Database Kafka Database Systems 124

Teradata

FEBRUARY 3, 2021

Read how a principal data scientist at Teradata leveraged his cross-industry expertise to build an algorithm to help doctors better understand & fight COVID-19.

Algorithm 71

Algorithm Building Data 71

Monte Carlo

FEBRUARY 22, 2021

Companies spend upwards of $15 million an nually tackling data downtime , in other words, periods of time where data is missing, broken, or otherwise erroneous, and over 88 percent of U.S. bu sinesses have lost money as a result of data quality issues. Fortunately, there’s hope in the n ext frontier of data engineering: observability. Here’s how the data engineering team at Blinkist, a book-summarizing subscription service, increases cost savings, collaboration, and productivity with data observ

Data Pipeline 52

Data Pipeline Data Government Data Engineering 52

Rockset

FEBRUARY 18, 2021

Rockset released new numbers for the Star Schema Benchmark in April 2022. Learn how Rockset is 1.67 times faster than ClickHouse and 1.12 times faster than Druid in the latest performance blog post. Real-time analytics is all about deriving insights and taking actions as soon as data is produced. When broken down into its core requirements, real-time analytics means two things: access to fresh data and fast responses to queries.

Analytics Application 52

Analytics Application SQL Database Structured Data 52

Speaker: Conrado Morlan

In this engaging and witty talk, industry expert Conrado Morlan will explore how artificial intelligence can transform the daily tasks of product managers into streamlined, efficient processes. Using the lens of a superhero narrative, he’ll uncover how AI can be the ultimate sidekick, aiding in data management and reporting, enhancing productivity, and boosting innovation.

Management