François Nguyen

APRIL 29, 2021

This is my second article on “Why do you find Data so interesting after all these years ?” and my anwser is always “it is not about the subject, it is about the people”. A distinctive and instantly-recognizable style I was reading this article “ Is the Tableau Era Coming to an End? ” with no author and long before the conclusion I was telling to myself “looks like an article from Taylor Brownlow” It is clearly not easy with so many authors on the Data topic to have a dist

BI 130

BI SQL Data Data Science 130

Start Data Engineering

APRIL 26, 2021

Introduction 1. Using generators Using generator expression Using generator yield Mini batching Reading in batches from a database Pros & Cons 2. Using distributed frameworks Pros & Cons Conclusion Further reading References Introduction If you are Wondering how to write memory efficient data pipelines in python Working with a dataset that is too large to fit into memory Then this post is for you.

Data Pipeline 130

Data Pipeline Python Datasets Database 130

Uber Engineering

APRIL 12, 2021

Introduction. Uber’s many software systems require a high volume of changes every day. Because of our systems’ size and complexity, it is a significant challenge to implement these changes without unintended consequences, ultimately slowing down developer productivity. Flipr is a … The post Flipr: Making Changes Quickly and Safely at Scale appeared first on Uber Engineering Blog.

Engineering 142

Engineering Systems IT Architecture 142

Confluent

APRIL 19, 2021

I’m proud to announce the release of Apache Kafka 2.8.0 on behalf of the Apache Kafka® community. The 2.8.0 release contains many new features and improvements. This blog post highlights […].

Kafka 138

Kafka 138

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Superset is an open source option that has been gaining popularity due to its flexibility and extensible feature set. In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Cloudera

APRIL 21, 2021

Like most of our customers, Cloudera’s internal operations rely heavily on data. For more than a decade, Cloudera has built internal tools and data analysis primarily on a single production CDH cluster. This cluster runs workloads for every department – from real-time user interfaces for Support to providing recommendations in the Cloudera Data Platform (CDP) Upgrade Advisor to analyzing our business and closing our books.

Cloud 121

Cloud Professional Services Java Data Warehouse 121

Start Data Engineering

APRIL 8, 2021

Introduction Gathering requirements 0. Understand the current state of the data pipeline 1. Think like the end user 2. Know the why 3. End user interviews 4. Reduce the scope 5. End user walkthrough for proposed solution 6. Timelines & deliverables Deliver iteratively Conclusion Further reading References Introduction As data engineers, you will have to re-engineer legacy data pipelines.

Data Pipeline 130

Data Pipeline Engineering Data Engineering Data Engineer 130

Teradata

APRIL 28, 2021

Customers expect organizations to know them, provide relevant & personalized experiences, and be good stewards of their data. Yet many businesses still struggle with this. Why?

Data 91

Data 91

Confluent

APRIL 27, 2021

There is a class of applications that cannot afford to be unavailable—for example, external-facing entry points into your organization. Typically, anything your customers interact with directly cannot go down. As […].

Kafka 134

Kafka Data Ingestion Data 134

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. Sometimes, however, one of those transformations is actually a full-fledged machine learning project in its own right. In this episode Tal Galfsky explains how he and the team at Cherre tackled the problem of messy data for Addresses by building a natural language processing and entity resolution system that is served

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Cloudera

APRIL 23, 2021

Our latest Influential Women in Data session featured Brenda Le Sueur from Cambridge Assessments. Brenda has worked across many organisations and continents, but what has always been crucial to her is relationships – how we cultivate them, how we nurture them and how they, in turn, define us. I sat down with Brenda to ask her about her journey as a woman in tech and understand more about the impact of relationships on our career.

Technology 105

Technology Building Coding Project 105

DataKitchen

APRIL 28, 2021

Industry analysts who follow the data and analytics industry tell DataKitchen that they are receiving inquiries about “data fabrics” from enterprise clients on a near-daily basis. Forrester relates that out of 25,000 reports published by the firm last year, the report on data fabrics and DataOps ranked in the top ten for downloads in 2020. Gartner included data fabrics in their top ten trends for data and analytics in 2019.

Data Pipeline 98

Data Pipeline Data Data Analytics Architecture 98

Grouparoo

APRIL 29, 2021

I became very close with SQLite in the few weeks it took me to build out Grouparoo's SQLite plugin. Through that process I came to find that SQLite is not like the others. It has a handful of quirks, caveats, and gotchas when compared to other databases like MySQL and PostgreSQL. Here are seven of those quirks that I find most interesting: 1. SQLite is serverless SQLite doesn't require a separate process to run, as other databases do.

PostgreSQL 52

PostgreSQL MySQL Database SQL 52

Teradata

APRIL 4, 2021

The CFO’s broad remit & natural ownership of core financial data can provide the foundation for an enhanced role that leverages data analytics to enable new value opportunities.

Data Analytics 76

Data Analytics Data 76

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Confluent

APRIL 14, 2021

Updating a fundamental paradigm in your React app can be as easy as search and replace, or at other times, as difficult as convincing your entire frontend engineering to buy […].

Building 125

Building Engineering 125

Data Engineering Podcast

APRIL 12, 2021

Summary "Business as usual" is changing, with more companies investing in data as a first class concern. As a result, the data team is growing and introducing more specialized roles. In this episode Josh Benamram, CEO and co-founder of Databand, describes the motivations for these emerging roles, how these positions affect the team dynamics, and the types of visibility that they need into the data platform to do their jobs effectively.

Data Warehouse 100

Data Warehouse Data Pipeline BI Data Engineering 100

Cloudera

APRIL 22, 2021

This post was co-authored by two Cisco Employees as well: Karthik Krishna, Silesh Bijjahalli. Today’s enterprise data analytics teams are constantly looking to get the best out of their platforms. Storage plays one of the most important roles in the data platforms strategy, it provides the basis for all compute engines and applications to be built on top of it.

Pipeline-centric 101

Pipeline-centric Data Lake Hadoop Metadata 101

DataKitchen

APRIL 13, 2021

The post 10 Upcoming Data Science Platforms for Massive Disruption first appeared on DataKitchen.

Data Science 97

Data Science Data 97

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Monte Carlo

APRIL 29, 2021

Monte Carlo, the data reliability company, today announced a partnership with Snowflake , the Data Cloud company, to help data teams trust their data and accelerate the adoption of analytics in the Data Cloud. This combination can provide Snowflake customers with end-to-end Data Observability across their entire Snowflake Data Cloud, from ingestion to analytics.

Insurance 52

Insurance Business Intelligence Data Pipeline Cloud 52

Grouparoo

APRIL 27, 2021

Last week, I upgraded to a M1 Macbook Pro. I got it configured for development and 48 hours later, through a series of unfortunate events and hardware failure, I ended up with a second M1 Macbook Pro instead. The transition between computers wasn’t too bad thanks to Apple’s Migration Assistant. I ran into an interesting situation, though. About 90% of the migration worked as expected or better, but the other 10% presented some puzzling blockers.

Programming 52

Programming Database Coding IT 52

Confluent

APRIL 13, 2021

Morgan Stanley uses Apache Kafka® to publish market data to internal clients and to persist it for replay purposes. We started out using librdkafka’s C++ API, which maintains C++98 compatibility. […].

Kafka 124

Kafka IT Data 124

Data Engineering Podcast

APRIL 5, 2021

Summary One of the biggest obstacles to success in delivering data products is cross-team collaboration. Part of the problem is the difference in the information that each role requires to do their job and where they expect to find it. This introduces a barrier to communication that is difficult to overcome, particularly in teams that have not reached a significant level of maturity in their data journey.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project

Cloudera

APRIL 28, 2021

As we continue to work toward diversity, equality, and inclusion in every aspect of our company culture and beyond, we’ve learned so much from our employees’ unique perspectives on allyship. One such employee is Suzy Tonini, a Talent Researcher with a globe-trotting childhood. Growing up with parents who worked for the U.S. State Department, Suzy had the opportunity to hop from country to country with her family, experiencing a variety of cultures. .

Education 98

Education Building IT 98

DataKitchen

APRIL 6, 2021

The post DevOps and agile still hindered by enterprise silos, inertia first appeared on DataKitchen.

84

84

Retail Insight

APRIL 23, 2021

Social distancing and a life lived largely online have been the reality for over a year. But, as the world gradually emerges from lockdown, ha s the shape of retail really changed forever?

Retail 52

Retail Data 52

Grouparoo

APRIL 21, 2021

Two fun facts about SQLite : The initial release was more than 20 years ago! It is the most widely used database (and likely one of the most widely deployed pieces of software). And here are a few of my opinions on SQLite: It's super cool. We don't talk about it enough. It's actually really easy to use (which is likely why it's so widely used).

Bytes 52

Bytes Database PostgreSQL MySQL 52

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government

Confluent

APRIL 26, 2021

We’re excited to announce ksqlDB 0.17, a big release for 2021. This version adds support for managing the lifecycle of your queries from CI servers, a first-class timestamp data type, […].

Management 123

Management Data Process 123

Silectis

APRIL 19, 2021

Should you build your own tooling, take a “best of breed” approach, or buy a turnkey data engineering platform? We’ve got you covered. Data Engineering Platforms: Build, Best of Breed, or Buy? Every company wants to be data-driven. Modern organizations that thrive based on data have a common strength: a solid data engineering practice.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. You can read part 1 here and part 2 here , and watch part 3 here. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The first blog introduced a mock vehicle manufacturing company, The Electric Car Company (ECC) and focused on Data Collection.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Teradata

APRIL 13, 2021

As finance teams evolve into the providers of strategic insights, leveraging analytics will result in a new user base, new insights & reposition the CFO to a predictor of the future.

Finance 52

Finance 52

Advertisement

Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.

IT