Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster. The goal is to touch on the common data engineering challenges and using promising new technologies, tools or frameworks, which most of them I wrote about in Business Intelligence

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

François Nguyen

MARCH 7, 2021

Just an illustration – not the truth and we will pivot if it does not work. I discovered Zhamak Dehghani’s first article about Data Mesh in August 2020. Thanks to Youtube, you have the live illustration in this video with even more context and explanations. And then, you have this second video that is an introduction to her second article (december 2020).

Government 130

Government Data Governance Data Metadata 130

Confluent

MARCH 9, 2021

Real-time analytics has become the need of the hour for modern internet companies. The ability to derive internal insights around business metrics, user growth and adoption as well as security […].

Kafka 144

Kafka Architecture 144

Data Engineering Podcast

MARCH 8, 2021

Summary A majority of the time spent in data engineering is copying data between systems to make the information available for different purposes. This introduces challenges such as keeping information synchronized, managing schema evolution, building transformations to match the expectations of the destination systems. H.O. Maycotte was faced with these same challenges but at a massive scale, leading him to question if there is a better way.

IT 100

IT Data Warehouse MongoDB Kafka 100

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster. The goal is to touch on the common data engineering challenges and using promising new technologies, tools or frameworks, which most of them I wrote about in Business Intelligence

Data Engineer 130

Data Engineer Data Engineering Engineering Project 130

Netflix Tech

MARCH 8, 2021

By Michelle Brenner Netflix is poised to become the world’s most prolific producer of visual effects and original animated content. To meet that demand, we need to attract the world’s best artistic talent. Artists like to work at places where they can create groundbreaking entertainment instead of worrying about getting access to the software or source files they need.

Entertainment 122

Entertainment Java AWS Accessible 122

Cloudera

MARCH 11, 2021

Data flows are an integral part of every modern enterprise. No matter whether they move data from one operational system to another to power a business process or fuel central data warehouses with the latest data for near-real-time reporting, life without them would be full of manual, tedious and error-prone data modification and copying tasks. At Cloudera, we’re helping our customers implement data flows on-premises and in the public cloud using Apache NiFi , a core component of Cloudera DataFl

Cloud 119

Cloud Data Data Warehouse Data Integration 119

Teradata

MARCH 11, 2021

Getting your Cloud data architecture right starts with understanding which data products you need, the roles they perform, & the functional & non-functional characteristics that those roles demand.

Cloud 110

Cloud Systems Data Architecture Architecture 110

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. We’re very excited to bring you ConsoleMe (pronounced: kuhn-soul-mee ), and its CLI utility, Weep (pun intended)! If you missed the talk, check it out here.

AWS 102

AWS Accessible Accessibility Cloud 102

Confluent

MARCH 11, 2021

At Funding Circle, we rely heavily on Kafka as the main piece of infrastructure to enable our event-driven-based microservices architecture. There are numerous organizational benefits of microservices, however a key […].

Kafka 83

Kafka Architecture 83

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Cloudera

MARCH 9, 2021

Governance and the sustainable handling of data is a critical success factor in virtually all organizations. While Cloudera Data Platform (CDP) already supports the entire data lifecycle from ‘Edge to AI’, we at Cloudera are fully aware that enterprises have more systems outside of CDP. It is crucial to avoid that CDP becomes the next silo in your IT landscape.

Data Governance 95

Data Governance Government Metadata Datasets 95

Teradata

MARCH 9, 2021

All companies want a golden data analytics platform. But instead of looking at the real properties of the platform, they are often mislead by its shine & look. Find out more.

Data Analytics 52

Data Analytics IT Data 52

Netflix Tech

MARCH 12, 2021

Written by Anton Margoline , Avinash Dathathri , Devang Shah and Murthy Parthasarathi. Credit to Netflix Studio’s Product, Design, Content Hub Engineering teams along with all of the supporting partner and platform teams. In this post, we will share a behind-the-scenes look at how Netflix delivers technology and infrastructure to help production crews create and exchange media during production and post production stages.

Media 71

Media Cloud Management Metadata 71

DataKitchen

MARCH 11, 2021

Gartner identified XOps (DataOps, ModelOps, DevOps) as one of the top trends in data and analytics for 2021. Below we provide additional suggestions for further reading based on Gartner’s recommendations. What is XOps? . Gartner: “The multiplication of Ops disciplines stemming out of DevOps best practices has caused significant confusion in the marketplace.

Machine Learning 52

Machine Learning Utilities Data Data Analytics 52

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Cloudera

MARCH 8, 2021

Cloudera is happy to be an official supporter of International Women’s Day 2021. We at Cloudera believe in the undeniable power of data to build a more equitable future, and we are humbled to be building the products that make it possible for data to change the world for the better. . The theme of this year’s IWD is #ChooseToChallenge. As w e celebrate the social, economic, cultural, and political achievements of women, we’re building a foundation for our future young women, raising awareness ab

Data Science 74

Data Science Banking Education Building 74

Teradata

MARCH 11, 2021

Getting your Cloud data architecture right starts with understanding which data products you need, the roles they perform, & the functional & non-functional characteristics that those roles demand.

Analytics Architecture 52

Analytics Architecture Architecture Cloud Data Architecture 52

Grouparoo

MARCH 9, 2021

Rate limiting is the method by which an API limits the calls for its use. When creating a data sync implementation with an API, it's important to adapt the approach that the remote system takes. Whether stated or not, all systems have a rate limit. Even if not addressed explicitly, there is still some finite number of parallel connections that a set of servers can handle.

Systems 52

Systems Coding Accessibility Accessible 52

Zalando Engineering

MARCH 10, 2021

In 2015, we wanted to improve how we delivered features to customers and move away from a monolithic shop system. Project Mosaic and its microservices approach for the frontend were vital to support this transition. Mosaic enabled a relatively large number of teams to work on the main Zalando website independently and without performance compromises.

Architecture 52

Architecture Coding Engineering Designing 52

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Cloudera

MARCH 11, 2021

Karen Ji, is Cloudera’s Senior Manager Customer operations ensuring the success of our global customers, with a regional focus on China and Korea. Multi-tasking is my superpower. Karen joined Cloudera as a Solutions Engineer before switching to lead customer support. On a daily basis Karen collaborates across the business with different functions, but works extremely closely with the field, sales and professional services teams to ensure that customers have the support and insights they need to

Management 73

Management Recruitment Professional Services Building 73

Silectis

MARCH 9, 2021

The importance of data engineering is on the rise, with organizations increasingly investing in talent and infrastructure. Here at Silectis, we are in the fortunate position of working with a wide range of enterprises across multiple industries. I caught up with a few members of the team to take note of some of the data engineering trends we anticipate seeing more of this year and beyond. 1.

Data Engineer 52

Data Engineer Data Engineering Engineering ETL Tools 52

Ripple Engineering

MARCH 9, 2021

Welcome back to the second post of this Inclusive Language blog series! Previously, we contextualized the importance of eliminating terms with problematic and racist origins from our codebase, such as “master” and “slave”, or “blacklist” and “whitelist” We then suggested changing them with equally clear and more agreeable words such as “primary” and “secondary”, “denylist” and “allowlist

Engineering 52

Engineering Python Project Coding 52

Teradata

MARCH 7, 2021

Vantage scales in-database R/Python models on 70M clients. The customer analytics are transforming Bradesco to become the bank of the future, scaling insights and accelerating time-to-value.

Banking 52

Banking Python Database 52

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

DataKitchen

MARCH 8, 2021

The post 6 security risks in software development and how to address them first appeared on DataKitchen.

52

52

Rockset

MARCH 12, 2021

“Two of the most important things as a CEO of a company are to make sure you have money in the bank and recruit amazing people.” - Venkat Venkataramani, CEO and Co-Founder of Rockset We hosted a Clubhouse event with VPs of Engineering from Gusto and Robinhood, Nimrod Hoofien and Adam Wolff, on their tips for recruiting top engineering talent in startups.

Recruitment 40

Recruitment Engineering Banking Coding 40

Monte Carlo

MARCH 9, 2021

Today, I am proud to announce the formation of Monte Carlo’s Chief Data Officer (CDO) advisory board. The advisory board was launched to help Monte Carlo and the emerging data observability market better serve customers on their journeys to data trust, advise their product roadmap, and pioneer the data observability category. This announcement comes just weeks after our $25M Series B funding round this February, led by Redpoint Ventures, backers of Snowflake and Looker, and GGV Capital, in

Insurance 40

Insurance Data Governance Retail Government 40

Teradata

MARCH 7, 2021

The future of payments is rapidly evolving toward seamless omni-channel customer journeys and ultimately, payments becoming invisible. Find out more.

52

52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

DataKitchen

MARCH 9, 2021

The post Should You Build or Buy a DataOps Solution? first appeared on DataKitchen.

Building 52

Building 52

Preset

MARCH 10, 2021

Superset can integrate with almost any SQL speaking database because of SQLAlchemy, Python DB-API 2, and some minimal custom logic.

Database 40

Database Building SQL Python 40

Maxime Beauchemin

MARCH 8, 2021

While “software is [still actively] eating the world” , it’s also clear that open source is taking over software. Simply put, open source is a superior approach at building and distributing software because it provides important guaranties around how software can be discovered, tried, operated, collaborated on and packaged. For those reasons, it is not surprising that it has taken over most of the modern data stack: infrastructure, databases, orchestration, data processing, AI/ML and beyond.

Business Intelligence 130

Business Intelligence BI Database-centric Google Cloud 130