Christophe Blefari

FEBRUARY 23, 2024

My ideas these days ( credits ) Hey, fresh Data News edition. This week I've participated to a round table about data and did a cool presentation about Engines. — Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Is it DataFrames or SQL?

Data Lake 130

Data Lake PostgreSQL MongoDB MySQL 130

Marc Lamberti

SEPTEMBER 22, 2023

Wondering how to share data between tasks? At the end of this tutorial, you will have a solid knowledge of XComs and be able to share data between your tasks efficiently. One solution could be to store the accuracies in a database and fetch them back in the task choosing_model with an SQL request. Let’s get started!

MySQL 246

MySQL Data Pipeline Database Python 246

Christophe Blefari

JULY 26, 2024

Tallinn ( credits ) Dear members, it's Summer Data News, the only news you can consume by the pool, the beach or at the office—if you're not lucky. Joe is a great speaker, he wrote Fundamentals of Data Engineering , which is one of the bibles in data engineering and I can't wait to hear him at Forward Data.

MySQL 130

MySQL Data PostgreSQL Datasets 130

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Christophe Blefari

APRIL 19, 2024

easy ( credits ) Hey, new Friday, new Data News. Structured generative AI — Oren explains how you can constraint generative algorithms to produce structured outputs (like JSON or SQL—seen as an AST). This week, I feel like the selection is smaller than usual, so enjoy the links. But don't worry it gonna be out soon.

MySQL 130

MySQL Data Datasets SQL 130

Hevo

SEPTEMBER 24, 2023

MySQL has remained the most popularly used open-source relational database for many years and continues to maintain its dominant position in the industry. Migrating data from PostgreSQL on Google Cloud SQL to MySQL […] Migrating data from PostgreSQL on Google Cloud SQL to MySQL […]

Google Cloud 52

Google Cloud MySQL PostgreSQL SQL 52

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models.

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Christophe Blefari

JUNE 21, 2024

This year, the Snowflake Summit was held in San Francisco from June 2 to 5, while the Databricks Data+AI Summit took place 5 days later, from June 10 to 13, also in San Francisco. Using a quick semantic analysis, "The" means both want to be THE platform you need when you're doing data.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Monte Carlo

MAY 9, 2023

Judging by the conversations happening across the data engineering community, you’d think every team was in the midst of adopting an advanced data mesh architecture or weaving an industry-leading data fabric. Or that these innovative leaders only permit the latest and greatest tools and platforms to populate their data stacks.

MySQL 52

MySQL Database SQL Relational Database 52

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone. Can you describe what Tonic is and the story behind it?

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. Data pipelines power foundational parts of LinkedIn's infrastructure, including replication between data centers.

Data Pipeline 112

Data Pipeline Kafka SQL MySQL 112

Hevo

MAY 10, 2024

Organizations often manage operational data using open-source databases like MySQL, frequently deployed on local machines. To enhance data management and security, many organizations prefer deploying these databases on cloud providers like AWS, Azure, or Google Cloud Platform (GCP).

MySQL 52

MySQL Google Cloud AWS SQL 52

Data Engineering Podcast

NOVEMBER 20, 2022

Sifflet is a platform that brings your entire data stack into focus to improve the reliability of your data assets and empower collaboration across your teams. In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures.

Data Lake 130

Data Lake Data Ingestion MongoDB Google Cloud 130

Towards Data Science

MARCH 30, 2023

Back To Basics | SQL fundamentals for beginners Image by author, created on canva In order to understand the enterprise data; you have to query it a lot. Working with unfamiliar piles of data is often daunting and it’s always a good practice to take some time to explore and understand the data itself. What’s the need?

SQL 79

SQL MySQL Retail Data Analysis 79

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Building 130

Building IT Metadata MongoDB 130

Hevo

AUGUST 17, 2023

Are you looking to migrate your data from MongoDB Atlas to MySQL? Migrating data from MongoDB Atlas to MySQL can be a complex process, especially when handling large datasets and different database structures. However, moving data from MongoDB Atlas to MySQL can help you leverage SQL querying […]

MongoDB 52

MongoDB MySQL Datasets SQL 52

Meltano

OCTOBER 19, 2022

PostgreSQL and MySQL are among the most popular open-source relational database management systems (RDMS) worldwide. Both RDMS enable businesses to organize and interlink large amounts of data, allowing for effective data management. For all of their similarities, PostgreSQL and MySQL differ from one another in many ways.

PostgreSQL 52

PostgreSQL MySQL Database SQL 52

Data Engineering Podcast

JUNE 12, 2022

Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information.

Unstructured Data 130

Unstructured Data MongoDB MySQL Scala 130

Knowledge Hut

DECEMBER 21, 2023

Swiftly understanding the information is important in today's data-driven world. When managing massive amounts of data, having the right tools is vital. That is why we have compiled a MySQL tools list to consider in 2024. These advances help you improve your process and easily extract useful insights from your data.

MySQL 52

MySQL PostgreSQL Programming Language SQL 52

Netflix Tech

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events.

MySQL 88

MySQL PostgreSQL Database Transportation 88

Data Engineering Podcast

DECEMBER 28, 2022

Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of his time thinking and talking about the details of product thinking and its application to data.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB MySQL Scala 130

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. It was interesting to learn about some of the custom data types and performance optimizations that are included. Integrating data across the enterprise has been around for decades – so have the techniques to do it.

Data Warehouse 100

Data Warehouse MySQL Hadoop Data Lake 100

Data Engineering Podcast

OCTOBER 23, 2022

In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. Enter Metaplane, the industry’s only self-serve data observability tool.

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

DECEMBER 18, 2022

Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. In order to reduce the potential for broken pipelines some teams have started to adopt the idea of data contracts.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Analytics Vidhya

MARCH 10, 2023

Introduction DataHour sessions are an excellent opportunity for aspiring individuals looking to launch a career in the data-tech industry, including students and freshers. In this blog post, we […] The post Explore the World of Data-Tech with DataHour appeared first on Analytics Vidhya.

Data Science 241

Data Science Data MySQL Machine Learning 241

Netflix Tech

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events.

MySQL 83

MySQL PostgreSQL Database Transportation 83

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

MAY 13, 2021

Summary There is a lot of attention on the database market and cloud data warehouses. While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. Firebolt is the fastest cloud data warehouse. Visit dataengineeringpodcast.com/firebolt to get started.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Data Engineering Podcast

DECEMBER 4, 2022

Summary The term "real-time data" brings with it a combination of excitement, uncertainty, and skepticism. In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Knowledge Hut

APRIL 24, 2024

To understand SQL, you must first understand DBMS (database management systems) and databases in general. Data refers to a set of facts regarding a specific object. Whereas, a database refers to a set of small data units organized in a logical order. It is used to store, manipulate, and retrieve data from databases.

SQL 52

SQL MySQL Database Relational Database 52

Data Engineering Podcast

OCTOBER 23, 2022

Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. Atlan is the metadata hub for your data ecosystem. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Project 130

Project Metadata MongoDB MySQL 130

Data Engineering Podcast

MARCH 25, 2023

Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. As your business adapts, so should your data. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code.

MySQL 130

MySQL Python Architecture Machine Learning 130