Why Uber Engineering Switched from Postgres to MySQL
Uber Engineering
JULY 26, 2016
The early architecture of Uber consisted of a monolithic backend application written in Python that used Postgres for data persistence.
Engineering MySQL Python 
Airflow XCOM: The Ultimate Guide
Marc Lamberti
SEPTEMBER 22, 2023
With the PythonOperator, we can use the keyword return along with the value in the Python callable function to push the value we want to share into the database and create the corresponding XCom automatically. With the PythonOperator you can access it by passing the parameter ti to the Python callable function. In Airflow 1.10.x,
MySQL 
246
246 Join 37,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Trending Sources
Data Engineering Weekly #181
Data Engineering Weekly
JULY 21, 2024
Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!
Data Engineering 117
117 
Top 15 Software Engineer Projects 2023 [Source Code]
Knowledge Hut
OCTOBER 27, 2023
In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)
Maintain Your Data Engineers' Sanity By Embracing Automation
Data Engineering Podcast
JULY 10, 2022
Summary Building and maintaining reliable data assets is the prime directive for data engineers. In order to make this a tractable problem it is essential that engineers embrace automation at every opportunity. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
Data Engineering 100
100 
Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast
Data Engineering Podcast
JULY 17, 2022
Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder. why not a data science podcast?
Data Engineering 100
100 
Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt
Data Engineering Podcast
OCTOBER 30, 2022
It’s hard to have a conversation about data engineering or analysis without mentioning it. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.
Engineering 100
100 
Data News — Week 24.08
Christophe Blefari
FEBRUARY 23, 2024
This week I've participated to a round table about data and did a cool presentation about Engines. The idea was to depict the history of engines over the last 40 years and what leads to polars and DuckDB. Engines evolution (me) There are 3 points that have triggered discussion about the visualisation I done What about Arrow?
Data Lake 130
130 
Top Software Engineer Skills You Should Have in 2024
Knowledge Hut
DECEMBER 27, 2023
What will be the key skills for a software engineer in 2024? The top software developers stand out from the crowd thanks to their extensive technical skills as software engineers. Thus, to become a highly sought-after software developer, you must increase and update the technical skills required for software engineering.
Databricks, Snowflake and the future
Christophe Blefari
JUNE 21, 2024
One way to read data platforms When we look at platforms history what characterises evolution is the separation (or not) between the engine and the storage. A UX where you buy a single tool combining engine and storage, where all you have to do is flow data in, write SQL, and it's done. —with Databricks you buy an engine.
Metadata 147
147 
Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB
Data Engineering Podcast
OCTOBER 23, 2022
In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. Data teams are increasingly under pressure to deliver.
Database 100
100 
Discover And De-Clutter Your Unstructured Data With Aparavi
Data Engineering Podcast
JUNE 12, 2022
From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.
Python for Data Engineering
Ascend.io
SEPTEMBER 14, 2023
The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?
Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg
Data Engineering Podcast
NOVEMBER 6, 2022
Summary Despite the best efforts of data engineers, data is as messy as the real world. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs.
MongoDB 130
130 
Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet
Data Engineering Podcast
NOVEMBER 20, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Data Lake 130
130 
Automating dead code cleanup
Engineering at Meta
OCTOBER 24, 2023
We discussed how SCARF saves engineering time by identifying the correct order of tasks for cleaning up a product and how it can be blocked from automating the cleanup when there are intersystem dependencies. Dead code removal at scale SCARF uses CodemodService to create code change requests for engineers to review.
Coding 139
139 
DevOps Roadmap to Become a Successful DevOps Engineer
Knowledge Hut
FEBRUARY 6, 2024
What are the tasks of a DevOps Engineer? A Summary of day-to-day tasks carried out by a DevOps engineer - Design, build, test and deploy scalable, distributed systems from development through production Manage the code repository(such as Git, SVN, BitBucket, etc.) Who can become a DevOps Engineer?
Engineering 98
98 
Unlocking The Potential Of Streaming Data Applications Without The Operational Headache At Grainite
Data Engineering Podcast
MARCH 25, 2023
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Businesses that adapt well to change grow 3 times faster than the industry average. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code.
MySQL 130
130 
An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality
Data Engineering Podcast
AUGUST 28, 2022
Summary The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution. __init__ covers the Python language, its community, and the innovative ways it is being used.
Data Engineering 100
100 
Level Up Your Data Platform With Active Metadata
Data Engineering Podcast
JUNE 19, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Metadata 130
130 
Scaling Salt for Remote Execution to support LinkedIn Infra growth
LinkedIn Engineering
APRIL 18, 2023
At LinkedIn, site engineers like to automate operational tasks at various infrastructure layers to minimize manual interventions, which can scale well and be easy to operate. The Zen of python) by Tim Peters. Certain automations are performed via onDemand job executions. cfengine & puppet.
MySQL 103
103 
Build Data Products Without A Data Team Using AgileData
Data Engineering Podcast
NOVEMBER 13, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Building 130
130 
Data News — Week 24.30
Christophe Blefari
JULY 26, 2024
Joe is a great speaker, he wrote Fundamentals of Data Engineering , which is one of the bibles in data engineering and I can't wait to hear him at Forward Data. Querying 1TB on a laptop with Python dataframes — A benchmark on a 96 GB of memory laptop using DuckDB, DataFusion and Polars. Crazy what we can do nowadays.
MySQL 130
130 
Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas
Data Engineering Podcast
JUNE 19, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Metadata 130
130 
Most Popular Programming Certifications for 2024
Knowledge Hut
DECEMBER 26, 2023
Most Popular Programming Certifications C & C++ Certifications Oracle Certified Associate Java Programmer OCAJP Certified Associate in Python Programming (PCAP) MongoDB Certified Developer Associate Exam R Programming Certification Oracle MySQL Database Administration Training and Certification (CMDBA) CCA Spark and Hadoop Developer 1.
Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data
Data Engineering Podcast
SEPTEMBER 11, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Sifflet also offers a 2-week free trial.
Data Pipeline 130
130 
Run Your Applications Worldwide Without Worrying About The Database With Planetscale
Data Engineering Podcast
DECEMBER 11, 2022
Planetscale is a serverless option for your MySQL workloads that lets you focus on your applications without having to worry about managing the database or fight with differences between development and production. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests.
Database 100
100 
Making Sense Of The Technical And Organizational Considerations Of Data Contracts
Data Engineering Podcast
DECEMBER 18, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Struggling with broken pipelines? Stale dashboards?
Metadata 130
130 
Safely Test Your Applications And Analytics With Production Quality Data Using Tonic AI
Data Engineering Podcast
JANUARY 22, 2023
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. __init__ covers the Python language, its community, and the innovative ways it is being used.
PostgreSQL 130
130 
Securely Connect to LLMs and Other External Services from Snowpark
Snowflake
SEPTEMBER 7, 2023
Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Airbyte enables customers to continuously load data from external APIs and databases into Snowflake.
Make Database Performance Optimization A Playful Experience With OtterTune
Data Engineering Podcast
JUNE 22, 2021
We spend countless hours designing the data model, updating engine versions, and tuning performance. In this episode he explains how the system works, the challenge of scaling it to work across different database engines, and his hopes for the future of database systems. Write some Python scripts to automate it?
Database 100
100 
Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense
Data Engineering Podcast
OCTOBER 30, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Build Better Data Products By Creating Data, Not Consuming It
Data Engineering Podcast
NOVEMBER 6, 2022
Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day. For more information on Prefect, visit dataengineeringpodcast.com/prefect.
Building 130
130 
Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus
Data Engineering Podcast
AUGUST 6, 2022
In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.
Machine Learning 100
100 
Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI
Data Engineering Podcast
DECEMBER 29, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Struggling with broken pipelines? Stale dashboards?
Management 130
130 
Graph Databases In Production At Scale Using DGraph with Manish Jain - Episode 44
Data Engineering Podcast
AUGUST 19, 2018
Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. Luckily, there’s an easy way to get involved.
Database 100
100 
Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform
Data Engineering Podcast
JUNE 26, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Datasets 130
130 
Making The Total Cost Of Ownership For External Data Manageable With Crux
Data Engineering Podcast
JULY 17, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!
Data Management 130
130 
How To Bring Agile Practices To Your Data Projects
Data Engineering Podcast
OCTOBER 23, 2022
In this episode Shane Gibson shares practical advice and insights from his years of experience as a consultant and engineer working in data about how to adopt agile principles in your data work so that you can move faster and provide more value to the business, while building systems that are maintainable and adaptable.
Project 130
130 
A Look At The Data Systems Behind The Gameplay For League Of Legends
Data Engineering Podcast
NOVEMBER 20, 2022
Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.
Systems 130
130 
Top 15 Software Engineering Projects 2024 [Source Code]
Knowledge Hut
APRIL 24, 2024
In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)
Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda
Data Engineering Podcast
JULY 31, 2022
The Arkouda project is a Python interface built on top of the Chapel compiler to bring back those interactive speeds for exploratory analysis on horizontally scalable compute that parallelizes operations on large volumes of data. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
Data Analysis 100
100 
Prompt Engineering for Code Generation
Edureka
MARCH 4, 2024
Prompt engineering is the secret key in the world of technology, allowing the transformation of words into code by providing clear commands to AI. We’ll look at the fundamental principles of prompt engineering for code generation and synthesis, highlighting their key points. This is achieved by utilizing prompt engineering.
Coding 52
52 
Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams
Data Engineering Podcast
DECEMBER 28, 2022
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. or any other destination you choose. Upsolver : 
Let's personalize your content