Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python 96

Python Utilities Coding Data Engineering 96

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python 218

Python Project Management Data Engineering 218

KDnuggets

NOVEMBER 5, 2024

Enhance your data science portfolio with these seven engaging Python projects that demonstrate essential programming and software engineering skills.

Portfolio 112

Portfolio Data Science Python Project 112

Confessions of a Data Guy

JUNE 11, 2024

One of the things I love about Python is its flexibility and huge community, a community that puts out a never-ending stream of useful packages for the average Software Engineer.

Python 100

Python Building Software Engineer Software Engineering 100

Start Data Engineering

MAY 28, 2024

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline 219

Data Pipeline Python Building Data 219

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

KDnuggets

MARCH 15, 2022

This article discusses and demonstrates how to quickly engineer some common date features using Python.

Python 152

Python Engineering 152

Confessions of a Data Guy

JUNE 3, 2024

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python 130

Python Programming IT Data 130

Engineering at Meta

FEBRUARY 12, 2024

By now you’re already aware that Python 3.12 Learn how and why they built these new features for Python and how they worked with and engaged with the Python community. The post Meta loves Python appeared first on Engineering at Meta. has been released. Send us feedback on Instagram , Threads , or X.

Python 97

Python Software Engineer Software Engineering Engineering 97

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Knowledge Hut

SEPTEMBER 24, 2024

If you're considering pursuing a career as a software engineer, it's an exciting field with lots of potential for growth and opportunity. But becoming a software engineer requires more than having the right degree and technical skills. Who is a Software Engineer? So, what happened?

Software Engineer 98

Software Engineer Software Engineering Engineering Amazon Web Services 98

Lyft Engineering

MARCH 6, 2024

DEED In this post, we’ll cover how Lyft upgrades Python at scale — 1500+ repos spanning 150+ teams — and the latest iteration of the tools and strategy we’ve built to optimize both the overall time to upgrade and the work required from our engineers. Python, How Do I L̶o̶v̶e̶ Use Thee? and hope you find it useful!

Python 73

Python Metadata Datasets Coding 73

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Engineering at Meta

NOVEMBER 21, 2023

Python plays a big part at Meta. Meta even made contributions to Python 3.12 , the latest version of Python. On this episode of the Meta Tech Podcast , Meta engineer Pascal Hartig ( @passy ) is joined by Amethyst Reese, a production engineer at Meta, to discuss all things Python at Meta.

Python 96

Python Engineering Systems IT 96

Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Engineering at Meta

OCTOBER 5, 2023

Python 3.12 It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 This week’s release of Python 3.12

Python 110

Python Coding Programming Language Programming 110

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessibility 234

Accessibility Accessible Python Database 234

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. The blog Prompt Engineering for a Better SQL Code Generation With LLMs is a pretty good guide on applying prompt engineering to improve productivity.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Confessions of a Data Guy

AUGUST 1, 2024

The post Daft: Distributed Dataframes with Python. appeared first on Confessions of a Data Guy.

Python 100

Python Data Big Data Data Engineering 100

Edureka

AUGUST 19, 2024

I’ll guide you step-by-step on your journey to becoming a DevOps engineer. First and foremost, we will start with the basic skills required to become a DevOps engineer, then gradually explore the major milestones you need to reach to succeed in this field. Python Go Why Learn Python in DevOps Roadmap? In the U.S.,

Engineering 98

Engineering Programming Language Python Cloud 98

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python 141

Python Cloud Data Science Machine Learning 141

Towards Data Science

JUNE 12, 2024

Python tricks and techniques for data ingestion, validation, processing, and testing: a practical walkthrough Continue reading on Towards Data Science »

Python 52

Python Data Engineering Data Engineer Coding 52

Confessions of a Data Guy

FEBRUARY 25, 2024

I love to write Rust … but I deploy Python. Even when I know I […] The post Why I Love Rust, but Deploy Python appeared first on Confessions of a Data Guy. I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice.

Python 113

Python Building Data IT 113

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Knowledge Hut

FEBRUARY 1, 2024

Variables in Python are fundamental containers used for storing and manipulating data in a program. In Python programming, variables are the backbone of data manipulation and program logic. Mastering Python variables is essential, as they are fundamental to developing efficient and dynamic programming solutions.

Python 98

Python Programming Language Programming Coding 98

Data Engineering Weekly

JULY 14, 2024

[link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. link] Ibis: Querying 1TB on a laptop with Python dataframes Ibis is an open-source dataframe framework to interact with multiple database engines.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Confessions of a Data Guy

MARCH 25, 2024

Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? link] The post How To Build and Open Source PYPI Python Package appeared first on Confessions of a Data Guy.

Python 100

Python Building Project Data 100

Confessions of a Data Guy

MAY 2, 2024

When it comes to debugging […] The post Reading and Processing JSON with Rust vs Python. I’ve found I like being explicit and verbose when writing code, rather than being vague in what I’m doing most of the time. appeared first on Confessions of a Data Guy.

Python 100

Python Process Coding Data 100

Towards Data Science

MAY 22, 2023

Solving data preparation tasks with ChatGPT Photo by Ricardo Gomez Angel on Unsplash Data engineering makes up a large part of the data science process. In our article we solve typical data engineering tasks using ChatGPT and Python. Specifying an action to be performed by an LLM is referred to as “prompt engineering”.

Data Engineering 91

Data Engineering Data Engineer Engineering Data Science 91

Knowledge Hut

JUNE 25, 2024

Python could be a high-level, useful programming language that allows faster work. Python was designed by Dutch computer programmer Guido van Rossum in the late 1980s. For those interested in studying this programming language, several best books for python data science are accessible. out of 5 on the Goodreads website.

Data Science 97

Data Science Python Hadoop Media 97

Knowledge Hut

SEPTEMBER 24, 2024

There is an upswing in the consideration of Software Engineer as a career choice. Software engineers make a huge contribution to the success of many IT ventures or businesses, making them earn a considerable amount. Learn the skills and update your software engineer profile with software development courses.

Software Engineer 98

Software Engineer Software Engineering Engineering Programming Language 98

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. Python, Java, C++, etc.) With Materialize, you can! How does that translate into computer languages?

Software Engineering 147

Software Engineering Software Engineer Engineering Data Lake 147

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. __init__ covers the Python language, its community, and the innovative ways it is being used. In that time there have been a number of generational shifts in how data engineering is done. Closing Announcements Thank you for listening!

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147