This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Data Data Engineer Python 
Confessions of a Data Guy
FEBRUARY 26, 2023
Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.
356 
Data Engineering Podcast
JUNE 25, 2023
Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?
225 
Start Data Engineering
MAY 30, 2024
Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)
147 
KDnuggets
APRIL 25, 2024
Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.
160 
KDnuggets
SEPTEMBER 2, 2024
Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.
152 
Analytics Vidhya
FEBRUARY 6, 2023
This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.
234 
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.
217 
Analytics Vidhya
FEBRUARY 6, 2023
Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.
218 
Snowflake
APRIL 17, 2024
In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. Recognizing this shift, Snowflake is taking a Python-first approach to bridge the gap and help users leverage the power of both worlds.
113 
Christophe Blefari
JANUARY 20, 2024
Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?
130 
Data Engineering Podcast
FEBRUARY 5, 2023
In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.
147 
Data Engineering Podcast
FEBRUARY 18, 2024
Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.
262 
Jesse Anderson
DECEMBER 12, 2022
They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data.
130 
Data Engineering Weekly
JULY 21, 2024
Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.
119 
Seattle Data Guy
FEBRUARY 11, 2023
Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?
147 
Confessions of a Data Guy
APRIL 16, 2023
You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.
147 
Confessions of a Data Guy
SEPTEMBER 9, 2023
In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.
162 
Waitingforcode
FEBRUARY 3, 2023
In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.
130 
Confessions of a Data Guy
JANUARY 19, 2023
I would dare say it’s a rite of passage for most data folk. For those using Python, it’s probably one of the […] The post Replacing Pandas with Polars. appeared first on Confessions of a Data Guy. We’ve all been there, done that. Pandas I mean. A Practical Guide.
361 
Data Engineering Weekly
JULY 28, 2024
Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.
111 
Snowflake
OCTOBER 23, 2023
One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.
141 
Data Engineering Podcast
JULY 2, 2023
In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is feature engineering is and why/to whom it matters?
130 
Data Engineering Podcast
MAY 18, 2024
Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.
278 
Confessions of a Data Guy
OCTOBER 6, 2023
I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. Running Rust inside Apache Airflow.
147 
Seattle Data Guy
NOVEMBER 19, 2024
Scraping data from PDFs is a right of passage if you work in data. Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Someone somewhere always needs help getting invoices parsed, contracts read through, or dozens of other use cases.
130 
KDnuggets
MARCH 22, 2023
SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0
134 
Confessions of a Data Guy
JUNE 3, 2024
Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.
130 
Data Engineering Podcast
SEPTEMBER 17, 2023
Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!
189 
Data Engineering Podcast
SEPTEMBER 10, 2023
Summary Data systems are inherently complex and often require integration of multiple technologies. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. With Materialize, you can!
208 
Cloudyard
SEPTEMBER 2, 2024
Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.
96 
Christophe Blefari
MARCH 2, 2024
Mistral ( credits ) Hello all, this is the Data News, this week edition might be smaller than usual in term of comments as I'm working on a Data News related project that takes me a bit of time, which will probably lead to a series of articles. In 2024 we are more than ever tools to move data from sources to destinations.
162 
Data Engineering Weekly
NOVEMBER 24, 2024
Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. We are planning many exciting product lines to trial and launch in 2025.
Data Engineering Podcast
JUNE 16, 2024
Summary Stripe is a company that relies on data to power their products and business. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.
147 
Analytics Vidhya
FEBRUARY 20, 2023
This blog is a tutorial for building intuitive frontend interfaces for Machine Learning models using two popular open-source libraries […] The post Streamlit vs Gradio – A Guide to Building Dashboards in Python appeared first on Analytics Vidhya.
152 
Data Engineering Weekly
JULY 14, 2024
link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. Techniques for turning text data and documents into vector embeddings and structured data.
Data Engineering Podcast
DECEMBER 24, 2023
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.
245 
Data Engineering Podcast
FEBRUARY 4, 2024
In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.
173 
Confessions of a Data Guy
FEBRUARY 25, 2024
I love to write Rust … but I deploy Python. Even when I know I […] The post Why I Love Rust, but Deploy Python appeared first on Confessions of a Data Guy. I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice.
113 
KDnuggets
APRIL 19, 2024
The collection includes free courses on Python, SQL, Data Analytics, Business Intelligence, Data Engineering, Machine Learning, Deep Learning, Generative AI, and MLOps.
155 
Snowflake
JUNE 20, 2024
In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. The tool serves two primary functions: assessment and conversion.
KDnuggets
JUNE 25, 2024
This tutorial will teach you how to create minimal Docker images for Python applications.
134 
databricks
DECEMBER 10, 2024
Data engineering teams are frequently tasked with building bespoke ingestion solutions for myriad custom, proprietary, or industry-specific data sources. Many teams find that.
103 
Knowledge Hut
FEBRUARY 1, 2024
Variables in Python are fundamental containers used for storing and manipulating data in a program. In Python programming, variables are the backbone of data manipulation and program logic. They hold and transform data, allowing for the execution of algorithms and the management of large datasets.
98 
Analytics Vidhya
MARCH 5, 2023
Introduction NumPy is an open-source library in python and a must-learn if you want to enter the data science ecosystem. It is the library underpinning other important libraries such as Pandas, matplotlib, Scipy, scikit-learn, etc. One of the reasons this library is so foundational is because of its array of programming capabilities.
269
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content