Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is.

SQL 182

SQL Python Coding IT 182

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0

SQL 120

SQL Python Data Engineer Data Engineering 120

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. These two tools are hot.

Data Engineer 147

Data Engineer Data Engineering Engineering SQL 147

KDnuggets

FEBRUARY 13, 2023

SQL and Python Interview Questions for Data Analysts • Learn Machine Learning From These GitHub Repositories • Learn Data Engineering From These GitHub Repositories • The ChatGPT Cheat Sheet • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2

SQL 100

SQL Python Machine Learning Data Engineer 100

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Weekly

OCTOBER 13, 2024

Genie addresses these challenges by providing quick and accurate answers to questions, retrieving relevant information from internal knowledge bases, and reducing the need for constant interaction with on-call engineers.

Data Engineer 52

Data Engineer Data Engineering Engineering PostgreSQL 52

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python 141

Python Cloud Data Science Machine Learning 141

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Data Engineering Weekly

MAY 26, 2024

link] Akshay Agrawal: Lessons learned reinventing the Python notebook I wrote about the need for a Notebook-like orchestration system: A Notebook is all I want or Don't. link] Ryan Eakman: SQLFrame - Turning PySpark into a Universal DataFrame API SQL or DataFrames, two programming models often used interchangeability in engineering data.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. We can use this to steal sensitive information or make unauthorized changes to the data stored in the database.

Database 204

Database SQL Coding Accessible 204

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. Your user base has quintupled, analytics requests are piling up, and that trusty Python script crashes more often than it runs. Back then, it worked just fine for your small team and handful of customers.

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Towards Data Science

NOVEMBER 20, 2024

It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc). Similarly, data engineering positions have seen a 98% increase.

Data Engineer 62

Data Engineer Data Engineering Engineering Electronics 62

Snowflake

JUNE 20, 2024

In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. Snowflake customers see an average of 4.6x

Data Engineer 92

Data Engineer Data Engineering Scala Engineering 92

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 79

Data Engineer Data Engineering Engineering SQL 79

Data Engineering Weekly

JULY 14, 2024

[link] Wasteman.codes: Engineering Principles for Building Financial Systems Any system that involves moving money or counting money is always complex to get it correct. I used to joke the entire Walstreet runs on unknown SQL codes that no one understands. Streaming execution to process a small chunk of data at a time.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. With Materialize, you can! Hex brings everything together. Hex brings everything together.

Building 189

Building SQL BI Python 189

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Cleanse 98

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Team Data Science

JANUARY 8, 2021

The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.

Data Engineer 130

Data Engineer Data Engineering Engineering Unstructured Data 130

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Cloud 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

KDnuggets

FEBRUARY 8, 2023

SQL and Python Interview Questions for Data Analysts • 20 Questions (with Answers) to Detect Fake Data Scientists: ChatGPT Edition, Part 2 • ChatGPT for Beginners • Python String Matching Without Complex RegEx Syntax • Learn Data Engineering From These GitHub Repositories

Python 78

Python SQL Data Engineer Data Engineering 78

Data Engineering Podcast

FEBRUARY 6, 2022

In order to make geospatial analytics more maintainable and scalable there has been an increase in the number of database engines that provide extensions to their SQL syntax that supports manipulation of spatial data. Can you describe what spatial SQL is and some of the use cases that it is relevant for?

SQL 100

SQL Building IT Google Cloud 100

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

SEPTEMBER 24, 2023

In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

SQL 147

SQL BI Machine Learning Python 147

Data Engineering Podcast

SEPTEMBER 3, 2023

In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Data Integration 162

Data Integration BI SQL Python 162

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. Contents: What is the role of an Azure Data Engineer?

Data Engineer 52

Data Engineer Data Engineering Engineering Amazon Web Services 52

Data Engineering Podcast

SEPTEMBER 10, 2023

In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 208

BI SQL Machine Learning Data 208

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Edureka

AUGUST 29, 2024

The responsibilities of a data engineer imply that the person in this role designs, creates, develops, and maintains systems and architecture that allow them to collect, store, and interpret data. What Does a Data Engineer Do? Why Choose Data Engineering as a Career? How to Become a Data Engineer?

Data Engineer 40

Data Engineer Data Engineering Engineering Generalist 40

Data Engineering Weekly

AUGUST 13, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. However, I do think SQL is not fully reflecting the underlying pattern.

Data Engineer 52

Data Engineer Data Engineering Engineering Food 52