Analytics Vidhya

FEBRUARY 21, 2023

Introduction SQL is a database programming language created for managing and retrieving data from Relational databases like MySQL, Oracle, and SQL Server. SQL(Structured Query Language) is the common language for all databases. In other terms, SQL is a language that communicates with databases.

SQL 168

SQL MySQL Programming Language Relational Database 168

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. appeared first on Confessions of a Data Guy.

SQL 182

SQL Python Coding IT 182

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 137

SQL Data Engineer Data Engineering Engineering 137

KDnuggets

DECEMBER 6, 2022

SQL is the essential language for developers, engineers, and data professionals. Intermediate knowledge in SQL gives you an edge in your data science career.

SQL 155

SQL Data Science Data Engineering 155

Start Data Engineering

OCTOBER 17, 2024

Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3. STRUCT data types are sorted based on their keys from left to right 1.4.

SQL 130

SQL Data Engineer Data Engineering Engineering 130

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.

Data Engineer 240

Data Engineer Data Engineering Engineering Data 240

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. ) In a simple words dbt sits on top of your raw data to organise all your SQL queries that are defining your data assets.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. These two tools are hot. appeared first on Confessions of a Data Guy.

Data Engineer 147

Data Engineer Data Engineering Engineering SQL 147

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). From March 26th-28th 2024, they'll play host to hundreds of attendees, 100 top speakers, and dozens of startups that are advancing data science, engineering and AI.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K So I came up with a solution.

SQL 100

SQL Data Engineer Data Engineering Engineering 100

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!

Data Engineer 119

Data Engineer Data Engineering Engineering Generalist 119

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Start Data Engineering

MAY 21, 2024

Introduction 2. Enabling Stakeholder data access with RAGs 3.1. Set up 3.1.1. Pre-requisite 3.1.2. Demo 3.1.3. Key terminology 3.2. Loading: Read raw data and convert them into LlamaIndex data structures 3.2.1. Read data from structured and unstructured sources 3.2.2. Transform data into LlamaIndex data structures 3.3.

Accessibility 213

Accessibility Accessible SQL Raw Data 213

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.0

SQL 134

SQL Python Data Engineer Data Engineering 134

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.

Data Engineer 111

Data Engineer Data Engineering Engineering Database-centric 111

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

KDnuggets

JANUARY 19, 2023

In this article, we will discuss use cases and methods for using ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes along with SQL to integrate data from various sources.

SQL 137

SQL Data Integration Data Process 137

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Look no further than Materialize, the streaming database you already know how to use.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. BigQuery supports DELETE to delete partitions in a SQL query. New Apache Arrow engines — Arrow has become one of the most used library when it comes to built in-memory engines. I'm speechless.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Christophe Blefari

OCTOBER 19, 2023

With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries. Fast forward to 2 years later, you find yourself with hundreds or thousands of SQL queries.

Project 264

Project Finance SQL Government 264

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineer Data Engineering 135

Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

DoorDash Engineering

FEBRUARY 27, 2024

Two primary aspects of that search engine were causing the trouble: its document-replication mechanism and its lack of support for complex document relationships. We decided the best way to address these challenges was to move away from Elasticsearch to a homegrown search engine. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Data Engineering Podcast

SEPTEMBER 10, 2023

This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

BI 208

BI SQL Data Machine Learning 208

Seattle Data Guy

FEBRUARY 26, 2024

SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services.

SQL 130

SQL Project Data Consulting 130

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above.

ETL Tools 113

ETL Tools SQL Data Engineer Data Engineering 113

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Our previous Python connector API , primarily available for those who need to run SQL via a Python script, enabled a connection to Snowflake from Python applications.

Data Pipeline 113

Data Pipeline Python Data Engineer Data Engineering 113

databricks

NOVEMBER 5, 2024

We’re excited to announce that materialized views (MVs) and streaming tables (STs) are now Generally Available in Databricks SQL on AWS and Azure.

SQL 132

SQL AWS Engineering 132

Data Engineering Weekly

JUNE 10, 2024

Learn More → Cube Research: Crystallizing Snowflake Summit 2024 We should officially call the first week of June the data engineering week, as two major data companies are running their developer conference. As I have shared , its impact on data engineering is exciting.

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Data Engineering Podcast

OCTOBER 29, 2023

In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 189

Building SQL BI Python 189

databricks

NOVEMBER 13, 2023

Today, we introduce the new availability of named arguments for SQL functions. With this feature, you can invoke functions in more flexible ways.

SQL 105

SQL Data Engineer Data Engineering Engineering 105

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! Learn more about how batch and streaming data pipelines are built at Netflix.

Data Engineer 106

Data Engineer Data Engineering Engineering Metadata 106