Analytics Vidhya

MARCH 14, 2023

SQL is a structured query language used to read and write these databases. In simple words, SQL is used […] The post Top 5 SQL Interview Questions With Implementation appeared first on Analytics Vidhya. This results in the generation of so much data daily.

SQL 202

SQL Database Technology Data 202

Analytics Vidhya

FEBRUARY 27, 2023

So, we are […] The post How to Normalize Relational Databases With SQL Code? If a corrupted, unorganized, or redundant database is used, the results of the analysis may become inconsistent and highly misleading. appeared first on Analytics Vidhya.

Relational Database 255

Relational Database Database SQL Coding 255

KDnuggets

FEBRUARY 24, 2023

This article will discuss SQL visualization, its role in augmenting the modern-day data engineer, and five categories of SQL visualization tools.

SQL 98

SQL Data Engineering Data Engineer Engineering 98

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Analytics Vidhya

FEBRUARY 28, 2023

SQL is widely used in the field of data science and is considered an essential skill to have if you work with data. After being introduced in the 70s, it has become the standard querying language for relational databases. […] The post Step-by-Step Roadmap to Learn SQL in 2023 appeared first on Analytics Vidhya.

SQL 223

SQL Relational Database Data Science Database 223

Analytics Vidhya

FEBRUARY 2, 2023

Introduction SQL injection is an attack in which a malicious user can insert arbitrary SQL code into a web application’s query, allowing them to gain unauthorized access to a database. It occurs when user input is […] The post SQL Injection: The Cyber Attack Hiding in Your Database appeared first on Analytics Vidhya.

Database 204

Database SQL Coding Accessible 204

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools. Stay tuned!!!

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 129

SQL Data Engineering Data Engineer Engineering 129

Confessions of a Data Guy

MAY 7, 2023

I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. appeared first on Confessions of a Data Guy.

SQL 182

SQL Python Coding IT 182

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Start Data Engineering

JULY 1, 2024

Code is an interface to the execution engine 3. How to choose the execution engine and the coding interface 3.1. Chose execution engine based on your workload 3.1.1. Types of execution engine 3.1.2. Criteria to chose your execution engine 3.2. Introduction 2. Types of coding interfaces 3.2.2.

SQL 130

SQL Python Coding Engineering 130

KDnuggets

DECEMBER 6, 2022

SQL is the essential language for developers, engineers, and data professionals. Intermediate knowledge in SQL gives you an edge in your data science career.

SQL 154

SQL Data Science Data Engineering 154

Confessions of a Data Guy

APRIL 16, 2023

I was wondering the other day … since Polars now has a SQL context and is getting more popular by the day, do I need DuckDB anymore? You might think […] The post DuckDB vs Polars for Data Engineering. These two tools are hot. appeared first on Confessions of a Data Guy.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Confessions of a Data Guy

DECEMBER 21, 2023

Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above.

ETL Tools 113

ETL Tools SQL Data Engineering Data Engineer 113

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineering 147

Software Engineering Software Engineer Engineering Data Lake 147

Monte Carlo

AUGUST 30, 2024

Ensuring Data Quality In Dremio Dremio and its SQL Query Engine efficiently queries (but doesn’t move) data across a diverse set of sources. With Monte Carlo, data teams can easily discover virtualized datasets within the platform and create SQL Rules to monitor every data source to which Dremio is connected.

SQL 40

SQL Engineering Data Lake High Quality Data 40

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Uber Engineering

JANUARY 15, 2020

As Uber’s operations became more complex and we offered additional features and … The post Engineering SQL Support on Apache Pinot at Uber appeared first on Uber Engineering Blog.

SQL 112

SQL Engineering Aggregated Data Hadoop 112

databricks

NOVEMBER 5, 2024

We’re excited to announce that materialized views (MVs) and streaming tables (STs) are now Generally Available in Databricks SQL on AWS and Azure.

SQL 114

SQL AWS Engineering 114

Cloudera

FEBRUARY 21, 2023

SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL as a part of Cloudera Streaming Analytics, built on top of Apache Flink. It enables users to easily write, run, and manage real-time continuous SQL queries on stream data and a smooth user experience. What is a data transformation?

SQL 110

SQL Kafka Raw Data Data 110

Data Engineering Weekly

SEPTEMBER 29, 2024

Airbnb writes about Sandcastle, an Airbnb-internal prototyping platform that enables data scientists, engineers, and product managers to bring data/AI ideas to life. While visually suggesting a sequence, Pipe SQL maintains SQL's declarative nature. What do you think about Pipe syntax in SQL?

Data Engineering 62

Data Engineering Data Engineer Engineering SQL 62

Seattle Data Guy

FEBRUARY 26, 2024

SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services.

SQL 130

SQL Project Data Consulting 130

Data Engineering Podcast

DECEMBER 22, 2019

To address that need more completely the team at Materialize has created an engine that allows for building queryable views of your data as it is continually updated from the stream of changes being generated by your applications. In the list of features, you highlight full support for ANSI SQL 92.

SQL 100

SQL Engineering Building Java 100

Data Engineering Weekly

JUNE 10, 2024

Learn More → Cube Research: Crystallizing Snowflake Summit 2024 We should officially call the first week of June the data engineering week, as two major data companies are running their developer conference. As I have shared , its impact on data engineering is exciting.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 131

SQL Data Data Engineering Data Engineer 131

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Snowflake

APRIL 17, 2024

Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Our previous Python connector API , primarily available for those who need to run SQL via a Python script, enabled a connection to Snowflake from Python applications.

Data Pipeline 109

Data Pipeline Python Data Engineering Data Engineer 109

Data Engineering Podcast

SEPTEMBER 7, 2020

To address this problem Martin Traverso and his colleagues at Facebook built the Presto distributed query engine. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Weekly

MAY 26, 2024

[link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineering 76

Data Engineering Data Engineer Engineering Google Cloud 76

Data Engineering Weekly

OCTOBER 6, 2024

montecarlodata.com Uber: Making Uber’s ExperimentEvaluation Engine 100x Faster Uber writes about redesigning its Experimentation platform to reduce the latency of experiment evaluations by 100x. “text-to-SQL” and “text-to-insight.” link] Rachana Aluri: Text-to-SQL’s Power Players: Comparing Claude 3.5

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Confessions of a Data Guy

DECEMBER 29, 2023

The post SQL Bad, Reddit Mad appeared first on Confessions of a Data Guy.

SQL 100

SQL Data Data Engineering Data Engineer 100

Data Engineering Weekly

JULY 14, 2024

[link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. I used to joke the entire Walstreet runs on unknown SQL codes that no one understands. The buffer manager caches as many pages as possible.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! Learn more about how batch and streaming data pipelines are built at Netflix.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Look no further than Materialize, the streaming database you already know how to use.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

DoorDash Engineering

FEBRUARY 27, 2024

Two primary aspects of that search engine were causing the trouble: its document-replication mechanism and its lack of support for complex document relationships. We decided the best way to address these challenges was to move away from Elasticsearch to a homegrown search engine. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Jesse Anderson

DECEMBER 12, 2022

Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Spark came in 2009 and gave a unified batch and streaming engine. Apache Flink came in 2011 and gave us our first real streaming engine. It is prominently data engineering and all of the technical debt difficulties that come with data.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130