Start Data Engineering

OCTOBER 14, 2024

Using nested data types effectively 3.1. Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Nested data types can be sorted 3.3.3. Use STRUCT for one-to-one & hierarchical relationships 3.2.

SQL 130

SQL Data Schemas Data Coding 130

Data Engineering Podcast

SEPTEMBER 24, 2021

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. What are the techniques/technologies that teams might use to optimize or scale out their data processing workflows? Can you describe what Bodo is and the story behind it?

Data Process 100

Data Process Python Process Data Lake 100

DataKitchen

NOVEMBER 4, 2021

The typical pharmaceutical organization faces many challenges which slow down the data team: Raw, barely integrated data sets require engineers to perform manual , repetitive, error-prone work to create analyst-ready data sets. Cloud computing has made it much easier to integrate data sets, but that’s only the beginning.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Data Engineering Podcast

FEBRUARY 20, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Can you describe what Fugue is and the story behind it?

Python 100

Python Data Process IT Building 100

Cloudera

MAY 18, 2021

Event-driven and streaming architectures enable complex processing on market events as they happen, making them a natural fit for financial market applications. Flink SQL is a data processing language that enables rapid prototyping and development of event-driven and streaming applications. You can view the code here.

SQL 99

SQL Java Data Business Analyst 99

Snowflake

APRIL 9, 2025

It employs Snowpark Container Services to build scalable AI/ML models for satellite data processing and Snowflake AI/ML functions to enable advanced analytics and predictive insights for satellite operators. Sherloq aims to change this by offering a collaborative platform for managing and documenting data analytics workflows.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Snowflake

JUNE 5, 2024

With Snowpark’s existing DataFrame API , users have access to a robust framework for lazily evaluated, relational operations on data, closely resembling Spark’s conventions. pandas is the go-to data processing library for millions worldwide, including countless Snowflake users. Why introduce a distributed pandas API?

Python 128

Python Programming Language Government SQL 128

Towards Data Science

DECEMBER 8, 2023

Clustering: A simple way to group similar rows and prevent unnecessary data processing Continue reading on Towards Data Science »

SQL 98

SQL Data Science Data Process Process 98

Data Engineering Weekly

FEBRUARY 18, 2025

Fluss is a compelling new project in the realm of real-time data processing. I spoke with Jark Wu , who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Among the 20,000 Flink SQL jobs at Alibaba, only 49% of columns of Kafka data are read on average.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Striim

MARCH 21, 2025

To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing. Your electric consumption is collected during a month and then processed and billed at the end of that period.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Data Engineering Weekly

FEBRUARY 9, 2025

link] QuantumBlack: Solving data quality for gen AI applications Unstructured data processing is a top priority for enterprises that want to harness the power of GenAI. It brings challenges in data processing and quality, but what data quality means in unstructured data is a top question for every organization.

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

Snowflake

JUNE 4, 2024

Snowflake offers a secure, streamlined approach to developing across data workloads, reducing costs and reliance on external tools. This means faster development and happier data teams. Explore and experiment with data, visualize results, share insights — all in one place. Let’s dive deeper into what we announced.

Data Pipeline 122

Data Pipeline Python SQL Database 122

Team Data Science

DECEMBER 17, 2020

It’s not a must for data scientist to have skill in data engineering before they can analyze data processed by data engineer or before they can move uniformly with other group (involving data engineer) for the progress of the company. Data scientists should acquire some basic SQL functionality.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Snowflake

JUNE 6, 2024

Snowflake Notebooks aim to provide a convenient, easy-to-use interactive environment that seamlessly blends Python, SQL and Markdown, as well as integrations with key Snowflake offerings, like Snowpark ML, Streamlit, Cortex and Iceberg tables. Discover valuable business insights through exploratory data analysis.

SQL 115

SQL Python Machine Learning Data Workflow 115

Hevo

APRIL 26, 2024

Most organizations find it challenging to manage data from diverse sources efficiently. Amazon Web Services (AWS) enables you to address this challenge with Amazon RDS, a scalable relational database service for Microsoft SQL Server (MS SQL). However, simply storing the data isn’t enough.

AWS 52

AWS Amazon Web Services Data Process Process 52

Striim

MARCH 21, 2025

In addition to log files, sensors, and messaging systems, Striim continuously ingests real-time data from cloud-based or on-premises data warehouses and databases such as Oracle, Oracle Exadata, Teradata, Netezza, Amazon Redshift, SQL Server, HPE NonStop, MongoDB, and MySQL.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineering 248

Data Engineering Data Engineer Engineering Data 248

Netflix Tech

DECEMBER 17, 2024

To add this metric to DJ, they need to provide two pieces of information: The fact table that the metric comesfrom: SELECT account_id, country_iso_code, streaming_hours FROM streaming_fact_table The metric expression: `SUM(streaming_hours)` Then metric consumers throughout the organization can call DJ to request either the SQL or the resulting data.

Engineering 84

Engineering Entertainment Amazon Web Services Utilities 84

Knowledge Hut

JANUARY 18, 2024

“Big data Analytics” is a phrase that was coined to refer to amounts of datasets that are so large traditional data processing software simply can’t manage them. For example, big data is used to pick out trends in economics, and those trends and patterns are used to predict what will happen in the future.

Data Process 52

Data Process Process Hadoop Scala 52

Knowledge Hut

MARCH 14, 2024

Should that be the case, Azure SQL Database might be your best bet. Microsoft SQL Server's functionalities are fully included in Azure SQL Database, a cloud-based database service that also offers greater flexibility and scalability. In this article, I will cover the various aspects of Azure SQL Database.

Database 52

Database SQL Relational Database BI 52

Maxime Beauchemin

APRIL 26, 2023

To achieve this, we’re integrating AI into various aspects of our product, such as natural language data queries, text-to-SQL, and chart suggestions. Focusing on the text-to-SQL use case specifically, GPT-4 is exceptional at writing code and highly proficient in SQL. However, to perform optimally, GPT requires context.

SQL 148

SQL Database Engineering Software Engineer 148

Striim

MARCH 21, 2025

Change Data Capture (CDC) has emerged as an ideal solution for near real-time movement of data from relational databases (like SQL Server or Oracle) to data warehouses, data lakes or other databases. The final step of ETL involves loading data into the target destination.

IT 52

IT Data Lake Data Warehouse Relational Database 52

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. Use SQL, Python, R, no-code and AI to find and share insights across your organization.

Building 130

Building Machine Learning SQL Python 130

Snowflake

NOVEMBER 1, 2023

Snowflake’s latest ML announcements Develop interactively with SQL and Python in Snowflake Notebooks Snowflake Notebooks, in private preview, is a new development interface that offers an interactive, cell-based programming environment for Python and SQL users to explore, process and experiment with data in Snowpark.

Building 119

Building Python SQL Programming Language 119

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Set Up Auto-Scaling: Configure auto-scaling for your data processing and storage resources.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Data Engineering Weekly

JANUARY 12, 2025

link] Gradient Flow: Paradigm Shifts in Data Processing for the Generative AI Era data processing pipelines haven't kept pace with the rapid advancement of AI models The article highlights the growing importance of preprocessing data pipelines, but the pipeline processing techniques do not match the demand.

Pipeline-centric 57

Pipeline-centric Data Engineering Data Engineer Engineering 57

Snowflake

JUNE 20, 2024

In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models.

Data Engineering 111

Data Engineering Data Engineer Scala Engineering 111

Snowflake

MAY 31, 2023

Define a Sundeck SQL post-hook that examines current load and time of day to suspend idle warehouses. Implement a Sundeck SQL post-hook that collects query activity and records that in a Snowflake table, which is then consulted in a SQL pre-hook to reject excessive consumers (unless a manager overrides).

SQL 127

SQL Database Data Engineering Data Engineer 127

Netflix Tech

DECEMBER 3, 2022

Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring. See below example of hooking the table creation SQL file into the main workflow definition. -

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Cloudera

MARCH 2, 2023

Iceberg is a high-performance open table format for huge analytic data sets. It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. This enables you to maximize utilization of streaming data at scale.

Process 114

Process SQL Kafka Database 114

Data Engineering Weekly

JANUARY 15, 2025

The Critical Role of AI Data Engineers in a Data-Driven World How does a chatbot seamlessly interpret your questions? The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. How does a self-driving car understand a chaotic street scene?

Data Engineering 62

Data Engineering Data Engineer Unstructured Data Engineering 62

Snowflake

JANUARY 23, 2024

Behind the scenes, Snowpark ML parallelizes data processing operations by taking advantage of Snowflake’s scalable computing platform. This is a first-class, schema-level Snowflake object that provides a versioned container of ML model artifacts with full role-based access control (RBAC) support, and APIs for Python and SQL.

Machine Learning 138

Machine Learning Metadata Python Telecommunication 138

Christophe Blefari

DECEMBER 16, 2022

For instance a small team of 2 analytics engineers will pay now $2400/year just to have a server running their SQL queries and a web IDE that is yet to perfect. Query your data in Kafka using SQL — This is a post that compares Flink, ksqlDB, Trino, Materialize, RisingWave and timeplus (the authors) in order to query Kafka.

Kafka 130

Kafka Data SQL Cloud 130

Cloudyard

JANUARY 21, 2025

Read Time: 6 Minute, 6 Second In modern data pipelines, handling data in various formats such as CSV, Parquet, and JSON is essential to ensure smooth data processing. However, one of the most common challenges faced by data engineers is the evolution of schemas as new data comes in.

Data Schemas 59

Data Schemas Data Pipeline Data Warehouse Data Storage 59

Striim

NOVEMBER 17, 2023

Meanwhile, Google BigQuery ML is a machine learning service provided by Google Cloud, allowing you to create and deploy machine learning models using SQL-like syntax directly within the BigQuery environment. ensuring the consistency and integrity of your data. ELSE data[1] END, data[2] = CASE WHEN data[2] IS NULL THEN TO_FLOAT(0.0)

Machine Learning 52

Machine Learning Data Process PostgreSQL Process 52

Christophe Blefari

APRIL 19, 2024

Structured generative AI — Oren explains how you can constraint generative algorithms to produce structured outputs (like JSON or SQL—seen as an AST). This is super interesting because it details important steps of the generative process. SQLMesh is bringing fresh ideas to the SQL transformation landscape.

MySQL 130

MySQL Data Datasets SQL 130