Scala - Data Engineering Digest

Scala as a Junior Developer

Rock the JVM

SEPTEMBER 17, 2023

Lucas’ story is shared by lots of beginner Scala developers, which is why I wanted to post it here on the blog. I’ve watched thousands of developers learn Scala from scratch, and, like Lucas, they love it! If you want to learn Scala well and fast, take a look at my Scala Essentials course at Rock the JVM. sum > 8 ).

Scala

Scala Programming Coding Java

Useful classes for data engineers - Scala & Java

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. We all have our habits and as programmers, libraries and frameworks are definitely a part of the group. The part for Python will follow next week!

Scala

Scala Java Data Engineering Data Engineer

A Functional Load Balancer with Scala, Http4s and Cats Effect

Rock the JVM

OCTOBER 29, 2023

Project Structure We will use Scala 3.3.0, values :+ Url ( "url1" )) assertEquals ( obtained , expected ) } } Let’s run tests in SBT: sbt:loadbalancer> test [ info] compiling 1 Scala source to ~/loadbalancer/target/scala-3.3.0/test-classes. info] compiling 2 Scala sources to ~/loadbalancer/target/scala-3.3.0/test-classes.

Scala

Scala Bytes Algorithm Coding

Webinars

How to Achieve High-Accuracy Results When Using LLMs

MORE WEBINARS

WebSockets in Scala, Part 2: Integrating Redis and PostgreSQL

Rock the JVM

MAY 22, 2024

Let’s create a validateutility.scala in the following path, src/main/scala/rockthejvm/websockets/domain , and add the following code: package rockthejvm.websockets.domain import cats.data.Validated object validateutility { def validateItem [ F ]( value : String , userORRoom : F , name : String ) : Validated [ String , F ] = { Validated.

PostgreSQL

PostgreSQL Scala Database SQL

dbt on Databricks.

Confessions of a Data Guy

MARCH 4, 2025

Databricks: A platform that unifies data engineering and data science pipelines, typically with Spark (PySpark, Scala) or SparkSQL. Context and Motivation dbt (Data Build Tool): A popular open-source framework that organizes SQL transformations in a modular, version-controlled, and testable way.

Scala

Scala Data Science SQL Data Engineering

Guide to Scala 3 Macros

Rock the JVM

FEBRUARY 20, 2025

A long-form guide on Scala 3 macros - learn how to use them, how Scala macros work, and why they exist

Scala

A Detailed Guide of Interview Questions on Apache Kafka

Analytics Vidhya

APRIL 28, 2023

It is a famous Scala-coded data processing tool that offers low latency, extensive throughput, and a unified platform to handle the data in real-time. Introduction Apache Kafka is an open-source publish-subscribe messaging application initially developed by LinkedIn in early 2011.

Kafka

Kafka Scala Coding Data Process

Getting Started with Scala and Apache Kafka

Confluent

DECEMBER 8, 2020

If you’re getting started with Apache Kafka® and event streaming applications, you’ll be pleased to see the variety of languages available to start interacting with the event streaming platform. It […].

Kafka

Kafka Scala IT

A guide to UDP in Scala with FS2

Rock the JVM

DECEMBER 17, 2023

Setting Up Let’s create a new Scala 3 project and add the following to your build.sbt file. The UDP Server Create Fs2Udp.scala in the following path, src/main/scala/com/rockthejvm/fs2Udp/Fs2Udp.scala and add the following code: package com.rockthejvm.fs2Udp import cats.effect. val scala3Version = "3.3.1" lazy val root = project.

Scala

Scala Bytes Java Coding

Two-Factor Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

If you want to master the Typelevel Scala libraries (including Http4s) with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. HOTP scala implementation HOTP generation is quite tedious, therefore for simplicity, we will use a java library, otp-java by Bastiaan Jansen.

Scala

Scala Java Bytes Algorithm

Scala In Demand Technologies Built On Scala

Knowledge Hut

MAY 20, 2024

The term Scala originated from “Scalable language” and it means that Scala grows with you. In recent times, Scala has attracted developers because it has enabled them to deliver things faster with fewer codes. Developers are now much more interested in having Scala training to excel in the big data field.

Scala

Scala Technology Kafka Hadoop

Arbitrary stateful processing in PySpark with applyInPandasWithState

Waitingforcode

SEPTEMBER 27, 2023

It's always a huge pleasure to see the PySpark API covering more and more Scala API features. But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look. Starting from Apache Spark 3.4.0 you can even write arbitrary stateful processing jobs!

Process

Process Scala IT

gRPC in Scala with Fs2 and Scalapb

Rock the JVM

OCTOBER 1, 2023

Scala is not officially supported at the moment however the ScalaPB library provides a good wrapper around the official gRPC Java implementation, it provides an easy API that enables translation of Protocol Buffers to Scala case classes with support for Scala3, Scala.js, and Java Interoperability. Setting Up. in ( file ( "protobuf" )).

Scala

Scala Metadata Transportation Java

Scala 3 Inlines Explained

Rock the JVM

JANUARY 31, 2025

Learn Scala 3 inlines - a powerful tool for code expansion at compile time, which can improve type safety and (if you know what you're doing) performance

Scala

Scala Coding

Making applyInPandasWithState less painful

Waitingforcode

OCTOBER 4, 2023

However, due to Python duck typing, some operations are more difficult and more risky to express in the code than in the strongly typed Scala API. Having applyInPandasWithState in the PySpark API is huge!

Scala

Scala Python Coding

Unity Catalog Lakeguard: Industry-first and only data governance for multi-user Apache™ Spark clusters

databricks

APRIL 24, 2024

Run SQL, Python & Scala workloads with full data governance & cost-efficient multi-user compute. Unlock the power of Apache Spark™ with Unity Catalog Lakeguard on Databricks Data Intelligence Platform.

Data Governance

Data Governance Government Scala SQL

Adopting Spark Connect

Towards Data Science

NOVEMBER 6, 2024

However, this ability to remotely run client applications written in any supported language (Scala, Python) appeared only in Spark 3.4. In any case, all client applications use the same Scala code to initialize SparkSession, which operates depending on the run mode. getOrCreate() // If the client application uses your Scala code (e.g.,

Scala

Scala Java AWS Coding

A Comprehensive Guide to Choosing the Best Scala Course

Rock the JVM

MAY 22, 2023

This article is all about choosing the right Scala course for your journey. How should I get started with Scala? Do you have any tips to learn Scala quickly? How to Learn Scala as a Beginner Scala is not necessarily aimed at first-time programmers. Which course should I take?

Scala

Scala Java Programming Language Programming

How to Tune RocksDB for Your Kafka Streams Application

Confluent

MARCH 10, 2021

Apache Kafka ships with Kafka Streams, a powerful yet lightweight client library for Java and Scala to implement highly scalable and elastic applications and microservices that process and analyze data […].

Kafka

Kafka Scala Java Process

OAuth Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

SNAPSHOT" , scalaVersion := scala3Version , libraryDependencies ++= Seq ( emberServer , emberClient , http4sDsl , http4sCirce , ciris , cirisCirce ) ++ circeLibs ) We use Scala 3 in this tutorial, if you need to use Scala 2.13, you can do so with minimal code changes. val Http4sVersion = "0.23.18" val CirisVersion = "3.2.0" pure [ F ].

Scala

Scala Coding Accessible Accessibility

PySpark and vectorized User-Defined Functions

Waitingforcode

FEBRUARY 3, 2023

The Scala API of Apache Spark SQL has various ways of transforming the data, from the native and User-Defined Function column-based functions, to more custom and row-level map functions. PySpark doesn't have this mapping feature but does have the User-Defined Functions with an optimized version called vectorized UDF!

Scala

Scala SQL Data

Scala Vs Python Vs R Vs Java - Which language is better for Spark & Why?

Knowledge Hut

MAY 3, 2024

If you search top and highly effective programming languages for Big Data on Google, you will find the following top 4 programming languages: Java Scala Python R Java Java is one of the oldest languages of all 4 programming languages listed here. Scala is a highly Scalable Language. Scala is the native language of Spark.

Scala

Scala Java Python Programming Language

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

Rock the JVM

MAY 22, 2024

Learn how to integrate Redis with Scala using redis4cats and persist records in Postgres with skunk, enhancing your chatroom application with powerful new features

Scala

Scala PostgreSQL

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

Rock the JVM

MAY 22, 2024

Learn how to integrate Redis with Scala using redis4cats and persist records in Postgres with skunk, enhancing your chatroom application with powerful new features

Scala

Scala PostgreSQL

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

Rock the JVM

MAY 22, 2024

Learn how to integrate Redis with Scala using redis4cats and persist records in Postgres with skunk, enhancing your chatroom application with powerful new features

Scala

Scala PostgreSQL

A Distributed Code Execution Engine in Pekko with Scala

Rock the JVM

JUNE 13, 2024

A practical guide to building a distributed system with Scala and Pekko: learn how to run other people's code in a real-world scenario

Scala

Scala Coding Engineering Systems

The Dog Days of PySpark

Confessions of a Data Guy

APRIL 15, 2023

PySpark is the abstraction that lets a bazillion Data Engineers forget about that blight Scala and cuddle their wonderfully soft and ever-kind Python code, while choking down gobs of data like some Harkonnen glutton. One of those things to hate and love, well … kinda hard not to love.

Scala

Scala Python Data Engineering Data Engineer

WebSockets in Scala: Part 1 - http4s

Rock the JVM

FEBRUARY 18, 2024

Learn how to implement WebSockets in Scala with http4s to enable seamless two-way communication between your frontend and backend

Scala

WebSockets in Scala: Part 1 - http4s

Rock the JVM

FEBRUARY 18, 2024

Learn how to implement WebSockets in Scala with http4s to enable seamless two-way communication between your frontend and backend

Scala

WebSockets in Scala: Part 1 - http4s

Rock the JVM

FEBRUARY 18, 2024

Learn how to implement WebSockets in Scala with http4s to enable seamless two-way communication between your frontend and backend

Scala

UDP with Scala and FS2 Explained

Rock the JVM

DECEMBER 17, 2023

Learn how UDP works and discover how to transition from an old Java NIO implementation to a more modern and expressive API like FS2 in Scala

Scala

Scala Java

UDP with Scala and FS2 Explained

Rock the JVM

DECEMBER 17, 2023

Learn how UDP works and discover how to transition from an old Java NIO implementation to a more modern and expressive API like FS2 in Scala

Scala

Scala Java

How to Write a Full-Stack Scala 3 Application with the Typelevel Stack

Rock the JVM

JANUARY 22, 2024

Introduction The Typelevel stack is one of the most powerful sets of libraries in the Scala ecosystem. They allow you to write powerful applications with pure functional programming - as of this writing, the Typelevel ecosystem is one of the biggest selling points of Scala. The Typelevel stack is based on Cats and Cats Effect.

Scala

Scala SQL Database Coding

Building ETL Pipeline with Snowpark

Cloudyard

DECEMBER 24, 2024

Snowflakes Snowpark is a game-changing feature that enables data engineers and analysts to write scalable data transformation workflows directly within Snowflake using Python, Java, or Scala.

Building

Building Raw Data Scala Business Intelligence

Ready-to-go sample data pipelines with Dataflow

Netflix Tech

DECEMBER 3, 2022

A large number of our data users employ SparkSQL, pyspark, and Scala. Then we’ll segue into the Scala and R use cases. Currently supported workflow RECIPEs are: spark-sql, pyspark, scala and sparklyr. scala-workflow ? ??? It solidifies different recipes or repeatable templates for data extraction. pyspark-workflow ? ???

Data Pipeline

Data Pipeline Scala Metadata Food

Learn how to use PySpark in under 5 minutes (Installation + Tutorial)

KDnuggets

AUGUST 13, 2019

Apache Spark is one of the hottest and largest open source project in data processing framework with rich high-level APIs for the programming languages like Scala, Python, Java and R. It realizes the potential of bringing together both Big Data and machine learning.

Scala

Scala Programming Language Java Big Data

Functional Parallel Programming with Scala and Cats Effect

Rock the JVM

MARCH 17, 2024

Unlock the full potential of functional parallel processing: A hands-on guide to accelerating performance with Scala and Cats Effect fibers

Scala

Scala Programming Process

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. by Herbert Kateu Hey, it’s Daniel here. val scala3Version = "3.2.2"

Scala

Scala Coding Accessible Accessibility

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. by Herbert Kateu Hey, it’s Daniel here. val scala3Version = "3.2.2"

Scala

Scala Coding Accessible Accessibility

Open-sourcing Polynote: an IDE-inspired polyglot notebook

Netflix Tech

OCTOBER 23, 2019

Jeremy Smith , Jonathan Indig , Faisal Siddiqi We are pleased to announce the open-source launch of Polynote : a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more. which makes heavy use of Scala?—?with

Scala

Scala Machine Learning Python Coding

Modern Data Engineering: Free Spark to Snowpark Migration Accelerator for Faster, Cheaper Pipelines in Snowflake

Snowflake

JUNE 20, 2024

The assessment is built by scanning any codebase written in Python or Scala and outputting a readiness score for conversion to Snowpark. As a result, the tool can take in both code files and notebooks with multiple languages (such as Scala, Python and SQL) at the same time.

Data Engineering

Data Engineering Data Engineer Scala Engineering

REST APIs Using Play Framework and Scala: A Comprehensive Guide

Rock the JVM

SEPTEMBER 3, 2023

Play Framework “makes it easy to build web applications with Java & Scala”, as it is stated on their site, and it’s true. In this article we will try to develop a basic skeleton for a REST API using Play and Scala. PlayScala plugin defines default settings for Scala-based applications. import Keys._

Scala

Scala Database Project Coding

Building a Full-Stack Scala 3 Application with the Typelevel Stack

Rock the JVM

JANUARY 22, 2024

Unlock the secrets to crafting a full-stack Scala 3 application from scratch: dive into Cats Effect, doobie, http4s, and Tyrian and build robust, modern software with ease

Scala

Scala Building

Intersection and Union types with Java and Scala by Magnus Smith

Scott Logic

MARCH 4, 2025

Previous posts have looked at Algebraic Data Types with Java Variance, Phantom and Existential types in Java and Scala Intersection and Union Types with Java and Scala One of the difficult things for modern programming languages to get right is around providing flexibility when it comes to expressing complex relationships.

Scala

Scala Java Systems Coding

Scala as a Junior Developer

Useful classes for data engineers - Scala & Java

Webinars

Trending Sources

A Functional Load Balancer with Scala, Http4s and Cats Effect

Webinars

WebSockets in Scala, Part 2: Integrating Redis and PostgreSQL

dbt on Databricks.

Guide to Scala 3 Macros

A Detailed Guide of Interview Questions on Apache Kafka

Getting Started with Scala and Apache Kafka

A guide to UDP in Scala with FS2

Two-Factor Authentication in Scala with Http4s

Scala In Demand Technologies Built On Scala

Arbitrary stateful processing in PySpark with applyInPandasWithState

gRPC in Scala with Fs2 and Scalapb

Scala 3 Inlines Explained

Making applyInPandasWithState less painful

Unity Catalog Lakeguard: Industry-first and only data governance for multi-user Apache™ Spark clusters

Adopting Spark Connect

A Comprehensive Guide to Choosing the Best Scala Course

How to Tune RocksDB for Your Kafka Streams Application

OAuth Authentication in Scala with Http4s

PySpark and vectorized User-Defined Functions

Scala Vs Python Vs R Vs Java - Which language is better for Spark & Why?

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

WebSockets in Scala: Part 2 - Integrating Redis and PostgreSQL

A Distributed Code Execution Engine in Pekko with Scala

The Dog Days of PySpark

WebSockets in Scala: Part 1 - http4s

WebSockets in Scala: Part 1 - http4s

WebSockets in Scala: Part 1 - http4s

UDP with Scala and FS2 Explained

UDP with Scala and FS2 Explained

How to Write a Full-Stack Scala 3 Application with the Typelevel Stack

Building ETL Pipeline with Snowpark

Ready-to-go sample data pipelines with Dataflow

Learn how to use PySpark in under 5 minutes (Installation + Tutorial)

Functional Parallel Programming with Scala and Cats Effect

HTTP Authentication with Scala and Http4s

HTTP Authentication with Scala and Http4s

Open-sourcing Polynote: an IDE-inspired polyglot notebook

Modern Data Engineering: Free Spark to Snowpark Migration Accelerator for Faster, Cheaper Pipelines in Snowflake

REST APIs Using Play Framework and Scala: A Comprehensive Guide

Building a Full-Stack Scala 3 Application with the Typelevel Stack

Intersection and Union types with Java and Scala by Magnus Smith

Stay Connected