Project and Scala - Data Engineering Digest

Scala as a Junior Developer

Rock the JVM

SEPTEMBER 17, 2023

Lucas’ story is shared by lots of beginner Scala developers, which is why I wanted to post it here on the blog. I’ve watched thousands of developers learn Scala from scratch, and, like Lucas, they love it! If you want to learn Scala well and fast, take a look at my Scala Essentials course at Rock the JVM. sum > 8 ).

Scala

Scala Programming Coding Java

Useful classes for data engineers - Scala & Java

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. We all have our habits and as programmers, libraries and frameworks are definitely a part of the group. The part for Python will follow next week!

Scala

Scala Java Data Engineering Data Engineer

WebSockets in Scala, Part 2: Integrating Redis and PostgreSQL

Rock the JVM

MAY 22, 2024

lazy val root = ( project in file ( "." )). Next, we’ll create a user.scala file in the following path, src/main/scala/rockthejvm/websockets/domain. Create a codecs.scala file in the following path, src/main/scala/rockthejvm/websockets/codecs/codecs.scala and add the following code: package rockthejvm.websockets.codecs import skunk.

PostgreSQL

PostgreSQL Scala Database SQL

Webinars

Agent Tooling: Connecting AI to Your Tools, Systems & Data

How to Modernize Manufacturing Without Losing Control

Mastering Apache Airflow® 3.0: What’s New (and What’s Next) for Data Orchestration

MORE WEBINARS

Scala 3: General Type Projections

Rock the JVM

MAY 16, 2022

Scala's general type projections are considered unsound and were removed in Scala 3: discover what this means and how it affects your code

Scala

Scala Project Coding IT

Two-Factor Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

If you want to master the Typelevel Scala libraries (including Http4s) with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. HOTP scala implementation HOTP generation is quite tedious, therefore for simplicity, we will use a java library, otp-java by Bastiaan Jansen.

Scala

Scala Java Bytes Algorithm

A Scala Project with Akka, Cats, and Cassandra

Rock the JVM

MARCH 31, 2022

Akka, Cats, and Cassandra in a larger Scala project integrating multiple pieces of the Scala ecosystem

Scala

Scala Project

A guide to UDP in Scala with FS2

Rock the JVM

DECEMBER 17, 2023

Setting Up Let’s create a new Scala 3 project and add the following to your build.sbt file. lazy val root = project. Setting Up Let’s create a new Scala 3 project and add the following to your build.sbt file. lazy val root = project. val scala3Version = "3.3.1" in ( file ( "." )). val scala3Version = "3.3.1"

Scala

Scala Bytes Java Coding

Scala In Demand Technologies Built On Scala

Knowledge Hut

MAY 20, 2024

The term Scala originated from “Scalable language” and it means that Scala grows with you. In recent times, Scala has attracted developers because it has enabled them to deliver things faster with fewer codes. Developers are now much more interested in having Scala training to excel in the big data field.

Scala

Scala Technology Kafka Hadoop

gRPC in Scala with Fs2 and Scalapb

Rock the JVM

OCTOBER 1, 2023

Scala is not officially supported at the moment however the ScalaPB library provides a good wrapper around the official gRPC Java implementation, it provides an easy API that enables translation of Protocol Buffers to Scala case classes with support for Scala3, Scala.js, and Java Interoperability. lazy val protobuf = project.

Scala

Scala Metadata Transportation Java

Ready-to-go sample data pipelines with Dataflow

Netflix Tech

DECEMBER 3, 2022

project Manage a Dataflow project. The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. A large number of our data users employ SparkSQL, pyspark, and Scala. scala-workflow ? ??? tox.ini ???

Data Pipeline

Data Pipeline Scala Metadata Food

Adopting Spark Connect

Towards Data Science

NOVEMBER 6, 2024

However, this ability to remotely run client applications written in any supported language (Scala, Python) appeared only in Spark 3.4. In any case, all client applications use the same Scala code to initialize SparkSession, which operates depending on the run mode. getOrCreate() // If the client application uses your Scala code (e.g.,

Scala

Scala Java AWS Coding

Learn how to use PySpark in under 5 minutes (Installation + Tutorial)

KDnuggets

AUGUST 13, 2019

Apache Spark is one of the hottest and largest open source project in data processing framework with rich high-level APIs for the programming languages like Scala, Python, Java and R. It realizes the potential of bringing together both Big Data and machine learning.

Scala

Scala Programming Language Java Big Data

Scala 3: Mastering Path-Dependent Types and Type Projections

Rock the JVM

NOVEMBER 26, 2020

Unlock advanced programming techniques with Scala 3: explore dependent types, methods, and functions in this concise tutorial

Scala

Scala Project Programming

Scala CLI Tutorial: Creating a CLI Sudoku Solver

Rock the JVM

JANUARY 8, 2023

Antonio is an alumnus of Rock the JVM, now a senior Scala developer with his own contributions to Scala libraries and junior devs under his mentorship. Which brings us to this article: Antonio originally started from my Sudoku backtracking article and built a Scala CLI tutorial for the juniors he’s mentoring.

Scala

Scala Java Algorithm Utilities

Top 12 Data Engineering Project Ideas [With Source Code]

Knowledge Hut

JUNE 26, 2023

Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article. Best Data Science certifications online or offline are available to assist you in establishing a solid foundation for every end-to-end data engineering project.

Data Engineering

Data Engineering Data Engineer Coding Project

A Comprehensive Guide to Choosing the Best Scala Course

Rock the JVM

MAY 22, 2023

This article is all about choosing the right Scala course for your journey. How should I get started with Scala? Do you have any tips to learn Scala quickly? How to Learn Scala as a Beginner Scala is not necessarily aimed at first-time programmers. Which course should I take?

Scala

Scala Java Programming Language Programming

Getting Started with Scala sbt

Rock the JVM

AUGUST 28, 2022

Discover sbt: The popular Scala build tool that simplifies project management and enhances productivity

Scala

Scala Project Building Management

Scala Vs Python Vs R Vs Java - Which language is better for Spark & Why?

Knowledge Hut

MAY 3, 2024

If you search top and highly effective programming languages for Big Data on Google, you will find the following top 4 programming languages: Java Scala Python R Java Java is one of the oldest languages of all 4 programming languages listed here. Scala is a highly Scalable Language. Scala is the native language of Spark.

Scala

Scala Java Python Programming Language

How to Write a Full-Stack Scala 3 Application with the Typelevel Stack

Rock the JVM

JANUARY 22, 2024

Introduction The Typelevel stack is one of the most powerful sets of libraries in the Scala ecosystem. They allow you to write powerful applications with pure functional programming - as of this writing, the Typelevel ecosystem is one of the biggest selling points of Scala. lazy val app = ( project in file ( "app" )).

Scala

Scala SQL Database Coding

Scala For Big Data Engineering – Why should you care?

Advancing Analytics: Data Engineering

APRIL 23, 2020

The thought of learning Scala fills many with fear, its very name often causes feelings of terror. The truth is Scala can be used for many things; from a simple web application to complex ML (Machine Learning). The name Scala stands for “scalable language.” So what companies are actually using Scala?

Scala

Scala Big Data Data Engineering Data Engineer

OAuth Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

map ( _ % CirceVersion ) lazy val oauth = project. SNAPSHOT" , scalaVersion := scala3Version , libraryDependencies ++= Seq ( emberServer , emberClient , http4sDsl , http4sCirce , ciris , cirisCirce ) ++ circeLibs ) We use Scala 3 in this tutorial, if you need to use Scala 2.13, you can do so with minimal code changes.

Scala

Scala Coding Accessibility Accessible

Databricks, Snowflake and the future

Christophe Blefari

JUNE 21, 2024

billion—Databricks figures are not public and are therefore projected. you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with The project became a top-level Apache project in Nov 2018. The conferences were expecting 20,000 and 16,000 participants respectively. 3) Spark 4.0

Metadata

Metadata Data Warehouse BI MySQL

REST APIs Using Play Framework and Scala: A Comprehensive Guide

Rock the JVM

SEPTEMBER 3, 2023

Play Framework “makes it easy to build web applications with Java & Scala”, as it is stated on their site, and it’s true. In this article we will try to develop a basic skeleton for a REST API using Play and Scala. PlayScala plugin defines default settings for Scala-based applications. import Keys._ getOrElse ( 0L ), carDTO.

Scala

Scala Database Project Coding

Data pipeline asset management with Dataflow

Netflix Tech

FEBRUARY 9, 2022

It could be a JAR compiled from Scala, a Python script or module, or a simple SQL file. In Figure 1, you can see an illustration of a typical deployment pipeline manually constructed by a user for an individual project. scala-workflow ??? Gradle Assets: -> /scala-workflow/build.gradle Summary: 1 found. setup.py ???

Data Pipeline

Data Pipeline Management Scala Python

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

MongoDB

MongoDB MySQL Scala Machine Learning

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. You’re reading a big article about the Http4s library.

Scala

Scala Coding Accessibility Accessible

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. You’re reading a big article about the Http4s library.

Scala

Scala Coding Accessibility Accessible

Achieving 3.2x Faster Scala Compile Time

Zalando Engineering

APRIL 18, 2017

Toward the end of 2016, the Glitch team experienced a steady increase in compilation time on their project. The backend of Quala is called “tinbox” and is written in Scala , using many type-intensive libraries such as Shapeless , Circe , Grafter , and http4s/rho. Does the product come with the right washing machine instructions?

Scala

Scala Coding Project Java

Building a Machine Learning Application With Cloudera Data Science Workbench And Operational Database, Part 1: The Set-Up & Basics

Cloudera

JANUARY 6, 2021

For data professionals that want to make use of data stored in HBase the recent upstream project “hbase-connectors” can be used with PySpark for basic operations. 2) Make a new Project in CDSW and use a PySpark template. 3) Open the Project, go to Settings -> Engine -> Environment Variables. Restart Region Servers.

Machine Learning

Machine Learning Data Science Database Building

Discover And De-Clutter Your Unstructured Data With Aparavi

Data Engineering Podcast

JUNE 12, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Unstructured Data

Unstructured Data MongoDB MySQL Scala

The Future of Java: Top Trends and Technologies

Knowledge Hut

JULY 7, 2023

DevOps DevOps, or Development Operations, is a web development framework that allows programmers, designers, and other team members to collaborate and share resources on a single platform when working on a web development project. The reusability of Java objects makes it a versatile tool for use across projects.

Java

Java Technology Programming Language Scala

How Software Bill of Materials change the dependency game

Zalando Engineering

APRIL 12, 2023

Some teams use tools like dependabot , scala-steward that create pull requests in repositories when new library versions are available. Some projects, like openssl, preannounce security updates allowing for more preparation time. Dependency hygiene Dependency updates are a tedious task when maintaining thousands of microservices.

Java

Java Scala Python Metadata

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Data Ingestion MongoDB MySQL

Using Notebooks As The Unifying Layer For Data Roles At Netflix with Matthew Seal - Episode 54

Data Engineering Podcast

OCTOBER 28, 2018

What are some of the projects that are ongoing or planned for the future that you are most excited by? What are some of the projects that are ongoing or planned for the future that you are most excited by? What are some of the limitations of the notebook environment for the work that you are doing?

Scala

Scala Python Data Engineering Data Engineer

Package Management And Distribution For Your Data Using Quilt with Kevin Moore - Episode 37

Data Engineering Podcast

JUNE 24, 2018

Summary Collaboration, distribution, and installation of software projects is largely a solved problem, but the same cannot be said of data. In this episode he explains how the project came to be, how it works, and the many ways that you can start using it today. Can you step through a typical workflow of someone using Quilt?

Management

Management PostgreSQL Scala Programming Language

Intersection and Union types with Java and Scala by Magnus Smith

Scott Logic

MARCH 4, 2025

Previous posts have looked at Algebraic Data Types with Java Variance, Phantom and Existential types in Java and Scala Intersection and Union Types with Java and Scala One of the difficult things for modern programming languages to get right is around providing flexibility when it comes to expressing complex relationships.

Scala

Scala Java Systems Coding

Level Up Your Data Platform With Active Metadata

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata MongoDB MySQL Scala

Finagle Tutorial: Twitter’s RPC Library for Scala

Rock the JVM

NOVEMBER 27, 2022

This article is for Scala developers of all levels - you don’t need any fancy Scala knowledge to make the best out of this piece. For those who don’t know yet, almost the entire Twitter backend runs on Scala, and the Finagle library is at the core of almost all Twitter services. serve ( ":9090" , stringLengthService ) Await.

Scala

Scala Algorithm Systems Building

Why We Do Scala in Zalando

Zalando Engineering

JANUARY 8, 2018

Leveraging the full power of a functional programming language In Zalando Dublin, you will find that most engineering teams are writing their applications using Scala. We will try to explain why that is the case and the reasons we love Scala. How I came to use Scala I have been working with JVM for the last 18 years.

Scala

Scala Bytes Java Programming

Fundamentals of Apache Spark

Knowledge Hut

MAY 3, 2024

Spark offers over 80 high-level operators that make it easy to build parallel apps and one can use it interactively from the Scala, Python, R, and SQL shells. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development. Version 2.3

Hadoop

Hadoop Scala Healthcare Big Data

Apache Spark vs MapReduce: A Detailed Comparison

Knowledge Hut

MAY 2, 2024

Also, there is no interactive mode available in MapReduce Spark has APIs in Scala, Java, Python, and R for all basic transformations and actions. It also supports multiple languages and has APIs for Java, Scala, Python, and R. The Pig has SQL-like syntax and it is easier for SQL developers to get on board easily.

Hadoop

Hadoop Scala Datasets Java

Metabase Self Service Business Intelligence with Sameer Al-Sakran - Episode 29

Data Engineering Podcast

APRIL 29, 2018

In this episode the CEO of Metabase, Sameer Al-Sakran, discusses how and why the project got started, the ways that it can be used to build and share useful reports, some of the useful features planned for future releases, and how to get it set up to start using it in your environment.

Business Intelligence

Business Intelligence Scala Hadoop Machine Learning

Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57

Data Engineering Podcast

NOVEMBER 18, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. What are some of the primary ways that Flink is used?

Process

Process Google Cloud Scala Kafka

Top Hadoop Projects and Spark Projects for Beginners 2021

ProjectPro

NOVEMBER 14, 2015

Apache Hadoop and Apache Spark fulfill this need as is quite evident from the various projects that these two frameworks are getting better at faster data storage and analysis. These Apache Hadoop projects are mostly into migration, integration, scalability, data analytics, and streaming analysis. Data Integration 3.Scalability

Hadoop

Hadoop Project Big Data Healthcare

Scala as a Junior Developer

Useful classes for data engineers - Scala & Java

Webinars

Trending Sources

WebSockets in Scala, Part 2: Integrating Redis and PostgreSQL

Webinars

Scala 3: General Type Projections

Two-Factor Authentication in Scala with Http4s

A Scala Project with Akka, Cats, and Cassandra

A guide to UDP in Scala with FS2

Scala In Demand Technologies Built On Scala

gRPC in Scala with Fs2 and Scalapb

Ready-to-go sample data pipelines with Dataflow

Adopting Spark Connect

Learn how to use PySpark in under 5 minutes (Installation + Tutorial)

Scala 3: Mastering Path-Dependent Types and Type Projections

Scala CLI Tutorial: Creating a CLI Sudoku Solver

Top 12 Data Engineering Project Ideas [With Source Code]

A Comprehensive Guide to Choosing the Best Scala Course

Getting Started with Scala sbt

Scala Vs Python Vs R Vs Java - Which language is better for Spark & Why?

How to Write a Full-Stack Scala 3 Application with the Typelevel Stack

Scala For Big Data Engineering – Why should you care?

OAuth Authentication in Scala with Http4s

Databricks, Snowflake and the future

REST APIs Using Play Framework and Scala: A Comprehensive Guide

Data pipeline asset management with Dataflow

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

HTTP Authentication with Scala and Http4s

HTTP Authentication with Scala and Http4s

Achieving 3.2x Faster Scala Compile Time

Building a Machine Learning Application With Cloudera Data Science Workbench And Operational Database, Part 1: The Set-Up & Basics

Discover And De-Clutter Your Unstructured Data With Aparavi

The Future of Java: Top Trends and Technologies

How Software Bill of Materials change the dependency game

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

Using Notebooks As The Unifying Layer For Data Roles At Netflix with Matthew Seal - Episode 54

Package Management And Distribution For Your Data Using Quilt with Kevin Moore - Episode 37

Intersection and Union types with Java and Scala by Magnus Smith

Level Up Your Data Platform With Active Metadata

Finagle Tutorial: Twitter’s RPC Library for Scala

Why We Do Scala in Zalando

Fundamentals of Apache Spark

Apache Spark vs MapReduce: A Detailed Comparison

Metabase Self Service Business Intelligence with Sameer Al-Sakran - Episode 29

Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57

Top Hadoop Projects and Spark Projects for Beginners 2021

Stay Connected