Rock the JVM

JULY 26, 2023

If you want to master the Typelevel Scala libraries (including Http4s) with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. HOTP scala implementation HOTP generation is quite tedious, therefore for simplicity, we will use a java library, otp-java by Bastiaan Jansen.

Scala 92

Scala Bytes Java Algorithm 92

ProjectPro

JUNE 6, 2025

The distributed execution engine in the Spark core provides APIs in Java, Python, and Scala for constructing distributed ETL applications. The following are the persistence levels available in Spark: MEMORY ONLY: This is the default persistence level, and it's used to save RDDs on the JVM as deserialized Java objects.

Hadoop 68

Hadoop Metadata Java Datasets 68

Cloudyard

DECEMBER 24, 2024

Snowflakes Snowpark is a game-changing feature that enables data engineers and analysts to write scalable data transformation workflows directly within Snowflake using Python, Java, or Scala. RAW_CUSTOMERS : Stores customer information. The data resides in three tables: RAW_ORDERS : Captures order details.

Building 52

Building Raw Data Scala Java 52

ProjectPro

JUNE 6, 2025

Python, Java, and Scala knowledge are essential for Apache Spark developers. Various high-level programming languages, including Python, Java , R, and Scala, can be used with Spark, so you must be proficient with at least one or two of them. Creating Spark/Scala jobs to aggregate and transform data.

Scala 40

Scala Programming Language Java Hadoop 40

ProjectPro

JUNE 6, 2025

You will find data engineers using this to extract information from websites, dealing with JSON/HTML data formats, all for preparing their data. But, for those who still are not entirely confident about learning this programming language and want to know if there are any other choices, here are two for you: Java and Scala.

Data Engineer 40

Data Engineer Data Engineering Python Engineering 40

ProjectPro

JUNE 6, 2025

A lack of access to real-time information will result in billions of dollars in lost revenue. Apache Spark Streaming Use Cases Spark Streaming Architecture: Discretized Streams Spark Streaming Example in Java Spark Streaming vs. Structured Streaming Spark Streaming Structured Streaming What is Kafka Streaming?

Architecture 45

Architecture Kafka Java Scala 45

ProjectPro

JUNE 6, 2025

You can work in any sector, including finance, manufacturing, information technology, telecommunications, retail, logistics, and automotive. SQL, Data Warehousing/Data Processing, and Database Knowledge: This includes SQL knowledge to query data and manipulate information stored in databases.

Big Data 45

Big Data Big Data Skills Scala Hadoop 45

ProjectPro

JUNE 6, 2025

Every piece of information generated – be it from social media interactions, online purchases, sensor data, or any digital activity – is a potential nugget of gold because it’s rich with opportunities. They uncover valuable insights, patterns, and trends within these datasets, which can inform critical business decisions.

Big Data 40

Big Data Hadoop Scala NoSQL 40

Rock the JVM

DECEMBER 17, 2023

It works by bundling up data in a UDP packet, adding header information, and sending these packets to the target destination. In this article, we will first understand how to implement UDP with Java NIO and gradually transition to Fs2’s io library which provides binding for UDP networking. val scala3Version = "3.3.1" Stream import fs2.io.net.Network

Scala 90

Scala Bytes Java Coding 90

ProjectPro

JUNE 6, 2025

Time Travel The Delta lake transaction log has information about every change made to the data in the order of execution. Databricks also provides extensive delta lake API documentation in Python, Scala , and SQL to get started on delta lake quickly. Delta lake APIs exist for Python, Scala , Java, and SQL.

Data Lake 63

Data Lake Data Warehouse Metadata Unstructured Data 63

Scott Logic

MARCH 30, 2025

Previous posts have looked at Algebraic Data Types with Java Variance, Phantom and Existential types in Java and Scala Intersection and Union Types with Java and Scala In this post we will combine some ideas from functional programming with strong typing to produce robust expressive code that is more reusable.

Scala 40

Scala Java Coding Systems 40

Zalando Engineering

APRIL 12, 2023

Some teams use tools like dependabot , scala-steward that create pull requests in repositories when new library versions are available. The Software Bill of Materials contains information about the packages and libraries used by an application. next biggest app) and for Java it's tableau (3.14x next biggest app). What are SBOMs?

Java 98

Java Scala Metadata Python 98

Christophe Blefari

JUNE 21, 2024

you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with Here what Databricks brought this year: Spark 4.0 — (1) PySpark erases the differences with the Scala version, creating a first class experience for Python users. (2) Databricks sells a toolbox, you don't buy any UX. 3) Spark 4.0

Metadata 147

Metadata Data Warehouse BI Scala 147

ProjectPro

JUNE 6, 2025

Transport for London, on the other hand, uses statistical data to map passenger journeys, manage unforeseen scenarios, and provide passengers with customized transportation information. Data Architect - Key Skills Solid understanding of programming languages like Java, Python, R, or SQL. A solid grasp of natural language processing.

Data Science 56

Data Science Data Mining Data Architect BI 56

Data Engineering Podcast

JUNE 12, 2022

Another category of unstructured data that every business deals with is PDFs, Word documents, workstation backups, and countless other types of information. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

ProjectPro

JUNE 6, 2025

For example, C, C++, Go, Java, Node, Python, Rust, Scala , Swift, etc. Beginner Level MongoDB Project to Develop a Football Statistics App Image source: www.mongodb.com/developer/code-examples In this mongodb project, you will develop a prototype for a Football statistics app that stores information about Football player profiles.

MongoDB 66

MongoDB Coding Project NoSQL 66

Cloudera

MARCH 2, 2021

This data engineering skillset typically consists of Java or Scala programming skills mated with deep DevOps acumen. It’s also worth noting that even those with Java skills will often prefer to work with SQL – if for no other reason than to share the workload with others in their organization that only know SQL.

SQL 112

SQL Java Data Lake Scala 112

Snowflake

OCTOBER 30, 2023

Enter the new Event Tables feature, which helps developers and data engineers easily instrument their code to capture and analyze logs and traces for all languages: Java, Scala, JavaScript, Python and Snowflake Scripting. For further information about how Event Tables work, visit Snowflake product documentation.

Java 111

Java Scala Hadoop Data Ingestion 111

Rock the JVM

JANUARY 8, 2023

Antonio is an alumnus of Rock the JVM, now a senior Scala developer with his own contributions to Scala libraries and junior devs under his mentorship. Which brings us to this article: Antonio originally started from my Sudoku backtracking article and built a Scala CLI tutorial for the juniors he’s mentoring.

Scala 52

Scala Java Algorithm Utilities 52

ProjectPro

JUNE 6, 2025

Check out this career guide for the most up-to-date information about the role, skills, education, salary, and possible employment information to get you started in this exciting field. Are you interested in becoming a data architect? Machine Learning Architects build scalable systems for use with AI/ML models.

Data Architect 40

Data Architect Data Mining Programming Language Java 40

ProjectPro

JUNE 6, 2025

Data engineering is a critical function in modern organizations, as it allows companies to extract insights from large volumes of data and make informed decisions. A data warehouse allows stakeholders to make well-informed business decisions by supporting the process of drawing meaningful conclusions through data analytics.

Data Engineer 49

Data Engineer Data Engineering Engineering Hadoop 49

Monte Carlo

OCTOBER 31, 2024

In addition, AI data engineers should be familiar with programming languages such as Python , Java, Scala, and more for data pipeline, data lineage, and AI model development. This can come with tedious checks on secure information like PII, extra layers of security, and more meetings with the legal team.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Knowledge Hut

JANUARY 18, 2024

Data scientists are thought leaders who apply their expertise in statistics and machine learning to extract useful information from data. It is a declarative language for interacting with databases and allows you to create queries to extract information from your data sets. There are many languages required for data science.

Programming Language 98

Programming Language Data Science Programming Scala 98

Zalando Engineering

NOVEMBER 19, 2023

And even if you were to gather all the information from the docs, it's still not enough. It provides a powerful information retrieval language and engine that integrates several microservice components built by the Search Department. However: It’s in Java. Upgrading the Elasticsearch API to be able to work with version 8.x

Scala 83

Scala Java Coding IT 83

Data Engineering Podcast

NOVEMBER 6, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. How has that informed your efforts in the development and release of the project? How has that informed your efforts in the development and release of the project?

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems.

Metadata 130

Metadata MongoDB Scala MySQL 130

ProjectPro

JUNE 6, 2025

An AWS data engineer, for example, is in charge of preserving data integrity and building data models to collect information from various sources. Using Java, Python, and Scala , design and construct production data pipelines from intake to consumption within a significant data architecture.

AWS 45

AWS Data Engineer Data Engineering Amazon Web Services 45

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

ProjectPro

JUNE 6, 2025

Each event encompasses valuable information through messages flowing through Kafka topics. Significance of Kafka Events Kafka Events serve as the fundamental units of data flow, allowing for the continuous exchange of information between producers and consumers. Do I need to know Java to learn Kafka?

Kafka 40

Kafka Java Big Data Data Pipeline 40

Cloudera

MAY 3, 2021

This data engineering skill set typically consists of Java or Scala programming skills mated with deep DevOps acumen. They no longer need to ask a small subset of the organization to provide them with information, rather, they have tooling, systems, and capabilities to get the data they need. A rare breed.

Scala 103

Scala Retail Java Data Lake 103

ProjectPro

JUNE 18, 2021

Why do data scientists prefer Python over Java? Java vs Python for Data Science- Which is better? Which has a better future: Python or Java in 2021? This blog aims to answer all questions on how Java vs Python compare for data science and which should be the programming language of your choice for doing data science in 2021.

Java 52

Java Data Science Python Programming Language 52

Confluent

JULY 16, 2019

For general information about how to build scalable and reliable machine learning infrastructures with Apache Kafka ecosystem, check out the article Using Apache Kafka to Drive Cutting Edge Machine Learning. Based on this information, it’s wise to ensure that at least your front door is under constant surveillance.

Kafka 101

Kafka Machine Learning Java Scala 101

ProjectPro

JUNE 6, 2025

As we step into the latter half of the present decade, we can’t help but notice the way Big Data has entered all crucial technology-powered domains such as banking and financial services, telecom, manufacturing, information technology, operations, and logistics. It is an improvement over Hadoop’s two-stage MapReduce paradigm.

Hadoop 40

Hadoop Project Big Data Scala 40

Knowledge Hut

DECEMBER 26, 2023

Most Popular Programming Certifications C & C++ Certifications Oracle Certified Associate Java Programmer OCAJP Certified Associate in Python Programming (PCAP) MongoDB Certified Developer Associate Exam R Programming Certification Oracle MySQL Database Administration Training and Certification (CMDBA) CCA Spark and Hadoop Developer 1.

Certification 98

Certification Programming MongoDB MySQL 98

Data Engineering Podcast

AUGUST 13, 2022

In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Cloudera

JULY 18, 2022

In this blog we will explore how we can use Apache Flink to get insights from data at a lightning-fast speed, and we will use Cloudera SQL Stream Builder GUI to easily create streaming jobs using only SQL language (no Java/Scala coding required). It provides flexible and expressive APIs for Java and Scala. Use case recap.

Process 91

Process Kafka Scala SQL 91

Knowledge Hut

MAY 3, 2024

Spark offers over 80 high-level operators that make it easy to build parallel apps and one can use it interactively from the Scala, Python, R, and SQL shells. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development.

Scala 98

Scala Hadoop Healthcare Big Data 98