Advancing Analytics: Data Engineering

APRIL 23, 2020

The thought of learning Scala fills many with fear, its very name often causes feelings of terror. The truth is Scala can be used for many things; from a simple web application to complex ML (Machine Learning). The name Scala stands for “scalable language.” So what companies are actually using Scala?

Scala 52

Scala Big Data Data Engineering Data Engineer 52

Zalando Engineering

MAY 23, 2016

It’s no secret that Zalando Tech has had its hands full lately with its participation in several Scala conferences and meetups. As a company who practices Radical Agility , our use of Scala has skyrocketed and it’s now one of our most adopted programming languages amongst developers. So, where have we been in the Scala world?

Scala 40

Scala Programming Language Technology Programming 40

Knowledge Hut

MAY 2, 2024

Most cutting-edge technology organizations like Netflix, Apple, Facebook, and Uber have massive Spark clusters for data processing and analytics. Both technologies have their own pros and cons as we will see below. Both these technologies have made inroads in all walks of common man’s life. It can also run on YARN or Mesos.

Hadoop 96

Hadoop Scala Datasets Java 96

Monte Carlo

OCTOBER 31, 2024

In addition, AI data engineers should be familiar with programming languages such as Python , Java, Scala, and more for data pipeline, data lineage, and AI model development.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Rock the JVM

SEPTEMBER 3, 2023

Play Framework “makes it easy to build web applications with Java & Scala”, as it is stated on their site, and it’s true. In this article we will try to develop a basic skeleton for a REST API using Play and Scala. PlayScala plugin defines default settings for Scala-based applications. import Keys._

Scala 52

Scala Database Project Coding 52

Knowledge Hut

MAY 3, 2024

Spark offers over 80 high-level operators that make it easy to build parallel apps and one can use it interactively from the Scala, Python, R, and SQL shells. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development. Basic knowledge of SQL.

Hadoop 98

Hadoop Scala Healthcare Big Data 98

Christophe Blefari

JUNE 21, 2024

you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with Here what Databricks brought this year: Spark 4.0 — (1) PySpark erases the differences with the Scala version, creating a first class experience for Python users. (2) Databricks sells a toolbox, you don't buy any UX. 3) Spark 4.0

Metadata 147

Metadata Data Warehouse BI MySQL 147

Data Engineering Podcast

JUNE 12, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Unstructured Data 130

Unstructured Data MongoDB MySQL Scala 130

Data Engineering Podcast

APRIL 29, 2018

Contact Info Sameer salsakran on GitHub @sameer_alsakran on Twitter LinkedIn Metabase Website @metabase on Twitter metabase on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Business Intelligence 100

Business Intelligence Scala Hadoop Machine Learning 100

Data Engineering Podcast

OCTOBER 28, 2018

Contact Info Matthew Seal Email LinkedIn @codeseal on Twitter MSeal on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Scala 100

Scala Python Data Engineer Data Engineering 100

Data Engineering Podcast

NOVEMBER 6, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Data Engineering Podcast

NOVEMBER 20, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info LinkedIn @SalmaBakouk on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Snowflake

DECEMBER 20, 2023

To expand the capabilities of the Snowflake engine beyond SQL-based workloads, Snowflake launched Snowpark , which added support for Python, Java and Scala inside virtual warehouse compute.

Scala 142

Scala Government Java Cloud 142

Knowledge Hut

MAY 3, 2024

Dean Wampler (Renowned author of many big data technology-related books) Dean Wampler makes an important point in one of his webinars. And hence, there is a need to understand the concept of “stream processing “and the technology behind it. cache, local space) 8 It supports multiple languages such as Java, Scala, R, and Python.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Snowflake

NOVEMBER 29, 2023

Data cloud technology can accelerate FAIRification of the world’s biomedical patient data. Next-generation sequencing (NGS) technology has dramatically dropped the price of genomic sequencing, from about $1 million in 2007 to $600 today per whole genome sequencing (WGS).

Metadata 117

Metadata Healthcare Medical Data Storage 117

Data Engineering Podcast

DECEMBER 9, 2018

What are some of the common ways that Spark is deployed, in terms of the cluster topology and the supporting technologies? Contact Info @jgperrin on Twitter Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? What are the cases where Spark is the wrong choice?

MySQL 100

MySQL Scala Kafka Hadoop 100

Data Engineering Podcast

JUNE 19, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info LinkedIn @prukalpa on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Metadata 130

Metadata MongoDB MySQL Scala 130

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Use cases change, needs change, technology changes – and therefore data infrastructure should be able to scale and evolve with change.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Data Engineering Podcast

MAY 27, 2018

Contact Info LinkedIn @yairwein on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info LinkedIn @yairwein on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Pipeline 100

Data Pipeline MongoDB Google Cloud Scala 100

Data Engineering Podcast

JUNE 17, 2018

Contact Info @danlovesproofs on twitter dan@drob.us @drob on github heapanalytics.com / @heap on twitter [link] Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Links Heap Palantir User Analytics Google Analytics Piwik Mixpanel Hubspot Jepsen Chaos Engineering Node.js

Scala 100

Scala Kafka SQL Architecture 100

Data Engineering Podcast

JUNE 24, 2018

Contact Info Email LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info Email LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Management 100

Management PostgreSQL Scala Programming Language 100

Data Engineering Podcast

NOVEMBER 18, 2018

Contact Info LinkedIn @fhueske on Twitter fhueske on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Process 100

Process Google Cloud Scala Kafka 100

Data Engineering Podcast

MARCH 1, 2021

In this episode Yaron Haviv, co-founder of Iguazio, discusses the complexities inherent to the process, as well as how he has worked to democratize the technologies necessary to make machine learning operations maintainable.

Machine Learning 100

Machine Learning Data Warehouse Scala Hadoop 100

Data Engineering Podcast

SEPTEMBER 9, 2018

Contact Info Pete Cheslock @petecheslock on Twitter Website Thomas Hazel @thomashazel on Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

IT 100

IT PostgreSQL Scala AWS 100

Confluent

JUNE 7, 2019

Interestingly, blockchain technology is reducing the need for trust, but doesn’t make it obsolete. To bridge the gap between different Ethereum clients and Kafka, we developed an in-house solution named Ethsync, written in Scala, which allows us to propagate the data in a reliable at-least-once manner.

Accessible 94

Accessible Accessibility Kafka Scala 94

Christophe Blefari

JANUARY 14, 2023

The history repeat, we've seen it with Scala, Go or even Julia at some scale. One of the most interesting advice he gives that I can press is: you should spend time mastering the technologies you've chosen. Enjoy the Data News. In the end Python and SQL are still here for good. But with Rust the approach is different.

Python 130

Python Kafka Data Scala 130

Snowflake

JULY 13, 2023

Minimizing total cost of ownership (TCO) To minimize TCO by saving both time and manual effort, Data Platform Technology Lead Andy Brown realized he would have to replace both of ESO’s legacy data platforms with Snowflake. ESO’s data analytics platform was previously based on Cloudera running Scala and Spark.

Medical 98

Medical Hospitality Transportation Scala 98

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Snowflake

MARCH 29, 2023

show() And you’re not limited to only SQL—you can also query using DataFrames with other languages like Python and Scala. First, let’s see what tables are available to query. spark.sql("SELECT * FROM my_database.my_schema.iceberg_customer LIMIT 10").show()

Metadata 105

Metadata Scala Hadoop Relational Database 105

Cloudera

MARCH 2, 2021

This data engineering skillset typically consists of Java or Scala programming skills mated with deep DevOps acumen. But as data streaming technologies like Apache Kafka and Apache Flink have evolved, only until recently have SQL interfaces become deeply integrated. A rare breed.

SQL 112

SQL Java Data Lake Scala 112

Data Engineering Podcast

SEPTEMBER 11, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info LinkedIn @ManishJethani on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Data Engineering Podcast

AUGUST 21, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info LinkedIn @shrutibhat on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. What do you have planned for the future of the platform and business?

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

Data Engineering Podcast

JULY 31, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Contact Info Website LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Analysis 100

Data Analysis MongoDB Algorithm MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Engineering 100

Engineering MongoDB MySQL Scala 100

Data Engineering Podcast

MAY 22, 2022

Contact Info LinkedIn @_raj_bains on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info LinkedIn @_raj_bains on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Scala 100

Scala SQL Data Data Engineer 100

Pinterest Engineering

FEBRUARY 22, 2024

As with any new technology, the adoption of AI-assisted development comes with its fair share of risks and concerns. and ‘“Copilot was particularly useful when I had to make a change in Scala, a language I am not familiar with.

Scala 98

Scala Engineering Software Engineer Software Engineering 98