This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Engineering Hadoop SQL 
Analytics Vidhya
FEBRUARY 1, 2023
Introduction In this constantly growing technical era, big data is at its peak, with the need for a tool to import and export the data between RDBMS and Hadoop. Apache Sqoop stands for “SQL to Hadoop,” and is one such tool that transfers data between Hadoop(HIVE, HBASE, HDFS, etc.)
222 
Christophe Blefari
MARCH 1, 2023
dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. Jinja templating — Jinja is a templating engine that seems to exist forever in Python.
278 
Christophe Blefari
JANUARY 20, 2024
Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.
130 
Jesse Anderson
DECEMBER 12, 2022
Doug Cutting took those papers and created Apache Hadoop in 2005. They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. We lacked a scalable pub/sub system.
130 
Analytics Vidhya
FEBRUARY 7, 2023
Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.
240 
Uber Engineering
JANUARY 15, 2020
As Uber’s operations became more complex and we offered additional features and … The post Engineering SQL Support on Apache Pinot at Uber appeared first on Uber Engineering Blog.
112 
Data Engineering Podcast
FEBRUARY 5, 2023
In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Look no further than Materialize, the streaming database you already know how to use.
147 
AltexSoft
JUNE 7, 2021
Hadoop and Spark are the two most popular platforms for Big Data processing. To come to the right decision, we need to divide this big question into several smaller ones — namely: What is Hadoop? To come to the right decision, we need to divide this big question into several smaller ones — namely: What is Hadoop? scalability.
Data Engineering Podcast
SEPTEMBER 7, 2020
To address this problem Martin Traverso and his colleagues at Facebook built the Presto distributed query engine. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering?
100 
Maxime Beauchemin
JANUARY 20, 2017
I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? At the same time, data engineering was the slightly younger sibling, but it was going through something similar. I wasn’t promoted or assigned to this new role.
130 
Data Engineering Podcast
MARCH 24, 2024
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.
162 
Data Engineering Podcast
JANUARY 21, 2024
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.
147 
Christophe Blefari
SEPTEMBER 28, 2023
The era of Big Data was characterised by Hadoop, HDFS, distributed computing (Spark), above the JVM. We jumped from HDFS to Cloud Storage (S3, GCS) for storage and from Hadoop, Spark to Cloud warehouses (Redshift, BigQuery, Snowflake) for processing. I often say that data engineering is boring, insanely boring.
147 
Data Engineering Weekly
MAY 26, 2024
[link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. It is refreshing to see an open stack after the Hadoop era. and why one prefer dataframe over sql.
Data Engineering Podcast
JULY 10, 2022
Summary Building and maintaining reliable data assets is the prime directive for data engineers. In order to make this a tractable problem it is essential that engineers embrace automation at every opportunity. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
100 
AltexSoft
OCTOBER 30, 2021
Was Nikola Tesla a scientist or engineer? These men didn’t stop at scientific research and ended up conceptualizing or engineering their inventions. Engineers are not only the ones bearing helmets and operating on construction sites. Data science vs data engineering. Here, data scientists are supported by data engineers.
Data Engineering Weekly
DECEMBER 15, 2024
[link] Dani: Apache Iceberg: The Hadoop of the Modern Data Stack? The comment on Iceber, a Hadoop of the modern data stack, surprises me. Iceberg has not reduced the complexity of the data stack, and all the legacy Hadoop complexity still exists on top of Apache Iceberg. However, I 100% agree with the complex stack to maintain.
Christophe Blefari
FEBRUARY 23, 2024
This week I've participated to a round table about data and did a cool presentation about Engines. The idea was to depict the history of engines over the last 40 years and what leads to polars and DuckDB. Engines evolution (me) There are 3 points that have triggered discussion about the visualisation I done What about Arrow?
130 
Data Engineering Podcast
JANUARY 30, 2022
Summary The Data Engineering Podcast has been going for five years now and has included conversations and interviews with a huge number of guests, covering a broad range of topics. In this episode he shares some reflections on producing the podcast, compiling the book, and relevant trends in the ecosystem of data engineering.
100 
Data Engineering Podcast
JANUARY 13, 2019
Introduction Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode.
100 
Christophe Blefari
JUNE 21, 2024
One way to read data platforms When we look at platforms history what characterises evolution is the separation (or not) between the engine and the storage. A UX where you buy a single tool combining engine and storage, where all you have to do is flow data in, write SQL, and it's done.
147 
AltexSoft
JULY 29, 2022
Depending on how you measure it, the answer will be 11 million newspaper pages or… just one Hadoop cluster and one tech specialist who can move 4 terabytes of textual data to a new location in 24 hours. The Hadoop toy. So the first secret to Hadoop’s success seems clear — it’s cute. What is Hadoop?
59 
U-Next
MARCH 1, 2023
Introduction Data Engineer is responsible for managing the flow of data to be used to make better business decisions. A solid understanding of relational databases and SQL language is a must-have skill, as an ability to manipulate large amounts of data effectively. In 2022, data engineering will hold a share of 29.8%
Knowledge Hut
MAY 2, 2024
But at last, 5 years or so with Apache Spark gaining more ground, demand for MapReduce as the processing engine has reduced. The Pig has SQL-like syntax and it is easier for SQL developers to get on board easily. Compatibility MapReduce is also compatible with all data sources and file formats Hadoop supports.
96 
The Pragmatic Engineer
JUNE 13, 2023
👋 Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. The majority of the engineering team is in Bangkok, Thailand. It uses Spark for the data platform.
201 
Knowledge Hut
DECEMBER 28, 2023
That's where Hadoop comes into the picture. Hadoop is a popular open-source framework that stores and processes large datasets in a distributed manner. Organizations are increasingly interested in Hadoop to gain insights and a competitive advantage from their massive datasets. Why Are Hadoop Projects So Important?
52 
Data Engineering Podcast
JANUARY 6, 2019
Summary The Hadoop platform is purpose built for processing large, slow moving data in long-running batch jobs. For a perfect pairing, they made it easy to connect to the Impala SQL engine. How does it fit into the Hadoop ecosystem? Can you start by explaining what Kudu is and the motivation for building it?
100 
Netflix Tech
JULY 15, 2021
Data Engineers of Netflix?—?Interview Interview with Kevin Wylie This post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Kevin Wylie is a Data Engineer on the Content Data Science and Engineering team.
Knowledge Hut
APRIL 25, 2024
If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.
98 Data Engineering Podcast
MAY 20, 2018
Presto is a distributed SQL engine that allows you to tie all of your information together without having to first aggregate it all into a data warehouse. For someone who is using the Presto SQL interface, what are some of the considerations that they should keep in mind to avoid writing poorly performing queries?
100 
Ripple Engineering
JULY 9, 2024
We recently embarked on a significant data platform migration, transitioning from Hadoop to Databricks, a move motivated by our relentless pursuit of excellence and our contributions to the XRP Ledger's (XRPL) data analytics. High maintenance costs and a system that struggled to meet the real-time demands of our data-driven initiatives.
96 
Christophe Blefari
OCTOBER 6, 2024
Still I hope this edition finds you well, it will be a mix of personal news, OpenAI saga and usual data engineering stuff that I enjoy reading. I really like the program we put in place, it a mix of Engineering and Strategic / Vision talks. Current state of Databricks SQL — "The best data warehouse is a lakehouse", lmao.
130 
Knowledge Hut
MAY 3, 2024
Spark offers over 80 high-level operators that make it easy to build parallel apps and one can use it interactively from the Scala, Python, R, and SQL shells. Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. Basic knowledge of SQL. Yarn etc) Or, 2.
98 
Data Engineering Podcast
AUGUST 3, 2021
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!
130 
Data Engineering Podcast
AUGUST 19, 2019
One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system. This was an interesting view of how a well engineered product can survive massive evolutionary shifts in the industry while remaining relevant and useful.
100 
Christophe Blefari
APRIL 8, 2023
I was in the Hadoop world and all I was doing was denormalisation. The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill. YAML configured.
130 
Christophe Blefari
APRIL 8, 2023
I was in the Hadoop world and all I was doing was denormalisation. The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill. YAML configured.
130 
Knowledge Hut
MAY 2, 2024
It provides high-level APIs in Java, Scala, Python, and R and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools, including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. For Hadoop 2.7,
98 Confluent
SEPTEMBER 26, 2019
In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. Rockset supports JDBC and integrates with other SQL dashboards like Tableau, Grafana, and Apache Superset. However, Apache Kafka is more than just messaging. In the most critical use cases, every seconds counts.
21 
Data Engineering Podcast
JUNE 11, 2021
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. No more scripts, just SQL.
100 
Data Engineering Podcast
FEBRUARY 11, 2018
Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Is timescale compatible with systems such as Amazon RDS or Google Cloud SQL? What impact has the 10.0 What impact has the 10.0
100 
Data Engineering Weekly
DECEMBER 24, 2023
I love the rising, stable, and declining format for categorizing data engineering trends. However, I’m less optimistic about the “multi-engine” orchestrator part. link] DoorDash: Privacy Engineering at DoorDash Drive DoorDash writes about its architecture and policy to protect user privacy at scale.
Data Engineering Weekly
JUNE 29, 2023
But hey, I met my friends after a long time and got my copy of “ Fundamentals of Data Engineering ” signed by Joe Reis & Matt Housely. If you’re starting data engineering, I highly recommend reading it. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.
Data Engineering Podcast
DECEMBER 9, 2018
Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. What are some of the edge cases and architectural considerations that engineers should be considering as they begin to scale their deployments?
100
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content