Data Engineering Weekly

SEPTEMBER 8, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Join Monte Carlo for a deep dive session into how data governance leaders can take their data quality strategies to the next level with end-to-end data observability.

Data Engineer 52

Data Engineer Data Engineering Engineering Kafka 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ? We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing.

Technology 130

Technology Architecture Google Cloud Metadata 130

Knowledge Hut

MARCH 28, 2024

They use many data storage, computation, and analytics technologies to develop scalable and robust data pipelines. Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Develop data models, data governance policies, and data integration strategies.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Warehouse 52

Knowledge Hut

MARCH 20, 2024

The job description for Azure data engineer that I have elucidated below focuses more on foundational tasks while providing opportunities for learning and growth within the field: Data ingestion: This role involves assisting in the process of collecting and importing data from various sources into Azure storage solutions.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Governance 52

Knowledge Hut

MARCH 15, 2024

The job description for Azure data engineer that I have elucidated below focuses more on foundational tasks while providing opportunities for learning and growth within the field: Data ingestion: This role involves assisting in the process of collecting and importing data from various sources into Azure storage solutions.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Governance 52

Ascend.io

SEPTEMBER 5, 2023

Snowpark’s key benefit is its ability to support coding in languages other than SQL—such as Scala, Java, and Python—without moving data out of Snowflake and, therefore , take full advantage of its powerful capabilities through code. SQL, as a declarative language, offers a broad audience the power to query data.

IT 59

IT Scala Java Programming Language 59

Data Engineering Podcast

JUNE 5, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done.

Data Security 100

Data Security Metadata MongoDB Scala 100

Snowflake

SEPTEMBER 13, 2023

DevOps Deploy and Manage Snowflake objects and code with ease – public preview You now have the ability to execute a SQL script directly from a file on a stage, such as from VS code, or a Snowflake Native App with the EXECUTE IMMEDIATE FROM command now in public preview. Learn more about our SQL delight features. Learn more.

Python 77

Python SQL Data Pipeline Data Ingestion 77

Monte Carlo

JULY 15, 2024

Key Responsibilities of Data Owners The primary responsibility of a data owner is ensuring the quality, accuracy, and integrity of their assigned data assets. Career advancement: As organizations become more data-centric, your role as a data owner offers opportunities for career growth.

Pipeline-centric 52

Pipeline-centric Government Data Governance Retail 52

Monte Carlo

JUNE 2, 2024

To get a better understanding of their responsibilities, we analyzed 41 job postings on LinkedIn and found these key duties: Designing and Implementing Data Pipelines: The primary responsibility in 90% of the job postings is creating and managing data pipelines.

Engineering 52

Engineering Amazon Web Services Data Science AWS 52

Monte Carlo

JUNE 2, 2024

To get a better understanding of their responsibilities, we analyzed 41 job postings on LinkedIn and found these key duties: Designing and Implementing Data Pipelines: The primary responsibility in 90% of the job postings is creating and managing data pipelines.

Engineering 52

Engineering Amazon Web Services Data Science AWS 52

Monte Carlo

JUNE 6, 2023

People that are not proficient in SQL and business intelligence will no longer need to ask an analyst or analytics engineer to create a dashboard for them. Simultaneously, those who are proficient will be able to answer their own questions and build data products quicker and more efficiently.

Data Engineer 84

Data Engineer Data Engineering Engineering Business Intelligence 84

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional who is in charge of designing, implementing, and maintaining data processing systems and solutions on the Microsoft Azure cloud platform. A Data Engineer is responsible for designing the entire architecture of the data flow while taking the needs of the business into account.

Data Engineer 52

Data Engineer Data Engineering Project Coding 52

Data Engineering Podcast

JULY 10, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

APRIL 26, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Ascend.io

OCTOBER 12, 2023

By evaluating the current state of your data ecosystem and establishing explicit objectives, you set the stage for a successful automation transition. Additionally, considerations around data governance and initial workflow design ensure that when you do move forward, you do so with confidence and direction.

Hospitality 52

Hospitality Data Pipeline Healthcare Data Governance 52

RandomTrees

FEBRUARY 2, 2024

GPT-Based Data Engineering Accelerators: Given below is the list of some of the GPT-based data engineering accelerators. 1. DataGPT OpenAI developed DataGpt for performing data engineering tasks. Datagpt creates code for data pipelines and transformations.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Pipeline 52

Ascend.io

DECEMBER 19, 2023

That’s what we call a data pipeline. It could just as well be ‘ELT for Snowflake’ The key takeaway is that these terms are representative of the actual activity being undertaken: the construction and management of data pipelines within the Snowflake environment.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Data Engineering Podcast

OCTOBER 15, 2021

The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Can you describe what Acryl Data is and the story behind it?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

JULY 12, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Designing 100

Designing Data Warehouse Data Governance Architecture 100

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Monte Carlo

JULY 27, 2023

The skills, languages and tools of a data quality engineer Data quality engineers need to be highly skilled in multiple programming languages such as SQL (mentioned in 61% of postings), Python (56%), and Scala (13%). Assist in developing and maintaining data governance policies and standards.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Knowledge Hut

SEPTEMBER 21, 2023

Azure Data Engineer: Overview Being an Azure Data Engineer is an exhilarating journey filled with opportunities for growth and innovation and making a real impact in the world of data. The role of an Azure data engineer revolves around designing, building, and maintaining data pipelines and infrastructure on the Azure platform.

Data Engineer 52

Data Engineer Data Engineering Engineering Cloud Computing 52

DareData

JULY 5, 2023

Test, Test, Test With the flexibility of data lake infrastructures, there's also a higher likelihood that your pipelines may fail - particularly when you are acquiring data from sources that you don't control (APIs, Scraping the Web, etc.). This flexibility can lead to valuable insights and opportunities for business growth.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Knowledge Hut

APRIL 25, 2023

Data engineering is a field that requires a range of technical skills, including database management, data modeling, and programming. Data engineering tools can help automate many of these processes, allowing data engineers to focus on higher-level tasks like extracting insights and building data pipelines.

Data Engineer 52

Data Engineer Data Engineering Engineering Google Cloud 52

AltexSoft

FEBRUARY 11, 2023

Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer? This privacy law must be kept in mind when building data architecture.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Knowledge Hut

DECEMBER 26, 2023

If you are looking for a wholesome list of books, including one like statistics for data science book or even the best python data science books, you can refer to the below list:- 1. Programming Programming is crucial to have an in-depth understanding of data science.

Data Science 52

Data Science Scala Programming Language R (Programming) 52

Edureka

FEBRUARY 7, 2023

Azure Data Engineers use a variety of Azure data services, such as Azure Synapse Analytics, Azure Data Factory, Azure Stream Analytics, and Azure Databricks, to design and implement data solutions that meet the needs of their organization.

Data Engineer 52

Data Engineer Data Engineering Engineering Programming Language 52

Monte Carlo

JUNE 16, 2023

Version control and audit trails can be used to keep track of all changes made to the data, ensuring its integrity over time. The field of data engineering is rife with examples of common techniques and best practices designed to maintain data integrity.

Data Integration 52

Data Integration Hospitality Banking Healthcare 52

Monte Carlo

MARCH 28, 2023

How do we instill the sense of quality and timeliness in the data?” How does stale data happen and what are the consequences? Stale data can be the result of various factors, such as outdated data collection processes, bad SQL queries, and broken data pipelines.

IT 52

IT Machine Learning Data Business Intelligence 52

Monte Carlo

SEPTEMBER 12, 2022

Compared to software engineering, Maxime says, “I think the on-ramp for people to become more sophisticated with data is just easier and more accessible to more people.” And over the last five years, many data analysts have broadened their knowledge to include traditional engineering skills like Git and SQL.

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Databand.ai

DECEMBER 13, 2022

We’d be remiss not to share that Joseph was a recent guest on Databand’s MAD Data Podcast , where he discussed ways to keep data systems from becoming unwieldy and shared tips for data teams to manage their data warehouses and keep data pipelines running reliably. You can also watch the video recording.

Data Engineer 52

Data Engineer Data Engineering Engineering AWS 52

Data Engineering Podcast

NOVEMBER 30, 2020

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Immuta is an automated data governance solution that enables safe and easy data analytics in the cloud.

Data Integration 100

Data Integration Coding BI Kafka 100

Silectis

DECEMBER 15, 2020

A Unified Data Engineering Solution Your journey with Magpie starts with a seamless integration with AWS, GCP, or Azure. Magpie is an enterprise-ready solution built on the powerful Apache Spark, but with language support for SQL, Python, R, and Scala. No need to learn a new language or waste time writing tons of new code.

Data Engineer 52

Data Engineer Data Engineering Engineering Scala 52

Data Engineering Weekly

JUNE 18, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today. Kudos to the author.

Data Engineer 52

Data Engineer Data Engineering Engineering MySQL 52

Monte Carlo

JULY 11, 2023

Regardless of the approach you choose, it’s important to keep a scrutinous eye on whether or not your data outputs are matching (or close to) your expectations; often, relying on a few of these measures will do the trick. Data governance and management hurdles Poor data governance practices can jeopardize the accuracy of entire datasets.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40