Azure Data Engineering

MAY 28, 2022

ARM or Azure Resource Manager templates make it easy to manage deployments for Data Factory. When we connect Data Factory to a source control repository (e.g. GitHub or Azure DevOps Git), the data factory along with all its artefacts ( pipelines , datasets , linked services etc.) is saved in the repository in the form of ARM templates.

Datasets 130

Datasets Coding Data Management 130

The Pragmatic Engineer

AUGUST 1, 2023

Amazon’s SDE3 definition is closer to what is considered a “team lead” or “tech lead” role at many companies, and so I view this SDE3 role as being at the upper end of a senior engineer definition. ” A summary of this level’s typical expectations: A summary of Amazon’s SDE3 level.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Christophe Blefari

MARCH 1, 2023

In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data. You can read dbt's official definitions. You can read my What is a metrics store to help you understand.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

The Pragmatic Engineer

JUNE 22, 2023

The press release: “Squarespace announced today it has entered into a definitive asset purchase agreement with Google, whereby Squarespace will acquire the assets associated with the Google Domains business, which will be winding down following a transition period. ” So what’s being sold, exactly?

Google Cloud 251

Google Cloud Media Cloud Engineering 251

KDnuggets

MAY 9, 2022

Read this overview of 12 important machine learning concepts, presented in a no frills, straightforward definition style.

Machine Learning 160

Machine Learning 160

KDnuggets

JUNE 13, 2022

Gain a beginner's perspective on artificial neural networks and deep learning with this set of 14 straight-to-the-point related key concept definitions.

Deep Learning 160

Deep Learning Machine Learning 160

Data Engineering Podcast

FEBRUARY 18, 2024

To start, can you share your definition of what constitutes a "Data Lakehouse"? To start, can you share your definition of what constitutes a "Data Lakehouse"? What are the technical/architectural/UX challenges that have hindered the progression of lakehouses?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

KDnuggets

OCTOBER 11, 2023

The definitive guide for choosing the right method for your use case.

Process 141

Process 141

Waitingforcode

JANUARY 30, 2024

Previously you could read about transformation of a user job definition into an executable stream graph. Since this explanation was relatively high-level, I decided to deep dive into the final step executing the code.

Coding 130

Coding 130

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it.

Retail 176

Retail Utilities Software Engineer Software Engineering 176

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Reddit r/dataengineering wiki a place where some data eng definitions are written. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Waitingforcode

FEBRUARY 3, 2023

We all have our habits and as programmers, libraries and frameworks are definitely a part of the group. In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week!

Java 130

Java Scala Data Engineer Data Engineering 130

Waitingforcode

FEBRUARY 3, 2023

Watch out the definition of your predicate because from time to time, even though the pushdown predicate is supported by the data source, the predicate can still be executed by the Apache Spark job! Pushdowns in Apache Spark are great to delegate some operations to the data sources. However, there is one important gotcha.

IT 130

IT Process Data 130

The Pragmatic Engineer

APRIL 19, 2023

ChatGPT can be an unexpectedly useful to break down jargon terms: just don't forget to verify its definition, as ChatGPT can make things up. If someone is telling you jargon terms, ask them to explain simply, and challenge them if they cannot do so. Otherwise, understand the jargon in simple terms, yourself.

Software Engineer 295

Software Engineer Software Engineering Engineering Media 295

Data Engineering Podcast

MARCH 10, 2024

Links Project Nessie Article: What is Nessie, Catalog Versioning and Git-for-Data? Article: What is Lakehouse Management?: Links Project Nessie Article: What is Nessie, Catalog Versioning and Git-for-Data? Article: What is Lakehouse Management?:

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Seattle Data Guy

MARCH 14, 2023

But there are definite ways to avoid having your costs grow at an unsustainable rate. Photo by Lukas As you increase your analytical processes and abilities, you’ll unavoidably increase costs. … Read more The post How To Scale Your Data Team’s Impact Without Scaling Costs appeared first on Seattle Data Guy.

Data Science 130

Data Science Data Engineer Data Engineering Data 130

Simon Späti

JANUARY 3, 2023

Here’s the definition from our Data Glossary: The Modern Data Stack (MDS) is a heap of open-source tools to achieve end-to-end analytics from ingestion to [transformation]([link] We’ll cover integration, transformation, orchestration, analytics, and ML tools as a starter guide to the latest open data stack.

Data 130

Data IT 130

KDnuggets

JUNE 19, 2023

In this article we will learn about its definition, differences and how to calculate FLOPs and MACs using Python packages.

Deep Learning 122

Deep Learning Python IT Machine Learning 122

Netflix Tech

DECEMBER 3, 2022

Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ??? In every sample workflow package there are three workflow definition files that work together to provide flexible functionality. See an example high water mark job from the main workflow definition. -

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Christophe Blefari

OCTOBER 9, 2023

Introducing Python and Jinja in Cube — Cube, an open source semantic layer, has released a new writing capabilities in Python with Jinja in the YAML definitions. Joe covers why data modeling was put on the side in the recent years and why we need it back today, showcasing a few useful patterns and definitions.

Python 130

Python Data Database Java 130

Jesse Anderson

AUGUST 28, 2024

Figure 4 - Does the company definition of a team match the book’s definition? The individual contributors must meet the criteria and definitions to represent the job title. However, operations teams are only present in half of the respondents. Merely saying you have a team doesn’t mean it’s the right team.

Data 147

Data Consulting Big Data Data Engineer 147

Netflix Tech

JULY 22, 2024

A workflow definition is defined in a JSON format. Maestro combines user-supplied fields with those managed by Maestro to form a flexible and powerful orchestration definition. A Maestro workflow definition comprises two main sections: properties and versioned workflow including its metadata.

Machine Learning 104

Machine Learning Coding Data Pipeline Java 104

Tweag

NOVEMBER 1, 2023

Instead of writing functions that return records and pipe them together, we write records directly where all fields might not have a definition yet. The proposed layout is rather arbitrary: Nickel has no notion of inputs, local and outputs, but rather of fields with and without definition as well as priorities.

Metadata 113

Metadata Programming Programming Language Coding 113

The Pragmatic Engineer

SEPTEMBER 26, 2023

CDE vendors: a definite trend With more than 20 players in this space, let’s begin with a timeline of when the products launched. It’s also possible to posit that the “spark” which ignited this confluence of factors was the Covid-19 pandemic of 2020-21, and the accompanying rise of remote work.

Cloud 258

Cloud Software Engineer Software Engineering Cloud Computing 258

Jesse Anderson

MARCH 6, 2023

Figure 4 – Does the company definition of a team match the book’s definition? The individual contributors must meet the criteria and definitions to represent the job title. This divide definitely supports my thesis that all three teams are required to generate the highest possible value.

Data Science 130

Data Science Consulting Data Big Data 130

Knowledge Hut

JANUARY 11, 2024

By this definition, it is very evident that an epic is a set of user requirements that can be achieved or completed in either two or more sprints. Does all this jargon sound tricky and technical?

Certification 97

Certification Project Programming Management 97

Azure Data Engineering

JUNE 19, 2022

Azure Data Factory: Script Activity As shown in the screenshot above, this activity supports execution of custom Data Query Language (DQL) as well as Data Definition Language (DDL) and Data Manipulation Language (DML). For DQL, select the Query radio button and for DML, the Non-query radio button.

SQL 130

SQL Datasets Data Database 130

The Pragmatic Engineer

AUGUST 3, 2023

Now after 7 years, Google has announced it will retire Firebase Dynamic Links, but with no definite successor lined up. A timeline of more than 12 months is generous, and by the time Google announces the definite sunset timeline in Q3 of this year, they will likely have given around 18 months’ notice.

Metadata 163

Metadata Engineering Building Technology 163

Snowflake

MAY 22, 2024

Snowflake has entered into a definitive agreement to acquire the TruEra AI observability platform, which provides leading capabilities to evaluate and monitor LLM apps and ML models in production. TruEra’s technology helps evaluate the quality of inputs, outputs, and intermediate results of LLM apps.

Cloud 127

Cloud Data Governance Technology Machine Learning 127

Knowledge Hut

JANUARY 22, 2024

Explicit definition of self The Zen of Python says “Explicit is better than Implicit” Programmers of other languages often ask why self is passed as an explicit parameter every time to define a method. The self variable in Python can also be used to access a variable field within the class definition.

Python 98

Python Programming Language Programming Coding 98

Knowledge Hut

APRIL 16, 2024

Project Definition Projects need to be defined and scoped before they are planned and executed. The main question we ask during project definition is whether the project is worthwhile doing or not. You can capture the conclusion of this phase in a Project Definition Document, Project Charter, Terms of Reference or Business Case.

Project 97

Project Management Certification Process 97

Christophe Blefari

FEBRUARY 19, 2024

Probably because it's not an explicit semantic, definition is relating to the present or recent times as opposed to the remote past. In this definition there are 2 issues. The time came and everyone took their place at the table to eat a slice of cake. A lot of people have issues with the modern word.

Data 100

Data Government SQL Technology 100