The Pragmatic Engineer

SEPTEMBER 26, 2023

Every day, there’s more code at a tech company, not less. However, monorepos result in codebases growing large, so that even checking out the code or updating to the head can be time consuming. Concern about code leaks. Open source VS Code Server. In 2021, Microsoft open sourced VS Code Server.

Cloud 316

Cloud Software Engineer Software Engineering Cloud Computing 316

Netflix Tech

DECEMBER 17, 2024

Metric definitions are often scattered across various databases, documentation sites, and code repositories, making it difficult for analysts and data scientists to find reliable information quickly. DJ acts as a central store where metric definitions can live and evolve. Enter DataJunction (DJ).

Engineering 84

Engineering Entertainment Amazon Web Services Utilities 84

Towards Data Science

JANUARY 30, 2025

That said, this tutorial aims to introduce airflow-parse-bench , an open-source tool I developed to help data engineers monitor and optimize their Airflow environments, providing insights to reduce code complexity and parsetime. When writing Airflow DAGs, there are some important best practices to bear in mind to create optimized code.

Python 48

Python Coding Google Cloud Database 48

Knowledge Hut

FEBRUARY 1, 2024

If you are interested in building your own movie recommendation system, there are many resources available online, including tutorials, Data Science courses online & even open-source movie recommendation system source code that you can use as a starting point. values similar_users = similarity_matrix[i].argsort()[:-6:-1]

Systems 98

Systems Entertainment Algorithm Datasets 98

The Pragmatic Engineer

NOVEMBER 9, 2023

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

Software Engineer 290

Software Engineer Software Engineering Engineering Coding 290

Data Engineering Podcast

FEBRUARY 18, 2024

As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20. To start, can you share your definition of what constitutes a "Data Lakehouse"? Visit [dataengineeringpodcast.com/data-council]([link] and use code *depod20* to register today!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Unreliable data doesn’t live in a silo… it’s impacted by all three ingredients of the data ecosystem: data + code + infrastructure. You look at the code.

Coding 98

Coding Data Software Engineering Software Engineer 98

Knowledge Hut

JANUARY 2, 2024

Protect Your Inner Loops Limit the actions carried out in the inner loop when including metrics in code that are executed more than 100,000 times per second or is performance critical. Here are a few methods you can use to safeguard inner loops: Reduce the number of metrics your code uses. Do not call too many metrics in inner loops.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. This article looks at the top python IDEs and code editors along with their features, pros, and cons and discusses the best suited for writing Python codes. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

Booking.com Engineering

AUGUST 3, 2023

Leveraging The Powers of Functional Code — Part 2 The Fully Functional Haskell Solution Part one can be found here: [link] The Solution: Regarding the Haskell code — don’t worry if you don’t understand everything. It truly can change how you think about code. Math definition: (f (g x) = (f . You can read succ .

Coding 91

Coding Java Scala Programming 91

Christophe Blefari

MARCH 1, 2023

a macro — a macro is a Jinja function that either do something or return SQL or partial SQL code. In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

The Pragmatic Engineer

FEBRUARY 23, 2023

Chris Lee is the founder of US-based Launch School , which is one of the “anti bootcamp coding schools,” and an organization which impresses me. As a coding school operator, Chris has a unique perspective that gives him insight into lots of different companies and engineering departments.

Software Engineer 276

Software Engineer Software Engineering Recruitment Portfolio 276

The Pragmatic Engineer

OCTOBER 10, 2023

However, Martin had not written a line of production code for the last four years, as he’s taken on the role of CEO, and heads up observability scaleup Chronosphere – at more than 250 people and growing. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

Towards Data Science

JANUARY 30, 2025

Full code and results available here onGitHub. Moving experiment configs to a YAML, automatically saving results to a file, and having o1 write my visualization code made life mucheasier. Then using one of the strategies below I pruned the training dataset of MNIST and trained a model. Testing was done against the full testset.

Database-centric 106

Database-centric Datasets Data Architecture 106

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it.

Retail 207

Retail Utilities Software Engineer Software Engineering 207

Knowledge Hut

MAY 3, 2024

Any coding interview is a test that primarily focuses on your technical skills and algorithm knowledge. The type of interview you might face can be a remote coding challenge, a whiteboard challenge or a full day on-site interview. So, if you can prove your coding skills learnt in your python programming classes in the interview.

Python 98

Python Coding Data Programming 98

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests. Kent Beck ?

Coding 97

Coding SQL Software Engineering Software Engineer 97

dbt Developer Hub

APRIL 20, 2025

With the dbt MCP server, LLMs can understand and query these metrics directly, ensuring that AI-generated analyses are consistent with your organization's definitions. For human stakeholders : Request metrics using natural language.

Structured Data 98

Structured Data SQL BI Project 98

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? But beware, with ever-increasing data sources in your platform, that can only mean the following: Creating large volumes of code for every new connector. Maintaining complex code for every single data connector. Data flowing like cars in a highway.

Coding 75

Coding Cloud Storage Data Data Ingestion 75

Engineering at Meta

APRIL 28, 2025

We developed tools and APIs for developers to organize assets, classify data, and auto-generate annotation code. This diversity created a unique hurdle for offline assets: the inability to reuse schemas due to the limitations of physical table schemas in adapting to changing definitions.

Metadata 64

Metadata Data Utilities Data Warehouse 64

Monte Carlo

NOVEMBER 26, 2024

Prediction: AI copilots that can complete a sentence, correct code errors, etc. And if Twitter has taught us anything, Sam Altman definitely has a lot to say.) According to Tomasz, the current state of AI can be summed up in three categories. Search: tools that leverage a corpus of data to answer questions 3.

Data Engineering 105

Data Engineering Data Engineer Engineering Unstructured Data 105

Netflix Tech

DECEMBER 3, 2022

One of the main reasons this feature exists is just like with food samples, to give you “a taste” of the production quality ETL code that you could encounter inside the Netflix data ecosystem. " , country_code STRING COMMENT "Country code of the playback session." This is one way to build trust with our internal user base.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

DataKitchen

FEBRUARY 20, 2025

The Quality Dashboard identifies and tracks issues detected during profiling and testing, ensuring you have clear, actionable insights to improve data reliabilityall based on our no-code, generative AI data quality engine.

Datasets 73

Datasets Metadata Data Government 73

Waitingforcode

JANUARY 30, 2024

Previously you could read about transformation of a user job definition into an executable stream graph. Since this explanation was relatively high-level, I decided to deep dive into the final step executing the code.

Coding 130

Coding 130

Data Engineering Podcast

MARCH 10, 2024

As a listener to the Data Engineering Podcast you can get a special discount off regular priced and late bird tickets by using the promo code dataengpod20. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Promo Code: dataengpod20 Dagster : ![Dagster Article: What is Lakehouse Management?:

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Snowflake

NOVEMBER 20, 2023

While such apps are being created at a very fast pace, there are two main challenges: Many modern powerful apps utilize containers to package and use code; however, this typically requires data to be moved from protected environments, increasing data privacy and security risk.

Utilities 125

Utilities Machine Learning Coding AWS 125

Netflix Tech

JANUARY 6, 2025

While this is a critical business need and we definitely should solve it, its essential to evaluate how it stacks up against other priorities across different areas of the organization. Defining TitleHealth Navigating such an ambiguous space required a shared understanding to foster clarity and collaboration.

Metadata 87

Metadata Algorithm Systems Building 87

Netflix Tech

DECEMBER 19, 2024

A natural solution is to make flows configurable using configuration files, so variants can be defined without changing the code. Unlike parameters, configs can be used more widely in your flow code, particularly, they can be used in step or flow level decorators as well as to set defaults for parameters. cluster=sandbox, workflow.id=demo.branch_demox.EXP_01.training

Machine Learning 81

Machine Learning Project Data Warehouse Coding 81

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. What are the utilities that you and the community have built to reduce friction while writing the definitions of a cube?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Data Engineering Podcast

MARCH 3, 2024

As a listener to the Data Engineering Podcast you can get a special discount off regular priced and late bird tickets by using the promo code dataengpod20. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Promo Code: dataengpod20 Starburst : ![Starburst

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

The Pragmatic Engineer

APRIL 13, 2023

” My take is that in the way Covid-19 was an unforeseen ‘black swan’ event, so was the boom in tech and in VC-funding in 2021, which was definitely impacted by the pandemic, thanks to businesses and consumers shifting to digital, as a result of the lockdowns making in-person activities difficult and non-practical.

Finance 235

Finance Media Software Engineer Software Engineering 235

Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 135

MySQL Metadata Coding Data Validation 135

The Pragmatic Engineer

MAY 25, 2023

However, there’s a definite and ongoing uptick since the mid-2021. month-long code freeze at Stack Overflow. In May that year, the company announced a new Chief People Officer, and since then has been a lot more responsive in responding to Glassdoor reviews. What’s going on, and when will Bedrock be available?

Software Engineering 354

Software Engineering Software Engineer AWS Engineering 354

Jesse Anderson

AUGUST 28, 2024

Those using LLMs primarily do so for code generation, ideation or copy creation, and code debugging. Figure 4 - Does the company definition of a team match the book’s definition? The individual contributors must meet the criteria and definitions to represent the job title. As we see, 24.7%

Consulting 147

Consulting Data Big Data Data Engineer 147

Team Data Science

MAY 23, 2020

So you have your notebook, you write your code, then you can make sequel queries and visualize the stuff directly - as tables, bar charts, line graphs and so on. I'm definitely convinced that you need this Zeppelin stuff. The possibilities are really amazing. You can also easily download the plots as CSV files.

Hadoop 130

Hadoop Data Engineering Data Engineer Coding 130

Tweag

MARCH 8, 2023

Users benefit from uniform, comparable code style, across multiple languages, with the convenience of a single formatter tool. In this first release, we have concentrated on formatting OCaml code, capitalising on the OCaml expertise within the Topiary Team and our colleague, Nicolas Jeannerod. Expect idempotency. Prettier ).

Coding 141

Coding Engineering Designing Programming 141

François Nguyen

JANUARY 18, 2021

This article was published in October 2020 with this title : “Data Governance as Code” The idea behind that is you should “actively promotes the safe use of data with automation that improves governance while freeing data analysts and scientists from manual tasks” The article is illustrated with many examples.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130