Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

The Pragmatic Engineer

NOVEMBER 9, 2023

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

Software Engineer 251

Software Engineer Software Engineering Engineering Coding 251

Cloudera

NOVEMBER 4, 2024

Document analysis is crucial for efficiently extracting insights from large volumes of text. For example, cancer researchers can use document analysis to quickly understand the key findings of thousands of research papers on a certain type of cancer, helping them identify trends and knowledge gaps needed to set new research priorities.

Unstructured Data 74

Unstructured Data Architecture Algorithm Machine Learning 74

Snowflake

JUNE 5, 2024

Cortex Search (public preview soon): Quickly and securely find information by asking questions within a given set of enterprise documents using the state-of-the-art Arctic embed model. Snowflake AI & ML Studio for LLMs (private preview): Enable users of all technical levels to utilize AI with no-code development.

Coding 103

Coding Building Management Government 103

Data Engineering Podcast

OCTOBER 29, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. For example: A recently joined senior staff engineer decided to propose a project to do a re-architecture of a system. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineering 278

Software Engineering Software Engineer Engineering Building 278

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Outlining the past, present, and future of architecting reliable data systems. You look at the code. Image courtesy of the author. No dice, so what’s next?

Coding 98

Coding Data Software Engineer Software Engineering 98

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests.

Coding 97

Coding SQL Software Engineer Software Engineering 97

Knowledge Hut

OCTOBER 27, 2023

Code Example javascript import React, { useState, useEffect } from 'react'; import firebase from 'firebase'; function App() { const [courses, setCourses] = useState([]); useEffect(() => { firebase.database().ref('courses/').on('value', cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image,

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Towards Data Science

OCTOBER 12, 2023

The associated data in our scenario is stored in a SAP HCM system which is one of the leading applications for human resource management in enterprise environments. For the following scenario we will be using OpenAIs GPT-4 to create the code we need. To solve this we have to bring in some business knowledge to chatgpt.

Data Integration 97

Data Integration Systems Engineering Datasets 97

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. Python interpreters are available on various operating systems, such as Windows , Linux, and Mac OS. But first, let us see what code editors and IDEs are. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. The process flow for this project is shown in the following diagram: This project's documentation will serve as a starting point from which you may draw ideas for your own work.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Knowledge Hut

OCTOBER 29, 2023

MongoDB is a document-oriented No-SQL database used to hold back-end applications. are responsible for running JS code outside of a web browser. Source Code: Travel Log App 2. Source Code: To-Do List 3. Source Code: Media Player App 4. Source Code: Chat Messaging App 5. The V8 engines that power Node.js

Coding 98

Coding Project Food MongoDB 98

The Pragmatic Engineer

OCTOBER 18, 2024

  Modify the code so it removed ways to upgrade from free to paid Spotify features in this free app. Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. OpenAI’s impossible business projections.  According

Government 335

Government Engineering Project AWS 335

Cloudera

DECEMBER 9, 2022

They value NiFi’s visual, no-code, drag-and-drop UI, the 450+ out-of-the-box processors and connectors, as well as the ability to interactively explore data by starting individual processors in the flow and immediately seeing the impact as data streams through the flow. .

Designing 100

Designing Coding Google Cloud Cloud 100

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? But beware, with ever-increasing data sources in your platform, that can only mean the following: Creating large volumes of code for every new connector. Maintaining complex code for every single data connector. Data flowing like cars in a highway.

Coding 75

Coding Cloud Storage Data Data Ingestion 75

Picnic Engineering

MARCH 21, 2023

When building solutions, the code we write can last many years. While casually browsing legacy code we might wonder; is this still used? The missing documentation or outdated tests do not help us answer this. When it is actually not used anymore, it is oftentimes because of historical reasons that code is still around.

Java 75

Java Coding IT Process 75

Robinhood

MAY 30, 2024

While API trading is inherently geared towards more advanced traders, we have built a simple and easy to use API so that anyone who knows how to code can participate with confidence and precision. Our Security Team routinely reviews code and infrastructure powering Robinhood Crypto.

Insurance 142

Insurance Portfolio Algorithm Coding 142

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 110

Architecture Project Coding Designing 110

Data Engineering Weekly

MAY 3, 2024

However, modern Notebooks like Databricks seamlessly integrate with Git to build pull requests and code review processes. Code Reusability The Notebook code is visible within its scope. Dependency management is also bound to the individual scope of the Notebook and prevents production-grade code quality.

Programming Language 117

Programming Language ETL Tools Data Pipeline Coding 117

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Kafka Metadata 117

Knowledge Hut

SEPTEMBER 24, 2024

In the end, you will look at a system ready to serve the best results for the resources and time. With minimum coding effort, design your magnificent Desktop and Mobile App UIs. Features: Single code base compatible with every major platform Enables to connect to 20+ databases natively via FireDAC's high-paced direct access.

Electronics 98

Electronics MongoDB SQL Java 98

Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Coding 134

Coding Database Systems Engineering 134

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. All the code you get with the Dataflow sample workflows is fully functional, adjusted to your environment and isolated from other sample workflows that others generated.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Snowflake

SEPTEMBER 7, 2023

Users can now easily connect to external network locations from their Snowpark code (UDFs/UDTFs and Stored Procedures) while maintaining high security and governance over their data. For security reasons, the user’s code in the Snowpark sandbox is locked by default from accessing the public internet.

Amazon Web Services 130

Amazon Web Services AWS Government Python 130

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. By tracking metrics only at the level of service being updated, we might miss capturing deviations in broader end-to-end system functionality. Canaries and sticky canaries are valuable tools in the system migration process.

Systems 112

Systems Utilities Entertainment Coding 112

Lyft Engineering

JANUARY 30, 2023

Constraints imposed by our legacy system. LyftLearn Serving is a robust, performant, and decentralized system for deploying and serving ML models; it can be used by any team at Lyft to easily infer models online through network calls. This leads to a vast operating environment which is technically challenging to create and maintain.

Machine Learning 93

Machine Learning Systems Coding Python 93

Cloudera

APRIL 21, 2021

We started by looking at the CDP Upgrade Documentation paying particular attention to Requirements and Supported Versions and the Pre-upgrade Transition Steps , which call out the parts of the product that have changed the most. In our case, upgrading to CDP meant major upgrades of operating systems, RDBMS, and a minor Java upgrade.

Cloud 121

Cloud Professional Services Java Data Warehouse 121

Snowflake

JUNE 20, 2024

Designed for processing large data sets, Spark has been a popular solution, yet it is one that can be challenging to manage, especially for users who are new to big data processing or distributed systems. It provided us insights as to code compatibility and allowed us to better estimate our migration time.”

Data Engineering 105

Data Engineering Data Engineer Scala Engineering 105

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. This allows for irreproducible builds, and makes it easy to forget to explicitly list a system dependency if it happens to be installed on the author’s system. Like many other language-specific package managers (e.g.

Project 145

Project Programming Language Building Systems 145

Engineering at Meta

FEBRUARY 6, 2024

DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU. For example, the source code for many of our first-party tools lives alongside the projects that leverage them inside our massive monorepo. Because of how #! node --version.

Metadata 134

Metadata Coding Building Project 134

Cloudera

SEPTEMBER 28, 2020

Review the Upgrade document topic for the supported upgrade paths. Document the number of dev/test/production clusters. Document the operating system versions, database versions, and JDK versions. Review the JDK versions and determine if a JDK version change is needed and if so follow the documentation here to upgrade.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Engineering at Meta

JUNE 10, 2024

At Meta, Bento , our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call “lite” workloads that involve simple prototyping to heavier and more complex machine learning workflows.

SQL 116

SQL Python Coding Machine Learning 116

Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

MySQL 135

MySQL Metadata Data Validation Coding 135