Cloudera

NOVEMBER 4, 2024

Document analysis is crucial for efficiently extracting insights from large volumes of text. For example, cancer researchers can use document analysis to quickly understand the key findings of thousands of research papers on a certain type of cancer, helping them identify trends and knowledge gaps needed to set new research priorities.

Unstructured Data 80

Unstructured Data Architecture Algorithm Machine Learning 80

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In this blog post well talk about why a system like Glean is important, explain the rationale for Gleans design, and run through some of the ways were using Glean to supercharge our developer tooling at Meta.

Coding 55

Coding Programming Language SQL Programming 55

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. Python interpreters are available on various operating systems, such as Windows , Linux, and Mac OS. But first, let us see what code editors and IDEs are. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

The Pragmatic Engineer

NOVEMBER 9, 2023

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

Data Engineering Podcast

OCTOBER 29, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

Knowledge Hut

OCTOBER 27, 2023

Code Example javascript import React, { useState, useEffect } from 'react'; import firebase from 'firebase'; function App() { const [courses, setCourses] = useState([]); useEffect(() => { firebase.database().ref('courses/').on('value', cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image,

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. The process flow for this project is shown in the following diagram: This project's documentation will serve as a starting point from which you may draw ideas for your own work.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Snowflake

JUNE 5, 2024

Cortex Search (public preview soon): Quickly and securely find information by asking questions within a given set of enterprise documents using the state-of-the-art Arctic embed model. Snowflake AI & ML Studio for LLMs (private preview): Enable users of all technical levels to utilize AI with no-code development.

Coding 93

Coding Building Management Government 93

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. For example: A recently joined senior staff engineer decided to propose a project to do a re-architecture of a system. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253

Cloudera

DECEMBER 9, 2022

They value NiFi’s visual, no-code, drag-and-drop UI, the 450+ out-of-the-box processors and connectors, as well as the ability to interactively explore data by starting individual processors in the flow and immediately seeing the impact as data streams through the flow. .

Designing 105

Designing Coding Google Cloud AWS 105

Knowledge Hut

OCTOBER 29, 2023

MongoDB is a document-oriented No-SQL database used to hold back-end applications. are responsible for running JS code outside of a web browser. Source Code: Travel Log App 2. Source Code: To-Do List 3. Source Code: Media Player App 4. Source Code: Chat Messaging App 5. The V8 engines that power Node.js

Coding 98

Coding Project Food MongoDB 98

The Pragmatic Engineer

OCTOBER 18, 2024

  Modify the code so it removed ways to upgrade from free to paid Spotify features in this free app. Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. OpenAI’s impossible business projections.  According

Government 326

Government Engineering Project AWS 326

Picnic Engineering

MARCH 21, 2023

When building solutions, the code we write can last many years. While casually browsing legacy code we might wonder; is this still used? The missing documentation or outdated tests do not help us answer this. When it is actually not used anymore, it is oftentimes because of historical reasons that code is still around.

Java 75

Java Coding IT Process 75

Robinhood

MAY 30, 2024

While API trading is inherently geared towards more advanced traders, we have built a simple and easy to use API so that anyone who knows how to code can participate with confidence and precision. Our Security Team routinely reviews code and infrastructure powering Robinhood Crypto.

Insurance 142

Insurance Portfolio Algorithm Coding 142

Data Engineering Weekly

MAY 3, 2024

However, modern Notebooks like Databricks seamlessly integrate with Git to build pull requests and code review processes. Code Reusability The Notebook code is visible within its scope. Dependency management is also bound to the individual scope of the Notebook and prevents production-grade code quality.

Programming Language 119

Programming Language ETL Tools Data Pipeline Coding 119

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 107

Architecture Project Coding Designing 107

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Outlining the past, present, and future of architecting reliable data systems. You look at the code. Image courtesy of the author. No dice, so what’s next?

Coding 70

Coding Data Software Engineer Software Engineering 70

Knowledge Hut

SEPTEMBER 24, 2024

In the end, you will look at a system ready to serve the best results for the resources and time. With minimum coding effort, design your magnificent Desktop and Mobile App UIs. Features: Single code base compatible with every major platform Enables to connect to 20+ databases natively via FireDAC's high-paced direct access.

Electronics 98

Electronics MongoDB SQL Java 98

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Christophe Blefari

SEPTEMBER 28, 2023

When you are a data engineer you're getting paid to build systems that people can rely on. Big data technologies are dead—bye Zookeeper 👋—but data generated by systems are still massive and is the modern data stack relevant to answer this need in storage and processing? Something boring.

Big Data 147

Big Data Cloud Storage Hadoop SQL 147

Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Coding 133

Coding Database Systems Engineering 133

Engineering at Meta

FEBRUARY 6, 2024

DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU. For example, the source code for many of our first-party tools lives alongside the projects that leverage them inside our massive monorepo. Because of how #! node --version.

Metadata 133

Metadata Coding Building Project 133

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. All the code you get with the Dataflow sample workflows is fully functional, adjusted to your environment and isolated from other sample workflows that others generated.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Towards Data Science

OCTOBER 12, 2023

The associated data in our scenario is stored in a SAP HCM system which is one of the leading applications for human resource management in enterprise environments. For the following scenario we will be using OpenAIs GPT-4 to create the code we need. To solve this we have to bring in some business knowledge to chatgpt.

Data Integration 66

Data Integration Systems Engineering Datasets 66

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. By tracking metrics only at the level of service being updated, we might miss capturing deviations in broader end-to-end system functionality. Canaries and sticky canaries are valuable tools in the system migration process.

Systems 114

Systems Utilities Entertainment Coding 114

Cloudera

APRIL 21, 2021

We started by looking at the CDP Upgrade Documentation paying particular attention to Requirements and Supported Versions and the Pre-upgrade Transition Steps , which call out the parts of the product that have changed the most. In our case, upgrading to CDP meant major upgrades of operating systems, RDBMS, and a minor Java upgrade.

Cloud 122

Cloud Professional Services Java Data Warehouse 122

Lyft Engineering

JANUARY 30, 2023

Constraints imposed by our legacy system. LyftLearn Serving is a robust, performant, and decentralized system for deploying and serving ML models; it can be used by any team at Lyft to easily infer models online through network calls. This leads to a vast operating environment which is technically challenging to create and maintain.

Machine Learning 93

Machine Learning Systems Coding Python 93

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Cloudera

SEPTEMBER 28, 2020

Review the Upgrade document topic for the supported upgrade paths. Document the number of dev/test/production clusters. Document the operating system versions, database versions, and JDK versions. Review the JDK versions and determine if a JDK version change is needed and if so follow the documentation here to upgrade.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. This allows for irreproducible builds, and makes it easy to forget to explicitly list a system dependency if it happens to be installed on the author’s system. Like many other language-specific package managers (e.g.

Project 144

Project Programming Language Building Systems 144

Snowflake

SEPTEMBER 7, 2023

Users can now easily connect to external network locations from their Snowpark code (UDFs/UDTFs and Stored Procedures) while maintaining high security and governance over their data. For security reasons, the user’s code in the Snowpark sandbox is locked by default from accessing the public internet.

Amazon Web Services 122

Amazon Web Services AWS Government Python 122

Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

MySQL 134

MySQL Metadata Data Validation Coding 134

Engineering at Meta

JUNE 10, 2024

At Meta, Bento , our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call “lite” workloads that involve simple prototyping to heavier and more complex machine learning workflows.

SQL 115

SQL Python Coding Machine Learning 115

AltexSoft

AUGUST 25, 2021

Both in daily life and in business, we deal with massive volumes of unstructured text data : emails, legal documents, product reviews, tweets, etc. Besides simply looking for email addresses associated with spam, these systems notice slight indications of spam emails, like bad grammar and spelling, urgency, financial language, and so on.

Process 139

Process Deep Learning Datasets Machine Learning 139

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? But beware, with ever-increasing data sources in your platform, that can only mean the following: Creating large volumes of code for every new connector. Maintaining complex code for every single data connector. Data flowing like cars in a highway.

Coding 53

Coding Cloud Storage Data Data Ingestion 53