Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 98

Systems Raw Data Metadata Data Cleanse 98

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. For example: A recently joined senior staff engineer decided to propose a project to do a re-architecture of a system. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineering 278

Software Engineering Software Engineer Engineering Building 278

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager.

Software Engineering 293

Software Engineering Software Engineer Architecture Media 293

Towards Data Science

OCTOBER 12, 2023

The associated data in our scenario is stored in a SAP HCM system which is one of the leading applications for human resource management in enterprise environments. Due to the wide availability of open reference documents and SAP knowledge bases, the approach can be expanded to an Retrieval-Augmented Generation (RAG) solution.

Data Integration 97

Data Integration Systems Engineering Datasets 97

The Pragmatic Engineer

NOVEMBER 9, 2023

Several times throughout various testimonies, we’ve seen a document written by Sam Bankman-Fried, in which he describes his thinking that Alameda Research should be shut down. That document was, ultimately, how Singh learned in September 2022 that Alameda Research had taken billions of dollars of customer funds from FTX.

Software Engineering 251

Software Engineering Software Engineer Engineering Coding 251

Data Engineering Podcast

OCTOBER 29, 2023

I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. What is driving the broad adoption of columnar stores as a separate environment from transactional systems?

Database 189

Database SQL BI Machine Learning 189

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes.  According to internal documents, OpenAI expects to generate $100B in revenue in 5 years, which is 25x more than it currently makes.

Engineering 335

Engineering Government Project AWS 335

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 112

Architecture Project Coding Designing 112

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Max Kanat-Alexander and Grant Jenks Today we are open-sourcing the LinkedIn Developer Productivity & Happiness Framework (DPH Framework) - a collection of documents that describe the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn.

Data Schemas 121

Data Schemas Software Engineering Software Engineer Designing 121

Snowflake

JULY 25, 2024

Snowflake Cortex Search, a fully managed search service for documents and other unstructured data, is now in public preview. Yet, while retrieval is a fundamental component of any AI application stack, creating a high-quality, high-performance RAG system remains challenging for most enterprises.

Unstructured Data 124

Unstructured Data Metadata Government SQL 124

AltexSoft

AUGUST 25, 2021

Both in daily life and in business, we deal with massive volumes of unstructured text data : emails, legal documents, product reviews, tweets, etc. Besides simply looking for email addresses associated with spam, these systems notice slight indications of spam emails, like bad grammar and spelling, urgency, financial language, and so on.

Process 139

Process Deep Learning Datasets Machine Learning 139

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 138

Algorithm Kafka Metadata Systems 138

Seattle Data Guy

JANUARY 18, 2025

Because they can preserve the visual layout of documents and are compatible with a wide range of devices and operating systems, PDFs are used for everything from business forms and educational material to creative designs. PDF files are one of the most popular file formats today.

IT 130

IT Education Data Designing 130

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. What are the organizational/business factors that contribute to the complexity of these systems? Summary Databases and analytics architectures have gone through several generational shifts.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

LinkedIn Engineering

MAY 9, 2023

In the past, we addressed latency, throughput and cost issues by migrating off Oracle onto Espresso , an open-source document platform, and adding more nodes. Espresso supports document schema and the schema can evolve over time. The number of requests to our storage infrastructure doubles every year.

Database 133

Database Project Datasets Designing 133

Robinhood

MAY 30, 2024

We also engage third-party security experts to test our systems, helping us build some of the most secure systems in the industry. Resources and comprehensive documentation are available today at [link] which customers can use to start deploying their trading strategies within minutes.

Insurance 142

Insurance Portfolio Algorithm Coding 142

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. This allows for irreproducible builds, and makes it easy to forget to explicitly list a system dependency if it happens to be installed on the author’s system. Like many other language-specific package managers (e.g.

Project 145

Project Programming Language Building Systems 145

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 118

Machine Learning Building Kafka Metadata 118

Cloudera

JUNE 1, 2021

It also means we can complete our business transformation with the systems, processes and people that support a new operating model. . Cloudera may also file other documents with the SEC regarding the proposed transaction. You may obtain free copies of these documents using the sources indicated above. Our strategy.

Cloud 144

Cloud Big Data Data Lake Finance 144

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. Impacts source system performance during query execution.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Snowflake

JUNE 13, 2024

COPY INTO now supports use cases for unstructured data with the new ingestion capabilities for Document AI (generally available soon). Users can now use Document AI to create a model and use it in automated batch ingest of unstructured documents with formats like PDF, JPEG, HTML and more.

Data Ingestion 104

Data Ingestion MySQL PostgreSQL Data Pipeline 104

Knowledge Hut

SEPTEMBER 24, 2024

In the end, you will look at a system ready to serve the best results for the resources and time. Collaborator Collaborator is a software development tool that helps peer code and view documents that prioritize quality. This will enable you to choose the perfect fit equipped with technologies available at your disposal.

Electronics 98

Electronics MongoDB SQL Java 98

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. By tracking metrics only at the level of service being updated, we might miss capturing deviations in broader end-to-end system functionality. Canaries and sticky canaries are valuable tools in the system migration process.

Systems 112

Systems Utilities Entertainment Coding 112

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In this blog post well talk about why a system like Glean is important, explain the rationale for Gleans design, and run through some of the ways were using Glean to supercharge our developer tooling at Meta.

Coding 77

Coding Programming Language SQL Programming 77

Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Coding 134

Coding Database Systems Engineering 134

Data Engineering Weekly

MAY 3, 2024

However, it forces us to rethink the shared lib model with the shared Notebook, which can be challenging for overall system design. Documentation support Notebooks support intuitive code documentation, which allows developers to include explanations, thought processes, analysis conclusions, and project goals.

Programming Language 117

Programming Language ETL Tools Data Pipeline Coding 117

Snowflake

APRIL 7, 2023

It transforms multiple financial and operational systems’ raw data into a common, friendly data model that people can understand. With Maxa, business teams go from manually managing core systems of record data to working with a single system of insights. Maxa Maxa automates financial and ERP insights.

Raw Data 111

Raw Data Portfolio Building SQL 111

Snowflake

JUNE 5, 2024

Cortex Search (public preview soon): Quickly and securely find information by asking questions within a given set of enterprise documents using the state-of-the-art Arctic embed model. In this model, Cortex Search first fetches relevant documents or passages from a knowledge source, based on the input question.

Coding 103

Coding Building Management Government 103

Data Engineering Weekly

JULY 21, 2024

link] Manuel Faysse: ColPali - Efficient Document Retrieval with Vision Language Models 👀 80% of enterprise data exists in difficult-to-use formats like HTML, PDF, CSV, PNG, PPTX, and more. Vimeo discusses its Retrieval-Augmented Generation (RAG) system design for building a knowledge management system.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Scott Logic

DECEMBER 17, 2023

One of Scottbot’s requirements was to always cite the source of the information it used to generate a response and, as a result, every response was expected to contain a link to the relevant document. Examples of such changes are modifying the system prompt, the agent type, the tools’ descriptions, or the agent’s underlying LLM.

Project 96

Project Algorithm Process Utilities 96

The Pragmatic Engineer

AUGUST 22, 2023

Perhaps unlike any other software development field, the tools, technologies, practices and pipelines that are typical for a video game production are not well communicated or documented to the wider world. Tech debt can manifest quite easily as systems are built for one purpose, only for them later to be used for another.

Software Engineering 206

Software Engineering Software Engineer Consulting Entertainment 206

Snowflake

SEPTEMBER 7, 2023

Ingest data from source systems: Customers looking to ingest data from sources, such as Twitter, Google Sheets, MySQL or other data sources available on the public internet, can use External Access. Snowflake has simplified the process of connecting to external systems while maintaining the trusted perimeter that their customers rely upon.

Amazon Web Services 130

Amazon Web Services AWS Government Python 130

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Data Documentation 101: Why? — Marie wrote best practices for establishing complete and reliable data documentation. The first advice is about the documentation readers: data team, business users or other stakeholders.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Tweag

NOVEMBER 27, 2023

Sponsored by Antithesis (distributed systems reliability testing experts), I’ve developed a new library to filter local files in Nix which I’d like to introduce! This post notably won’t really teach you a lot about the new library, that’s what the official tutorial and the reference documentation is for.

Building 125

Building Systems IT Project 125

Towards Data Science

FEBRUARY 18, 2024

This document serves as a tutorial and offers best practices for effectively utilizing Faust. Following that, I explore the key functionalities of the Faust library, placing emphasis on Faust windows, which are often difficult to grasp from the available documentation and utilize efficiently.

Kafka 98

Kafka Python Process Google Cloud 98

Snowflake

APRIL 1, 2024

The bidirectional integration now enables data sharing from across business systems, Salesforce clouds and operational systems available to analyze data sets and drive future actions. Talk to your Snowflake and Salesforce representatives, explore the integration documentation , or watch our webinar.

Cloud 130

Cloud Retail Entertainment Media 130