The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. Documenting the steps. You can read this here.

Education 363

Education Project PostgreSQL Software Engineer 363

Christophe Blefari

MARCH 1, 2023

ℹ️ I want to mention that the dbt documentation is one of the best tools documentation out there. You just have to understand that there is the reference part which is the detailed documentation of function or configuration and there is the documentation part which is more about concepts and tutorials.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Analytics Vidhya

MARCH 5, 2023

It stores and retrieves large amounts of data, including photos, movies, documents, and other files, in a durable, accessible, and scalable manner. Introduction S3 is Amazon Web Services cloud-based object storage service (AWS).

Amazon Web Services 261

Amazon Web Services AWS Cloud Accessibility 261

Analytics Vidhya

JANUARY 30, 2023

Whether we are analyzing IoT data streams, managing scheduled events, processing document uploads, responding to database changes, etc. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions? appeared first on Analytics Vidhya.

Coding 237

Coding Database Management Process 237

The Pragmatic Engineer

MARCH 12, 2024

” They write the specification, code, tests it, and write the documentation. Edits documentation the chief programmer writes, and makes it production-ready. Brooks suggests the set up below, borrowed from Harlan Mills, could work well: The chief programmer. Brooks calls this person “the surgeon.” The copilot.

Engineering 274

Engineering Programming Language Hospitality Programming 274

The Pragmatic Engineer

SEPTEMBER 28, 2023

This person wrote up a neat document that was well thought out, and sent it around to other senior staff engineers. But there was a problem, this engineer took an existing document that other engineers had written a few months before, copy-pasted it, changed a few words, and presented it as their own work.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253

The Pragmatic Engineer

NOVEMBER 9, 2023

Several times throughout various testimonies, we’ve seen a document written by Sam Bankman-Fried, in which he describes his thinking that Alameda Research should be shut down. That document was, ultimately, how Singh learned in September 2022 that Alameda Research had taken billions of dollars of customer funds from FTX.

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

The Pragmatic Engineer

OCTOBER 10, 2023

In this document, we covered: The product The market The go-to-market (GTM) plan Our competitors … and many other things! With the plan in place, we sent this document – rather than the usual pitch deck – over to the VCs. Our approach was unorthodox, but worked!

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

KDnuggets

AUGUST 6, 2024

Learn statistics through interactive books, code examples, cheat sheets, guides, and tools documentation.

Coding 149

Coding Data Science Data 149

KDnuggets

NOVEMBER 15, 2024

Turn any document into a conversation-ready AI tool with AnythingLLM — a versatile, open-source platform for building a secure, private assistant.

Building 148

Building 148

DoorDash Engineering

FEBRUARY 27, 2024

Two primary aspects of that search engine were causing the trouble: its document-replication mechanism and its lack of support for complex document relationships. We designed the index to store multiple types of documents with relations between them. Our analysis identified Elasticsearch as our architecture’s primary bottleneck.

Engineering 130

Engineering Systems Designing Architecture 130

KDnuggets

MAY 29, 2023

Top free tools for detecting thesis, research papers, assignments, documentation, and blogs generated by AI models.

154

154

KDnuggets

AUGUST 5, 2024

Get noticed by recruiters and hiring managers by creating and documenting the following AI projects.

Portfolio 139

Portfolio Project Recruitment Management 139

databricks

OCTOBER 22, 2024

Over the years, organizations have amassed a vast amount of unstructured text data—documents, reports, and emails—but extracting meaningful insights has remained a challenge.

Data 137

Data 137

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 107

Architecture Project Coding Designing 107

Data Engineering Podcast

OCTOBER 29, 2023

I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data.

Database 189

Database BI SQL Machine Learning 189

Snowflake

JULY 25, 2024

Snowflake Cortex Search, a fully managed search service for documents and other unstructured data, is now in public preview. For document- or chunk-level access controls, you can use metadata filtering to ensure that the service only returns the results that the client is authorized to view.

Unstructured Data 115

Unstructured Data Metadata Government SQL 115

LinkedIn Engineering

MAY 9, 2023

In the past, we addressed latency, throughput and cost issues by migrating off Oracle onto Espresso , an open-source document platform, and adding more nodes. Espresso supports document schema and the schema can evolve over time. The number of requests to our storage infrastructure doubles every year.

Database 133

Database Project Datasets Designing 133

Christophe Blefari

SEPTEMBER 28, 2023

If we summarise the initial modern data stack vision , this is something like: move data with Fivetran store data in Snowflake transform data with dbt visualise with Looker document with a catalog, prevent with data observability, orchestrate So what's left of the original vision of the modern data stack that can be applied in 2023 and beyond?

Big Data 147

Big Data Cloud Storage Hadoop SQL 147

The Pragmatic Engineer

AUGUST 1, 2023

Expectations for moving into an SDE3 (senior) role Amazon is almost alone among most tech companies in the way its internal leveling document specifies levels and their expectations, and also provides guidance on how to move to the next level. Here’s the guidance Amazon offers SDE2 roles about moving to SDE3.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Data Engineering Podcast

JANUARY 21, 2024

I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Seattle Data Guy

OCTOBER 2, 2024

Or maybe you need to parse financial filing documents such as 10-Ks. you work in data, then at some point in your career, you’ll likely need to parse data from a PDF. You might need to parse thousands of PDFs in order to pull out invoice information. This can seem challenging at first.

Data 130

Data Big Data 130

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Max Kanat-Alexander and Grant Jenks Today we are open-sourcing the LinkedIn Developer Productivity & Happiness Framework (DPH Framework) - a collection of documents that describe the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn. Why Open-Source It?

Data Schemas 121

Data Schemas Software Engineer Software Engineering Data Pipeline 121

Christophe Blefari

JULY 28, 2023

📃 In every data team this is super important to write documentation, Marie wrote an awesome 101 about data documentation. The article gives best practices for establishing complete and reliable data documentation.

Data 130

Data SQL Python Programming 130

dbt Developer Hub

APRIL 19, 2023

dim_product.yml : This file will contain our documentation and tests for dim_product. fct_sales.yml : This file will contain our documentation and tests for fct_sales. Step 8: Create model documentation and tests ​ Alongside our fct_sales.sql model, we can populate the corresponding fct_sales.yml file to document and test our model.

Building 145

Building PostgreSQL BI Database 145

Snowflake

MAY 7, 2024

Document chatbots. Their knowledge was contained in more than 700,000 pages of private R&D documents. Using a RAG-based architecture that combines Cortex and Streamlit in Snowflake, the team has built a document chatbot. Ready to build your own document chatbot in Snowflake? Try this step-by-step quickstart.

Government 114

Government SQL Data Security Accessibility 114

KDnuggets

JUNE 14, 2023

GPT4All is the Local ChatGPT for your Documents and it is Free! Falcon LLM: The New King of Open-Source LLMs • Getting Started with ReactPy • Mastering the Art of Data Storytelling: A Guide for Data Scientists • How to Optimize SQL Queries for Faster Data Retrieval

SQL 130

SQL IT Data 130

databricks

DECEMBER 12, 2024

We just followed the documentation online, and within a few hours, we were operational and started running a job. We never had any.

Building 98

Building 98

Snowflake

NOVEMBER 1, 2023

Text Summarization (in private preview): Summarize long documents for faster consumption. With the initial release, you will be able to find tables, views, databases, schemas, Marketplace data products, and Snowflake documentation articles. Figure 4 – Quickly extract data from documents with Document AI.

Unstructured Data 111

Unstructured Data SQL Python Accessibility 111

KDnuggets

JULY 1, 2023

Imagine if the json documents are streaming or continuously flowing from healthcare vendors then we need a robust modern architecture that can deal with such a high volume. Building a datalake for semi-structured data or json has always been challenging.

Building 127

Building Healthcare Structured Data Architecture 127

Knowledge Hut

MAY 2, 2024

Who Owns and Documents User Stories? Anybody who has clarity on the requirement can add details, usually, if there is a Business Analyst in the team, they would document requirements, and in other teams the team member documents them. These may be documented for final verification however cannot implement as a separate story.

Business Analyst 98

Business Analyst Project Certification Management 98

Snowflake

APRIL 10, 2024

Account Executives can bring together documents, spreadsheets, presentations, Salesforce reports, and data from Snowflake to get a comprehensive view of the customer in a single interactive document in Coda. For example, Sales teams can use Coda and Snowflake together for account planning and customer engagement.

Finance 115

Finance Media Cloud Data 115

Knowledge Hut

FEBRUARY 28, 2024

Taking the Requirements granted 100% Complete & Accurate before Discussing with Stakeholders: The numbers of business analysts feel that the documented requirements are 100% correct even before sharing with stakeholders. The common understanding is must to complete the project successfully on the time.

Business Analyst 98

Business Analyst Data Science Project Designing 98

Data Engineering Weekly

MAY 3, 2024

Documentation support Notebooks support intuitive code documentation, which allows developers to include explanations, thought processes, analysis conclusions, and project goals. Many orchestration engines try supporting the Markdown for documentation, yet it is less intuitive than a Notebook.

Programming Language 119

Programming Language ETL Tools Data Pipeline Coding 119

The Pragmatic Engineer

OCTOBER 18, 2024

 According to internal documents, OpenAI expects to generate $100B in revenue in 5 years, which is 25x more than it currently makes.   Sudden layoffs at Meta, Spotify confirms work-from-anywhere, US mandates “click-to-cancel,” a historic SpaceX booster catch – and more.

Government 326

Government Engineering Project AWS 326

Knowledge Hut

FEBRUARY 14, 2024

The primary role of a business analyst in a project is to elicit, analyze and document requirements related to the solution being developed. This uncertainty results in difficulties with regards to eliciting, analyzing and documenting the exact need. The Business Analyst must document assumptions along with the associated attributes.

Business Analyst 98

Business Analyst BI Banking Finance 98

Christophe Blefari

JULY 3, 2023

They announced: Document AI — A new integrated product where you can ask questions in natural language on documents (PDF, etc.). With LLMs they will try to answer questions.

SQL 130

SQL Data Kafka AWS 130