Cloudera

NOVEMBER 4, 2024

Document analysis is crucial for efficiently extracting insights from large volumes of text. For example, cancer researchers can use document analysis to quickly understand the key findings of thousands of research papers on a certain type of cancer, helping them identify trends and knowledge gaps needed to set new research priorities.

Unstructured Data 80

Unstructured Data Architecture Algorithm Machine Learning 80

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In August 2021 we open-sourced our code indexing system Glean. Glean collects information about source code and provides it to developer tools through an efficient and flexible query language.

Coding 55

Coding Programming Language SQL Programming 55

Christophe Blefari

MARCH 1, 2023

a macro — a macro is a Jinja function that either do something or return SQL or partial SQL code. ref / source macros — ref and source macros are the most important macros you'll use. ℹ️ I want to mention that the dbt documentation is one of the best tools documentation out there.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. This article looks at the top python IDEs and code editors along with their features, pros, and cons and discusses the best suited for writing Python codes. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

The Pragmatic Engineer

NOVEMBER 9, 2023

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

The Pragmatic Engineer

OCTOBER 10, 2023

However, Martin had not written a line of production code for the last four years, as he’s taken on the role of CEO, and heads up observability scaleup Chronosphere – at more than 250 people and growing. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Snowflake

JUNE 5, 2024

Cortex Search (public preview soon): Quickly and securely find information by asking questions within a given set of enterprise documents using the state-of-the-art Arctic embed model. Snowflake AI & ML Studio for LLMs (private preview): Enable users of all technical levels to utilize AI with no-code development.

Coding 93

Coding Building Management Government 93

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. The process flow for this project is shown in the following diagram: This project's documentation will serve as a starting point from which you may draw ideas for your own work.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Knowledge Hut

OCTOBER 29, 2023

MongoDB is a document-oriented No-SQL database used to hold back-end applications. are responsible for running JS code outside of a web browser. Source Code: Travel Log App 2. Source Code: To-Do List 3. Source Code: Media Player App 4. Source Code: Chat Messaging App 5. The V8 engines that power Node.js

Coding 98

Coding Project Food MongoDB 98

Knowledge Hut

OCTOBER 27, 2023

Code Example javascript import React, { useState, useEffect } from 'react'; import firebase from 'firebase'; function App() { const [courses, setCourses] = useState([]); useEffect(() => { firebase.database().ref('courses/').on('value', cvtColor(image, cv2.COLOR_BGR2GRAY) COLOR_BGR2GRAY) _, thresh = cv2.threshold(gray_image,

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Data Engineering Podcast

OCTOBER 29, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

KDnuggets

AUGUST 6, 2024

Learn statistics through interactive books, code examples, cheat sheets, guides, and tools documentation.

Coding 149

Coding Data Science Data 149

The Pragmatic Engineer

SEPTEMBER 28, 2023

This person wrote up a neat document that was well thought out, and sent it around to other senior staff engineers. But there was a problem, this engineer took an existing document that other engineers had written a few months before, copy-pasted it, changed a few words, and presented it as their own work.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253

DoorDash Engineering

FEBRUARY 27, 2024

Two primary aspects of that search engine were causing the trouble: its document-replication mechanism and its lack of support for complex document relationships. We designed the index to store multiple types of documents with relations between them. Our analysis identified Elasticsearch as our architecture’s primary bottleneck.

Engineering 130

Engineering Systems Designing Architecture 130

Data Engineering Weekly

MAY 3, 2024

However, modern Notebooks like Databricks seamlessly integrate with Git to build pull requests and code review processes. Code Reusability The Notebook code is visible within its scope. Dependency management is also bound to the individual scope of the Notebook and prevents production-grade code quality.

Programming Language 119

Programming Language ETL Tools Data Pipeline Coding 119

Snowflake

APRIL 20, 2023

Protecting sensitive or proprietary data such as source code, PII, internal documents, wikis, code bases, and other sensitive data sets, along with prompts, used to contextualize the LLMs is particularly important. A common barrier to AI adoption for non-coding business teams has been the complexity of AI.

Building 120

Building Unstructured Data Government Coding 120

Robinhood

MAY 30, 2024

While API trading is inherently geared towards more advanced traders, we have built a simple and easy to use API so that anyone who knows how to code can participate with confidence and precision. Our Security Team routinely reviews code and infrastructure powering Robinhood Crypto.

Insurance 142

Insurance Portfolio Algorithm Coding 142

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? But beware, with ever-increasing data sources in your platform, that can only mean the following: Creating large volumes of code for every new connector. Maintaining complex code for every single data connector. Data flowing like cars in a highway.

Coding 53

Coding Cloud Storage Data Data Ingestion 53

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 107

Architecture Project Coding Designing 107

dbt Developer Hub

APRIL 19, 2023

Don’t repeat yourself** : Dimensions can be easily re-used with other fact tables to avoid duplication of effort and code logic. Step 1: Create model files ​ Let’s create the new dbt model files that will contain our transformation code. dim_product.yml : This file will contain our documentation and tests for dim_product.

Building 145

Building PostgreSQL BI Database 145

The Pragmatic Engineer

OCTOBER 18, 2024

  Modify the code so it removed ways to upgrade from free to paid Spotify features in this free app.  According to internal documents, OpenAI expects to generate $100B in revenue in 5 years, which is 25x more than it currently makes. Spotify’s former users on the App Store are now Apple's users.

Government 326

Government Engineering Project AWS 326

Snowflake

MAY 7, 2024

Document chatbots. Their knowledge was contained in more than 700,000 pages of private R&D documents. Using a RAG-based architecture that combines Cortex and Streamlit in Snowflake, the team has built a document chatbot. Arctic excels at enterprise tasks, such as SQL generation, coding and instruction following benchmarks.

Government 114

Government SQL Data Security Accessible 114

Netflix Tech

DECEMBER 3, 2022

One of the main reasons this feature exists is just like with food samples, to give you “a taste” of the production quality ETL code that you could encounter inside the Netflix data ecosystem. " , country_code STRING COMMENT "Country code of the playback session." This is one way to build trust with our internal user base.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Engineering at Meta

FEBRUARY 6, 2024

For example, the source code for many of our first-party tools lives alongside the projects that leverage them inside our massive monorepo. This practice also eliminates a large class of code provenance issues.) See the Generating DotSlash Files at Meta documentation for details. Because of how #! node --version.

Metadata 133

Metadata Coding Building Project 133

Snowflake

NOVEMBER 1, 2023

Text Summarization (in private preview): Summarize long documents for faster consumption. For example, these functions can be incorporated into a chatbot using Streamlit with just a few lines of code. Figure 4 – Quickly extract data from documents with Document AI. See Document AI in action on YouTube.

Unstructured Data 111

Unstructured Data SQL Python Accessible 111

Lyft Engineering

JUNE 28, 2023

Developers had to be able to develop a real-time pipeline in a notebook, test it on realistic streaming data (in that same notebook) to ensure their code is functionally and logically correct, commit the code to a GitHub repository, and deploy the pipeline to production — all with as little friction as possible.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Marc Lamberti

OCTOBER 20, 2023

I’m sure we all agree the code below won’t work: @task def extract_data(): date = '2023-01-01' extract(date) Hardcoding the date means changing it every day, which makes no sense. A template engine is a library that combines templates with data models to produce documents. Stick with me What is Jinja templating?

SQL 130

SQL Python Coding Metadata 130

Data Engineering Podcast

JULY 23, 2023

In Hex you can use SQL, Python, R, and no-code visualization together to explore, transform, and model data. Hex also has AI built directly into the workflow to help you generate, edit, explain and document your code. Use SQL, Python, R, no-code and AI to find and share insights across your organization.

Building 130

Building Machine Learning SQL Python 130

Christophe Blefari

JUNE 16, 2023

Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI. The past often changes, whether because of code or data. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Knowledge Hut

SEPTEMBER 24, 2024

With minimum coding effort, design your magnificent Desktop and Mobile App UIs. Features: Single code base compatible with every major platform Enables to connect to 20+ databases natively via FireDAC's high-paced direct access. Linx If you search for low code IDE and server, Linx is the one for you.

Electronics 98

Electronics MongoDB SQL Java 98

Christophe Blefari

JULY 28, 2023

📃 In every data team this is super important to write documentation, Marie wrote an awesome 101 about data documentation. The article gives best practices for establishing complete and reliable data documentation. 🐍 Finally a few useful code patterns in Python.

Data 130

Data SQL Python Programming 130

Engineering at Meta

SEPTEMBER 17, 2024

Bento allows our engineers to mix code, text, and multimedia in a single document and serves a wide range of use cases at Meta from prototyping to complex machine learning workflows. This episode of the Meta Tech Podcast is all about Bento , Meta’s internal distribution of Jupyter Notebooks, an open-source web-based computing platform.

Machine Learning 114

Machine Learning Engineering Coding Data 114

Data Engineering Podcast

MAY 26, 2024

This episode is supported by Code Comments, an original podcast from Red Hat. In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. My thanks to the team at Code Comments for their support.

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Knowledge Hut

FEBRUARY 19, 2024

This will further help in reducing the cost of development and your feedback will be implemented in the code which will avoid the defects coming from the end user. There is less documentation required. Advantages offered by Agile Methodology The most significant advantage of Agile Methodology is the saving of time and money.

Project 98

Project Coding Technology Management 98

Cloudera

JUNE 1, 2021

Future-proof, “no-code” connectors enable customers to extract data from a wide range of popular data sources, and multi-level transformations are automatically orchestrated using, just, SQL. Cloudera may also file other documents with the SEC regarding the proposed transaction. investor-relations@cloudera.com. 650-644-3900.

Cloud 144

Cloud Big Data Data Lake Finance 144

Knowledge Hut

JANUARY 23, 2024

Here, we will talk about what is markdown and what it's used for; then, we'll jump into VS code with the markdown preview extension and create a kind of cheat sheet that you can use for future reference.   Markdown is a style of writing documents that makes it easy to define what the content should look like.

Portfolio 96

Portfolio Python Media Programming Language 96

Christophe Blefari

SEPTEMBER 28, 2023

If we summarise the initial modern data stack vision , this is something like: move data with Fivetran store data in Snowflake transform data with dbt visualise with Looker document with a catalog, prevent with data observability, orchestrate So what's left of the original vision of the modern data stack that can be applied in 2023 and beyond?

Big Data 147

Big Data Cloud Storage Hadoop SQL 147