The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. This event is shameful and unprecedented in the history of open source on the web. Automattic raised $980M in venture funding and was valued at $7.5B

Government 325

Government Engineering Project AWS 325

Towards Data Science

OCTOBER 30, 2024

Demystifying Azure Storage Account Network Access Service endpoints and private endpoints hands-on: including Azure Backbone, storage account firewall, DNS, VNET and NSGs Connected Network — image by Nastya Dulhiier on Unsplash 1. Defense in depth measures must be in place before data scientists and ML pipelines can access the data.

Accessibility 57

Accessibility Accessible Data Lake Data Science 57

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! We are talking about a competitor to WP Engine (Automattic) with no concrete knowledge of WP Engine’s true revenue, and demanding full access to detailed revenue reports. 25 Sep: Block.

Consulting 305

Consulting AWS Engineering Software Engineer 305

Data Engineering Podcast

APRIL 21, 2024

How do you manage the personalization of the AI functionality in your system for each user/team? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Shortwave the wrong choice? What do you have planned for the future of Shortwave?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Weekly

SEPTEMBER 26, 2024

Privacy and access management within data infrastructure is not just a best practice; it's a necessity. Robust privacy and access management protocols are crucial for GDPR compliance, protecting sensitive information, and maintaining user trust. For example, GDPR and HIPAA require strict access controls to protect sensitive data.

Systems 67

Systems Engineering Data Warehouse Architecture 67

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Metadata Kafka Systems 139

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. In this blog, we will discuss: What is the Open Table format (OTF)? These systems are built on open standards and offer immense analytical and transactional processing flexibility.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 100

Datasets Computer Science Systems Kafka 100

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. Impacts source system performance during query execution.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Engineering at Meta

OCTOBER 23, 2023

To optimize build and performance, we developed our own build system called Buck , which was first open-sourced in 2013. We debated between using Java (like Buck1), Haskell (like the Shake build system ) or Go for the core programming language. Meta has a very large monorepo, with many different programming languages.

Programming Language 117

Programming Language Java Python Programming 117

Cloudera

MARCH 22, 2024

That’s why Cloudera and Red Hat , renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux ( RHEL ) to Cloudera on public cloud as the operating system for all of our public cloud platform images. Let’s dive into what this means and why it’s a game-changer for our customers.

Cloud 116

Cloud Certification Systems Architecture 116

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Weekly

JULY 21, 2024

The blog is an excellent summary of what one needs to know about Gen-AI to start. link] Vimeo: Unlocking knowledge sharing for videos with RAG One of the most significant impacts of large language models (LLMs) is their ability to make information retrieval from multimedia content more accessible.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Christophe Blefari

MARCH 31, 2023

We are slowly approaching the 2-years anniversary of the blog and the newsletter. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much. This pause should be use to develop with policymakers AI governance systems.

Bytes 130

Bytes Data Google Cloud Education 130

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements.

Government 102

Government Accessibility Accessible Cloud 102

Snowflake

APRIL 24, 2024

Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourced them to make LLMs more accessible and cost-effective for the community. license provides ungated access to weights and code.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages.

Project 144

Project Programming Language Building Systems 144

The Modern Data Company

APRIL 7, 2023

It is a big enough issue that several of our recent blog posts ( Lessons in Technical Debt from Southwest Airlines , Start Paying Down Your Technical Debt Today , and A Better Way to Plan the Payoff of Technical Debt) discussed it at length. This blog post will explain what data debt is, the impacts it can have, and how to handle it.

Management 98

Management Government Data Data Management 98

Edureka

FEBRUARY 8, 2023

In this blog we will get to know about the perks of ChatGPT for coding. This blog will help you learn how this effective tool can help you write code with ease, and we will also cover topics like: What is ChatGPT? Availability Regardless of time or location, ChatGPT can perform coding jobs because it is accessible around-the-clock.

Coding 130

Coding Deep Learning Programming Java 130

Edureka

APRIL 10, 2023

In this blog, you will learn how to start using GPT-4 by following the steps mentioned here. How to access GPT-4? GPT-4 is the newest version of OpenAI’s language model systems that accept image and text inputs, and emit text outputs. To learn more about ChatGPT-4 in detail, you can check out this blog.

Accessibility 98

Accessibility Accessible Systems IT 98

Cloudera

JANUARY 6, 2021

Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle. In this blog series, we will explain how to configure PySpark and HBase together for basic Spark use as well as for jobs maintained in CDSW. Below is a sample of how it should look. .

Machine Learning 134

Machine Learning Data Science Database Building 134

Tweag

OCTOBER 25, 2023

nixtract ’s goal is to make it accessible in a way that’s easier to consume. nixtract ’s goal is to make it accessible in a way that’s easier to consume. This is the mechanism that is used to access packages in Nixpkgs. pythonPackages is an attribute set, and we can then access its requests attribute. collect ( x : ( x.

Metadata 114

Metadata Accessibility Accessible Python 114

Tweag

NOVEMBER 20, 2024

Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. is widely taught, there’s really no tutorial nor blog post to teach me about assembly syntax for wasm!

Coding 137

Coding Programming Project Process 137

Netflix Tech

SEPTEMBER 10, 2024

On Titus , our multi-tenant compute platform, a "noisy neighbor" refers to a container or system service that heavily utilizes the server's resources, causing performance degradation in adjacent containers. This feature enables access to a wealth of information stored about a process.

Utilities 104

Utilities Coding Programming Java 104

Engineering at Meta

AUGUST 27, 2024

At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. A crucial aspect of purpose limitation is managing data as it flows across systems and services. These innovations mark a major milestone in our ongoing commitment to honoring user privacy.

Programming Language 118

Programming Language Coding Data Warehouse Systems 118

The Modern Data Company

FEBRUARY 14, 2023

After a winter storm blew through a large swath of the United States, Southwest’s systems and processes had a complete meltdown. While the weather certainly was a catalyst for the mess, it is widely understood that a high level of technical debt within Southwest’s operational systems made a bad situation much, much worse.

Government 105

Government Systems Accessibility Accessible 105

Scott Logic

OCTOBER 13, 2024

Elevate Your Accessibility Testing - 10 Key Steps and 5 Inclusive Strategies to help you raise your game I am often asked by colleagues and clients about accessibility testing, as it is something I have done on many occasions as part of my job as a tester. Accessibility - 10 Key Steps 1.

Accessibility 52

Accessibility Accessible Designing Project 52

Cloudera

SEPTEMBER 23, 2022

In a recent blog, Cloudera Chief Technology Officer Ram Venkatesh described the evolution of a data lakehouse, as well as the benefits of using an open data lakehouse, especially the open Cloudera Data Platform (CDP). You will have access to on-demand compute and storage at your discretion. To the user, it is a serverless experience.

Data Governance 110

Data Governance Government Amazon Web Services Cloud Computing 110

Robinhood

OCTOBER 21, 2024

Historically, UK retail investors have had challenges accessing features like margin investing that provide other ways to help maximise their investments. At Robinhood we understand that investors want access to expand and diversify their portfolios at industry leading rates, in an amazing user experience.”

Portfolio 95

Portfolio Insurance Retail Banking 95

Data Engineering Podcast

APRIL 14, 2024

Contact Info Blog LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info Blog LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Scott Logic

OCTOBER 18, 2023

Rust has always been a good fit for WebAssembly; it is a modern system-level language that has broad popularity (the Stack Overflow revealed it is the most desired language seven years in a row), it also happens to be a popular language for authoring WebAssembly runtimes and platforms. pass values to JavaScript within the browser).

Programming Language 145

Programming Language Coding Java Programming 145

Cloudera

MAY 28, 2024

Data architecture is what defines the structures and systems within an organization responsible for collecting, storing, and accessing data, along with the policies and processes that dictate how data is governed. The post Laying the Foundation for Modern Data Architecture appeared first on Cloudera Blog.

Data Architecture 89

Data Architecture Architecture Data Lake Data Warehouse 89

Cloudera

APRIL 22, 2022

This blog post was written by Dean Bubley , industry analyst, as a guest author for Cloudera. . They are using disaggregated systems, plus distributed compute and data, for their own requirements — running their networks more efficiently, and dealing with customers and operations more flexibly.

Data Solutions 135

Data Solutions Amazon Web Services Data Storage Google Cloud 135

Cloudera

DECEMBER 2, 2021

For the most security-conscious customers, it is a requirement that all network access be done over private networks. This reduces the threat surface area, rendering impossible many of the most common attack vectors that rely on public access to the customer’s systems. Private Endpoint Access for Required Azure Services.

PostgreSQL 124

PostgreSQL Data Lake Data Warehouse Retail 124

Knowledge Hut

FEBRUARY 29, 2024

A database is a structured data collection that is stored and accessed electronically. File systems can store small datasets, while computer clusters or cloud storage keeps larger datasets. Blogs KDnuggets: It is one of the compelling and regularly updated sites for blogs on analytics, Data Science, Big Data and machine learning.

Data Science 98

Data Science Datasets Machine Learning Database Design 98

Data Engineering Weekly

JUNE 16, 2024

link] Jack Vanlightly: A Cost Analysis Of Replication Vs. S3 Express One Zone In Transactional Data Systems S3 Express One Zone, with low latency and write ops certainty, is promising. Replication-based systems remain more economical at low to medium throughputs, especially with significant cross-AZ data transfer discounts.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 91

Metadata Management Data Governance Government 91