This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Accessibility Process Systems 
Data Engineering Podcast
MAY 26, 2024
Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. As you have gone through successive migration projects, how has that influenced the ways that you think about architecting data systems?
130 
ArcGIS
MAY 13, 2024
Review best practices for designing and testing for accessibility maps and apps throughout the ArcGIS system during the development process.
111 
Data Engineering Podcast
JUNE 9, 2024
Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. How have the requirements of generative AI shifted the demand for streaming data systems? Can you describe how Datorios is implemented?
147 
ArcGIS
MAY 13, 2024
Review best practices for designing and testing for accessibility maps and apps throughout the ArcGIS system during the development process.
98 
Advertisement
Think your customers will pay more for data visualizations in your application? Five years ago they may have. But today, dashboards and visualizations have become table stakes. Discover which features will differentiate your application and maximize the ROI of your embedded analytics. Brought to you by Logi Analytics.
Data Engineering Podcast
DECEMBER 3, 2023
Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. When is DoubleCloud Data Transfer the wrong choice?
130 
Data Engineering Podcast
JANUARY 7, 2024
Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? What do you have planned for the future of your academic research?
147 
The Pragmatic Engineer
NOVEMBER 21, 2023
If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications. Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process.
302 
Netflix Tech
MARCH 7, 2024
The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.
95 
Start Data Engineering
MAY 28, 2024
Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Project demo 3. Use DuckDB 4.4.
219 
Knowledge Hut
JANUARY 28, 2024
The change control process is a crucial aspect of project management intended to manage and regulate changes made to the project plan, schedule, and budget. These change control process steps are planning, analyzing, approval, testing, implementing, and closing. The change request kickstarts the process of change control.
98 
Analytics Vidhya
FEBRUARY 6, 2023
Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.
185 
The Pragmatic Engineer
OCTOBER 17, 2024
The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Code and raw data repository: Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns.
278 
Precisely
DECEMBER 16, 2024
Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. AIOps presents enormous promise, but many organizations face hurdles in its implementation: Complex ecosystems made of multiple, fragmented systems that lack interoperability.
52 
Data Engineering Weekly
SEPTEMBER 26, 2024
Last Friday, we discussed the challenges in bulk discovery and anonymization processes in data warehouses. Privacy and access management within data infrastructure is not just a best practice; it's a necessity. For example, GDPR and HIPAA require strict access controls to protect sensitive data. Why care about privacy?
69 
Data Engineering Podcast
DECEMBER 24, 2023
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Support Data Engineering Podcast Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems.
245 
Data Engineering Podcast
FEBRUARY 4, 2024
Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?
173 
The Pragmatic Engineer
MARCH 19, 2024
Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research. Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers.
Data Engineering Podcast
FEBRUARY 11, 2024
In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. What are the typical boundaries along which data access requires special consideration for how the sharing is managed?
147 
The Pragmatic Engineer
MARCH 13, 2023
The bank’s systems start to be overloaded to the point of customers not being able to log on and transfer. In the aftermath, we would learn that Silicon Valley Bank processed $42B in deposit outflows on Thursday: which is nearly a quarter of all deposits they held! Several vendors were proactive, offering payment support.
192 
Engineering at Meta
AUGUST 27, 2024
At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. Purpose limitation, a core data protection principle, is about ensuring data is only processed for explicitly stated purposes.
Data Engineering Podcast
JUNE 30, 2024
Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?
147 
The Pragmatic Engineer
APRIL 6, 2023
Our hope is that making salary ranges more accessible on Comprehensive.io For AI, we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing. Our system is using purely Serverless to process the data.
Netflix Tech
JUNE 1, 2023
In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.
140 
The Pragmatic Engineer
SEPTEMBER 19, 2024
In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.
176 
Data Engineering Podcast
MARCH 3, 2024
Who needs to be involved in the process of defining and developing that program? What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" What are the most interesting, unexpected, or challenging lessons that you have learned while working on powering AI systems?
162 
Engineering at Meta
JUNE 27, 2023
In this article we’ll go through a few of the tools in the development process. Sapling: Scaling version control Sapling is a version control system that can scale to huge sizes, but also emphasizes usability. There are three main components to Sapling – a server, a client, and a virtual file system.
132 
Data Engineering Podcast
APRIL 16, 2023
As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? What is your process for adding domain semantics to the operational aspect of your LLM?
Netflix Tech
NOVEMBER 12, 2024
However, this category requires near-immediate access to the current count at low latencies, all while keeping infrastructure costs to a minimum. Failures in a distributed system are a given, and having the ability to safely retry requests enhances the reliability of the service.
101 
Snowflake
APRIL 8, 2024
BigGeo BigGeo accelerates geospatial data processing by optimizing performance and eliminating challenges typically associated with big data. The Innova-Q dashboard provides access to product safety and quality performance data, historical risk data, and analysis results for proactive risk management.
129 
Rockset
MARCH 1, 2023
The Challenge of Compute Contention At the heart of every real-time application you have this pattern that the data never stops coming in and requires continuous processing, and the queries never stop – whether they come from anomaly detectors that run 24x7 or end-user-facing analytics. So they are not suitable for real-time analytics.
144 
Engineering at Meta
FEBRUARY 20, 2024
This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed. An introduction to Velox Velox is the first project in our composable data management system program.
112 
Data Engineering Weekly
AUGUST 6, 2024
CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. Impacts source system performance during query execution.
126 
Snowflake
SEPTEMBER 7, 2023
We are excited to announce the public preview of External Access, which enables customers to reach external endpoints from Snowpark seamlessly and securely. With this announcement, External Access is in public preview on Amazon Web Services (AWS) regions.
Snowflake
JUNE 5, 2024
At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements.
105 
Cloudera
MARCH 22, 2024
That’s why Cloudera and Red Hat , renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux ( RHEL ) to Cloudera on public cloud as the operating system for all of our public cloud platform images. Let’s dive into what this means and why it’s a game-changer for our customers.
120 
Engineering at Meta
APRIL 13, 2023
As this is rolled out, security-conscious users who utilize the verify security code page will notice this verification process occurs quickly and automatically. This system is a new service provided by WhatsApp that relies on public auditing to verify the end-to-end encryption status of personal conversations.
144 
Data Engineering Podcast
MAY 12, 2024
The services and systems need to be kept up to date, but so does the code that controls their behavior. Compounding that challenge is the fact that each of the components of our data platform interact with data in slightly different ways and need different processes for ensuring that changes are being promoted safely.
147 
Data Engineering Podcast
APRIL 28, 2024
In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. What are the features and focus of Pieces that might encourage someone to use it over the alternatives?
147 
Engineering at Meta
DECEMBER 6, 2023
Our aim is to ensure that everyone’s personal messages on Messenger can only be accessed by the sender and the intended recipients, and that everyone can be sure the messages they receive are from an authentic sender. Third-party scrutiny E2EE implies confidentiality even if the provider wants to access the contents of a communication.
145 
Snowflake
APRIL 24, 2024
Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourced them to make LLMs more accessible and cost-effective for the community. license provides ungated access to weights and code.
Knowledge Hut
OCTOBER 29, 2023
Hospital Management System Type : Application development, Database management, Programming There is no shortage of computer science project topics out there. But if you are looking for something that's both technically challenging and socially relevant, consider a hospital management system. Source Code: Weather Forecast App 3.
111 
Netflix Tech
SEPTEMBER 10, 2024
One issue that often complicates this process is the "noisy neighbor" problem. On Titus , our multi-tenant compute platform, a "noisy neighbor" refers to a container or system service that heavily utilizes the server's resources, causing performance degradation in adjacent containers.
100 
Engineering at Meta
OCTOBER 22, 2024
As part of WhatsApp’s new feature to privately add and manage your contacts on WhatsApp across linked devices, we’re announcing a novel encrypted storage system we’ve designed called Identity Proof Linked Storage (IPLS). IPLS is a novel system at WhatsApp that allows users to store their contact names in an encrypted way.
109
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content