Knowledge Hut

JANUARY 11, 2024

In today's digital age, the rise of the internet has given us two powerful mediums for expressing creativity: web design & graphic design. Web design involves creating websites, interfaces, & online applications while graphic design entails creating visual designs for print & digital media.

Designing 96

Designing Digital Media Media Programming Language 96

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Contact Info Ryan LinkedIn Paul LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. In the world of technology, things are always changing. In this blog post, we will discuss such technologies.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Knowledge Hut

FEBRUARY 1, 2024

Machine learning (ML) offers just that - an advanced technology to help users find the exact films they want. Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems?

Systems 98

Systems Entertainment Algorithm Datasets 98

Knowledge Hut

NOVEMBER 16, 2023

The best part to jump on the bandwagon of information technology or IT is, there is an enormous possibility for an individual if he or she starts studying for a diploma or a degree, does either a master's degree or a research course. SAS – Statistical Analysis System SAS is a popular course.

Technology 98

Technology MySQL MongoDB Google Cloud 98

Knowledge Hut

MARCH 5, 2024

Cloud Computing is the latest technology that has made it easier for organizations to sort their data and give seamless access to their remote employees. There is a need for various cloud technologies to work on a safer cloud network. It also helps in cost-cutting by eliminating the need for any hardware devices and storage setup.

Cloud Computing 90

Cloud Computing Technology Cloud Architecture 90

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Jesse Anderson

OCTOBER 5, 2023

Data quality problems are solved by more than just technology. Data products need to be exposed with the right technologies for the various use cases. In another post , I introduced a concept from Justin Coffey about how much systems can be changed from their original design. Flink is an excellent piece of technology.

Kafka 195

Kafka Finance Cloud Designing 195

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? What are the differences in terms of pipeline design/access and usage patterns when using a Trino/Iceberg lakehouse as compared to other popular warehouse/lakehouse structures?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

AUGUST 1, 2023

We put a lot of emphasis on communication and prioritization and the ability to unblock yourself or your team – this comes on top of the programming and design skills. Senior engineers are expected to tackle problems where the business problem (or customer case) is well defined, but the technology strategy is not. A lead role.

Software Engineering 245

Software Engineering Software Engineer Engineering Designing 245

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? When is Opaque the wrong choice?

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 233

Recruitment Java Coding Project 233

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. How have the design and goals/scope changed since you first started working on it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Cloudera

NOVEMBER 1, 2024

It also provided a moment for us to launch an important initiative for Cloudera: our Women Leaders in Technology (WLIT) initiative. Building Inclusive Data-Driven Organizations: Leadership Strategies for the Modern Workplace As it stands, women currently account for approximately 25% of the technology workforce.

Technology 93

Technology Recruitment Banking AWS 93

Data Engineering Podcast

JUNE 9, 2024

To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. How have the requirements of generative AI shifted the demand for streaming data systems? What role does Flink play in the architecture of generative AI systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. How have the design and goals of the product changed since you started it? How do you manage the personalization of the AI functionality in your system for each user/team? How have the design and goals of the product changed since you started it?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

The Pragmatic Engineer

SEPTEMBER 22, 2023

There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers.

Programming Language 254

Programming Language Project Coding Engineering 254

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able to provide by serving as a bridge between the technology and business.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. What are the other systems that feed into and rely on the Trino/Iceberg service?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

LinkedIn Engineering

OCTOBER 5, 2023

The magic in search happens because of the technology that powers it. And one of the key technologies behind LinkedIn's search and recommendation features is embedding based retrieval (EBR). LinkedIn is on the forefront of leveraging EBR technology to revolutionize the way we approach search and recommendation systems.

IT 133

IT Metadata Designing Cloud 133

Data Engineering Podcast

MAY 21, 2023

The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" What are the foundational/fundamental principles that you need to design around to allow for future flexibility?

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

MAY 12, 2024

The services and systems need to be kept up to date, but so does the code that controls their behavior. In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. Want to see Starburst in action?

Management 147

Management Data Lake High Quality Data Machine Learning 147

Monte Carlo

NOVEMBER 21, 2024

That’s where data pipeline design patterns come in. So, why does choosing the right data pipeline design matter? In this guide, we’ll explore the patterns that can help you design data pipelines that actually work. Table of Contents Common Data Pipeline Design Patterns Explained 1. Batch Processing Pattern 2.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Engineering at Meta

MARCH 12, 2024

We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We have been openly designing our GPU hardware platforms beginning with our Big Sur platform in 2015.

Building 145

Building Portfolio Utilities Data Storage 145

The Pragmatic Engineer

AUGUST 22, 2023

Perhaps unlike any other software development field, the tools, technologies, practices and pipelines that are typical for a video game production are not well communicated or documented to the wider world. I often explain this working relationship as that artists make it pretty , while designers and programmers make it work.

Software Engineering 206

Software Engineering Software Engineer Consulting Entertainment 206

Engineering at Meta

OCTOBER 15, 2024

We look forward to continued collaboration with OCP to open designs for racks, servers, storage boxes, and motherboards to benefit companies of all sizes across the industry. By breaking down traditional data center technologies into their core components we can build new systems that are more flexible, scalable, and efficient.

Manufacturing 112

Manufacturing Portfolio Designing Systems 112

Cloudera

DECEMBER 18, 2024

To amplify and support women’s voices, Cloudera has introduced the Women Leaders in Technology Initiative. Ive been fortunate to work with some of the most transformative brands in the technology industry. Having launched similar initiatives at HPE and ASG Technologies, Ive seen firsthand the transformative power they can have.

Technology 66

Technology Insurance Banking Programming 66

Data Engineering Podcast

MARCH 24, 2024

What are the most interesting, unexpected, or challenging lessons that you have learned while working on the design and launch of Dagster+? Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Dagster+ the wrong choice?

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

The Pragmatic Engineer

MARCH 19, 2024

In 1959, the programming language COBOL was designed by software engineer Grace Hopper. In an amusing twist, a few years ago, demand for COBOL developers reportedly soared as there are still critical banking and insurance systems using this now-ancient language. And COBOL was just one of many attempts.

Software Engineering 275

Software Engineering Software Engineer Programming Language Media 275

Robinhood

OCTOBER 10, 2024

Throughout the design process, we channeled the bold and disruptive spirit that defines Robinhood to create something completely different that elevates our brand and challenges the industry as a whole. The updated design features: An iconic, simplified logo symbol and supportive wordmark. An intentional modular layout system.

Banking 123

Banking Retail Designing Finance 123

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. What was the impact of leading multiple teams on the design of how to implement communication/contracts throughout the mesh? What do you have planned for the future of your data mesh at PayPal?

Building 147

Building Machine Learning Metadata Data Integration 147

Snowflake

NOVEMBER 16, 2023

The rise of new technologies such as 5G, cloud computing, and the Internet of Things (IoT) is putting pressure on telecom operators to find new ways to improve the performance of their networks, reduce costs and provide better customer service. These systems can make it difficult and expensive to integrate gen AI solutions.

Cloud Computing 103

Cloud Computing Cloud Systems Technology 103