The Pragmatic Engineer

MARCH 19, 2024

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of three topics from last week’s subscriber-only The Pulse issue. Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research.

Software Engineer 275

Software Engineer Software Engineering Programming Language Media 275

KDnuggets

MARCH 22, 2024

GenAI has enabled new search engine platforms with unique features and advantages, challenging Google's dominance.

Engineering 158

Engineering 158

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data.

Database 147

Database Data Lake High Quality Data Data Workflow 147

databricks

MARCH 19, 2024

Today, we are thrilled to announce that Lilac is joining Databricks. Lilac is a scalable, user-friendly tool for data scientists to search, cluster.

Unstructured Data 145

Unstructured Data Data 145

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Engineering at Meta

MARCH 21, 2024

Threads has entered the fediverse! As part of our beta experience, now available in a few countries, Threads users aged 18+ with public profiles can now choose to share their Threads posts to other ActivityPub-compliant servers. People on those servers can now follow federated Threads profiles and see, like, reply to, and repost posts from the fediverse.

Media 138

Media Building Government Engineering 138

KDnuggets

MARCH 20, 2024

Learn data science according to your expertise with these 4 different learning roadmaps.

Data Science 156

Data Science Data 156

Christophe Blefari

MARCH 22, 2024

Friday routine ( credits ) It's Friday and it's Data News. I don't go into too much detail about the magic of Data News, but every Friday is the same. At first, I'm: oh s**t, here we go again and 10 minutes later I'm lost in reading the content and picking too many articles to fit into a thousand word edition. Usually all the process takes me a whole Friday.

Electronics 130

Electronics Media Data Python 130

Waitingforcode

MARCH 22, 2024

Apache Spark leverages the observer design pattern for the framework-to-code communication. One of the consumers' implementations is StreamingQueryListener.

Coding 130

Coding Designing 130

KDnuggets

MARCH 21, 2024

Learn data science in 2024 for FREE with these online courses.

Data Science 153

Data Science Data 153

databricks

MARCH 21, 2024

We are excited to announce the release of the Databricks AI Security Framework (DASF) version 1.0 whitepaper! The framework is designed to improve.

Designing 128

Designing 128

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Snowflake

MARCH 19, 2024

A company’s data strategy is always in motion. Since the explosion of interest in generative AI and large language models (LLMs), that is more true than ever, with business leaders discussing how quickly they should adopt these technologies to stay competitive. Some emerging approaches may be seen in our newly released Snowflake Data Trends 2024 , looking at how users in the Data Cloud are working with their data.

Data Governance 122

Data Governance Government Unstructured Data Programming Language 122

Cloudera

MARCH 22, 2024

As enterprise AI technologies rapidly reshape our digital environment, the foundation of your cloud infrastructure is more critical than ever. That’s why Cloudera and Red Hat , renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux ( RHEL ) to Cloudera on public cloud as the operating system for all of our public cloud platform images.

Cloud 120

Cloud Certification Systems Architecture 120

KDnuggets

MARCH 19, 2024

Check out this beginner’s guide to understanding the role of Large Language Model Operations for seamless user experiences.

152

152

Confluent

MARCH 21, 2024

Learn the best practices for using Confluent Schema Registry, including using schema IDs, understanding subjects and versions, using data contracts, pre-registering schemas, and more.

Data 111

Data 111

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Snowflake

MARCH 21, 2024

Snowflake is committed to helping our customers unlock the power of artificial intelligence (AI) to drive better decisions, improve productivity and reach more customers using all types of data. Large Language Models ( LLMs ) are a critical component of generative AI applications, and multimodal models are an exciting category that allows users to go beyond text and incorporate images and video into their prompts to get a better understanding of the context and meaning of the data.

Government 121

Government Technology Cloud Building 121

Knowledge Hut

MARCH 22, 2024

When I worked in the hospitality industry, the electricity abruptly went out while we were establishing the network and door locks. To my surprise, the door lock opened like any other door. This was the first time in my life that I had ever seen a fail-safe door lock. I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems.

Systems 105

Systems Electronics Hospitality Architecture 105

KDnuggets

MARCH 18, 2024

Learn how to automate machine learning training and evaluation using scikit-learn pipelines, GitHub Actions, and CML.

Machine Learning 151

Machine Learning Programming 151

Engineering at Meta

MARCH 18, 2024

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases.

Engineering 105

Engineering Metadata Architecture Designing 105

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

databricks

MARCH 20, 2024

Next-generation customer experiences are built upon data and insights derived from various touchpoints. Through these, marketers can detect subtle differences in customer needs.

Retail 102

Retail Data 102

Confessions of a Data Guy

MARCH 18, 2024

Want to know how to grow to the Senior Engineering position? Take a look. The post The Path To Senior Engineer appeared first on Confessions of a Data Guy.

Engineering 100

Engineering Data Data Engineering Data Engineer 100

KDnuggets

MARCH 20, 2024

Are you curious about what it takes to become a professional data scientist? Look no further! By following these guides, you can transform yourself into a skilled data scientist and unlock endless career opportunities.

Data Analysis 149

Data Analysis SQL Python Data 149

Engineering at Meta

MARCH 20, 2024

At Meta, we support real-time communication (RTC) for billions of people through our apps, including Messenger, Instagram, and WhatsApp. We’ve seen significant benefits by adopting the AV1 codec for RTC. Here’s how we are improving the RTC video quality for our apps with tools like the AV1 codec, the challenges we face, and how we mitigate those challenges.

Algorithm 104

Algorithm Manufacturing Media Coding 104

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

databricks

MARCH 22, 2024

The GGUF file format is a binary file format used for storing and loading model weights for the GGML library. The library documentation.

Engineering 98

Engineering 98

Knowledge Hut

MARCH 20, 2024

Employees who are in the profession today will change occupations considerably more much of the time than in past time. Thus, you should be aware and proactive in dealing with your profession. Yet, does this mean you need to yield in different parts of your life that are important to you? Our lives are sufficiently occupied balancing work and family without finding time for making out significant profession improvements also.

Utilities 98

Utilities Building Systems Management 98

KDnuggets

MARCH 21, 2024

MetaGPT's newest agent addition makes running data interpretation and analysis tasks a breeze. Find out more and give it a try for yourself.

Data Solutions 148

Data Solutions Data IT 148

WeCloudData

MARCH 20, 2024

Machine learning is revolutionizing traffic prediction, enhancing route planning and reducing congestion in urban commuting. Explore advanced algorithms like Uni-LSTM and BiLSTM for accurate forecasts, along with Google Maps' integration of deep learning for improved ETA accuracy. Discover the practical utility of machine learning in everyday life. The post Navigating your way: Traffic Prediction with Machine Learning appeared first on WeCloudData.

Machine Learning 97

Machine Learning Deep Learning Algorithm Utilities 97

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

databricks

MARCH 21, 2024

Benchmarking for training (dense) models at scale. We demonstrate great performance (very high MFU) and highlight our use of NVIDIA's Transformer Engine, along with PyTorch FSDP and DTensor.

Engineering 95

Engineering 95

Knowledge Hut

MARCH 19, 2024

Logical thinking skills play a significant role in developing careers because they help you reason through vital decisions, generate creative ideas, set goals, and solve problems. You may encounter multiple challenges in your life when you enter the job industry or advance your career. Therefore, need strong logical reasoning skills to solve your problems.

Education 98

Education Certification Process Programming 98

KDnuggets

MARCH 18, 2024

Want to become an ML engineer? These free courses from Google can help you get there.

Machine Learning 147

Machine Learning Engineering 147

Confluent

MARCH 19, 2024

Confluent Cloud's serverless Flink offering is now available on all major clouds, offering a unified, managed platform for real-time data processing.

Cloud 93

Cloud Data Process Process Management 93

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud