The Pragmatic Engineer
MAY 11, 2023
The internet has been speculating the past few days on which crypto company spent $65M on Datadog in 2022. I confirmed it was Coinbase, and here are the details of what happened. Originally published on 11 May 2023. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of six topics in today’s subscriber-only The Scoop issue.
Seattle Data Guy
MAY 8, 2023
If you’re relying on your OLTP system to provide analytics, you might be in for a surprise. While it can work initially, these systems aren’t designed to handle complex queries. Adding databases like MongoDB and CassandraDB only makes matters worse, since they’re not SQL-friendly – the language most analysts and data practitioners are used to.… Read more The post OLTP Vs OLAP – What Is The Difference appeared first on Seattle Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confessions of a Data Guy
MAY 7, 2023
Polars is one of those tools that you just want … no … NEED a reason to use it. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy.
KDnuggets
MAY 8, 2023
Test out the top-of-the-line AI model for no charge.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
The Pragmatic Engineer
MAY 9, 2023
Insights from 50 publicly traded tech companies, and a list of those paying the most and the least in median total compensation. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover two out of seven topics from today’s subscriber-only deep-dive on Compensation at publicly traded tech companies.
Jesse Anderson
MAY 9, 2023
The results and analysis from my 2023 Data Teams Survey left a few open questions. Let’s revisit these questions with some answers. Methodologies and Size of Company Figure 1 – Methodologies Broken Down By Size of Company Using Them We see a few commonalities across different company sizes, as shown in Figure 1. One striking commonality is that so many companies are using data mesh.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
KDnuggets
MAY 10, 2023
Check out these essential statistical concepts for data science.
The Pragmatic Engineer
MAY 12, 2023
This is a response to a tweet asking: "Why is there no competition to PagerDuty/Opsgenie? People in my team say it’s “just connecting to the Twilio API” but if it were that easy, there’d probably be a ton of competition." PagerDuty is the market-leading incident alerting tool. OpsGenie is Atlassian's incident management tool, which is widespread thanks to distribution.
LinkedIn Engineering
MAY 9, 2023
Co-Authors: Estella Pham and Guanlin Lu At peak, LinkedIn serves over 1.4 million member profiles per second. The number of requests to our storage infrastructure doubles every year. In the past, we addressed latency, throughput and cost issues by migrating off Oracle onto Espresso , an open-source document platform, and adding more nodes. We are now at the point where some of the core components are straining under the increasing load, and we can no longer address scaling concerns with the node
databricks
MAY 12, 2023
Download the Databricks Insurance NLP Solution Accelerator Introduction The current economic and social climate has redefined customer expectations and preferences. Society has been.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
MAY 11, 2023
This article has provided an overview of data masking and its importance in ensuring compliance with GDPR and other global regulations.
Waitingforcode
MAY 11, 2023
I have used to say "Kinesis Data Streams is like Apache Kafka, an append-only streaming broker with partitions and offsets". Although often it's true, it's not that simple unfortunately.
Data Engineering Podcast
MAY 7, 2023
Summary Every business has customers, and a critical element of success is understanding who they are and how they are using the companies products or services. The challenge is that most companies have a multitude of systems that contain fragments of the customer's interactions and stitching that together is complex and time consuming. Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it.
databricks
MAY 9, 2023
As organizations are maturing their data infrastructure and accumulating more data than ever before in their data lakes, Open and Reliable table formats.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
KDnuggets
MAY 10, 2023
Chatbot Arena is a benchmark platform for large language models, where the community can contribute new models and evaluate them.
Snowflake
MAY 8, 2023
As of December, customers got a whole new level of insight into Snowflake query performance and query execution statistics when Snowflake announced the public preview of the new get_query_operator_stats function, opening up programmatic access to Snowflake query profiles and providing customers a whole new level of insight into Snowflake query performance and query execution statistics.
Confluent
MAY 9, 2023
Low utilization and operational complexity dramatically increases Kafka costs, so we reinvented Kafka as a cloud-native and complete service to reduce costs for thousands of businesses at any scale.
databricks
MAY 9, 2023
We are excited to announce that support for using Structured Streaming with Delta Sharing is now generally available (GA) in Azure, AWS, and.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
KDnuggets
MAY 12, 2023
Introducing Mojo Lang, the new programming language designed as a superset of Python.
LinkedIn Engineering
MAY 11, 2023
Guaranteeing that our servers are continually upgraded to secure and vetted operating systems is one major step that we take to ensure our members and customers can access LinkedIn to look for new roles, access new learning programs, or exchange knowledge with other professionals. LinkedIn has quite a large fleet of servers on-premise that depend on internal tooling to ensure they stay on the latest operating systems.
Confluent
MAY 10, 2023
Confluent manages 30,000+ Kafka clusters, produces over 3 trillion messages, and does durability checks on over 80 trillion Kafka messages per day while offering 99.99% uptime. Check out our cool stats!
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
KDnuggets
MAY 12, 2023
If you’re interested in trying out AI for fun or learning more about them, then take a look at our list and explore the cutting-edge LLMs available in the wild.
Towards Data Science
MAY 11, 2023
Data-Oriented Programming in Python A recap on Data-Oriented Programming by Yehonathan Sharvit but illustrated with Python examples (instead of JavaScript and Java) Photo by AltumCode on Unsplash Data-Oriented Programming by Yehonathan Sharvit is a great book that gives a gentle introduction to the concept of data-oriented programming (DOP) as an alternative to good old object-oriented programming (OOP).
Precisely
MAY 10, 2023
Technology is a vast industry that has something for everybody. Because of this, it attracts people from all backgrounds and areas of expertise. At Precisely, having diverse representation is the key to success, and as a result, it’s been highly important for the organization to support the unique perspective that employees bring to the table. The Precisely Women in Technology (PWIT) program was designed to connect women from across the organization to one another to offer support, an internal n
Christophe Blefari
MAY 6, 2023
It's wedding weekend (as you'll probably read it, congrats) ( credits ) Hey you, this is a Saturday edition of the Data News. I hope this email finds you well. This week you'll have less editorial content because I'm late. But still you'll find awesome articles that has been written recently. As a reminder on Tuesday next week I'm organising the Apache Airflow Paris meetup that you should consider joining if in Paris.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
MAY 12, 2023
Are you interested in exploring the topic of cognitive biases? Want to see how they may be affecting your data science practice? Check out this free ebook for this and more.
databricks
MAY 11, 2023
Introduction This blog is part of our Admin Essentials series, where we'll focus on topics important to those managing and maintaining Databricks environments.
Pinterest Engineering
MAY 9, 2023
Erika Sun ML Engineer | Advertiser Growth Modeling Team; Ogheneovo Dibie Engineering Manager | Advertiser Growth Modeling Team Photo by Jason Blackeye on Unsplash Summary In this blog post, we describe a Machine Learning (ML) powered proactive churn prevention solution that was prototyped with our small & medium business (SMB) advertisers. Results from our initial experiment suggest that we can detect future churn with a high degree of predictive power and consequently empower our sales par
Towards Data Science
MAY 9, 2023
Managing Python project dependencies in pyproject.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
347 
Let's personalize your content