Knowledge Hut
MAY 3, 2023
Did you know that data is now an essential component of modern business operations? With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.
KDnuggets
MAY 24, 2023
When it's all said and done, and AI has been universally recognized as our rightful overlords, the idea of data science as a standalone field will have been but a blip on our collective radar.
The Pragmatic Engineer
MAY 16, 2023
There are a growing number of AI coding tools that are alternatives to Copilot. A list of other popular, promising options.
Analytics Vidhya
MAY 22, 2023
Introduction Join us in this interview as Sumeet shares his background, journey as a former Data Scientist to a software engineer, and learn the captivating aspects of his current job. He provides insights into the future of data science and software engineering and offers valuable advice for career transitioners. Let’s dive into our conversation with […] The post Conversation with Sumeet, Software Engineer at Natwest Group appeared first on Analytics Vidhya.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Seattle Data Guy
MAY 8, 2023
If you’re relying on your OLTP system to provide analytics, you might be in for a surprise. While it can work initially, these systems aren’t designed to handle complex queries. Adding databases like MongoDB and CassandraDB only makes matters worse, since they’re not SQL-friendly – the language most analysts and data practitioners are used to.… Read more The post OLTP Vs OLAP – What Is The Difference appeared first on Seattle Data Guy.
Confessions of a Data Guy
MAY 7, 2023
Polars is one of those tools that you just want … no … NEED a reason to use it. It’s gotten so bad, I’ve started to use it in my Rust code on the side, Polars that is. I mean you have a problem if you could use Polars Python, and you find yourself using […] The post Polars – Laziness and SQL Context. appeared first on Confessions of a Data Guy.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
KDnuggets
MAY 26, 2023
ChatGPT might have some loophole to provide unreliable facts.
The Pragmatic Engineer
MAY 11, 2023
The internet has been speculating the past few days on which crypto company spent $65M on Datadog in 2022. I confirmed it was Coinbase, and here are the details of what happened. Originally published on 11 May 2023. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of six topics in today’s subscriber-only The Scoop issue.
Analytics Vidhya
MAY 24, 2023
As modern companies rely on data, establishing dependable, effective solutions for maintaining that data is a top task for each organization. The complexity of information storage technologies increases exponentially with the growth of data. From physical hard drives to cloud computing, unravel the captivating world of data storage and recognize its ever-evolving role in our […] The post What is Data Storage and How is it Used?
Seattle Data Guy
MAY 20, 2023
Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools. And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult. Here were the key… Read more The post 7 Data Engineering Projects To Put On Your Resume appeared first on Seattle Data Guy.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Jesse Anderson
MAY 9, 2023
The results and analysis from my 2023 Data Teams Survey left a few open questions. Let’s revisit these questions with some answers. Methodologies and Size of Company Figure 1 – Methodologies Broken Down By Size of Company Using Them We see a few commonalities across different company sizes, as shown in Figure 1. One striking commonality is that so many companies are using data mesh.
Data Engineering Podcast
MAY 21, 2023
Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale.
KDnuggets
MAY 2, 2023
In this article, we’ll cover what K-Means clustering is, how the algorithm works, choosing K, and a brief mention of its applications.
The Pragmatic Engineer
MAY 25, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and high-growth startups through the lens of engineering managers and senior engineers. In this issue, we cover one out of six topics from today’s subscriber-only The Scoop issue. To get full articles twice a week, subscribe here.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Analytics Vidhya
MAY 17, 2023
How can we sift through many variables to identify the most influential factors for accurate predictions in machine learning? Recursive Feature Elimination offers a compelling solution, and RFE iteratively removes less important features, creating a subset that maximizes predictive accuracy. By leveraging a machine learning algorithm and an importance-ranking metric, RFE evaluates each feature’s impact […] The post Recursive Feature Elimination: Working, Advantages & Examples ap
Waitingforcode
MAY 4, 2023
Open Source tools helped me switch to the cloud world a lot. The managed cloud services often share the same fundamentals as their Open alternatives. However, there is always something different. Today I'll focus on these differences for Amazon Kinesis service and Apache Kafka ecosystem.
Confluent
MAY 15, 2023
Take a tour of the internals of Confluent’s Apache Kafka® service, powered by Kora: the next-generation, cloud-native streaming engine.Kora.
Data Engineering Podcast
MAY 14, 2023
Summary All of the advancements in our technology is based around the principles of abstraction. These are valuable until they break down, which is an inevitable occurrence. In this episode the host Tobias Macey shares his reflections on recent experiences where the abstractions leaked and some observances on how to deal with that situation in a data platform architecture.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
KDnuggets
MAY 2, 2023
Bark is a versatile audio generation model that supports multi-language, music, voice cloning, and speaker prompts audio generation.
The Pragmatic Engineer
MAY 9, 2023
Insights from 50 publicly traded tech companies, and a list of those paying the most and the least in median total compensation. 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover two out of seven topics from today’s subscriber-only deep-dive on Compensation at publicly traded tech companies.
databricks
MAY 18, 2023
Today, we are thrilled to announce that serverless compute for Databricks SQL is Generally Available on AWS and Azure! Databricks SQL (DB SQL).
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Confluent
MAY 11, 2023
Running Kafka is costly, but Confluent has created a far more efficient product to lower your costs. Join the Cost Savings challenge to see for yourself.
Netflix Tech
MAY 4, 2023
Migrating Critical Traffic At Scale with No Downtime — Part 1 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Hundreds of millions of customers tune into Netflix every day, expecting an uninterrupted and immersive streaming experience. Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience.
KDnuggets
MAY 1, 2023
Have you thought of using ChatGPT to help augment your machine learning tasks? Check out our latest cheat sheet to find out how.
The Pragmatic Engineer
MAY 12, 2023
This is a response to a tweet asking: "Why is there no competition to PagerDuty/Opsgenie? People in my team say it’s “just connecting to the Twilio API” but if it were that easy, there’d probably be a ton of competition." PagerDuty is the market-leading incident alerting tool. OpsGenie is Atlassian's incident management tool, which is widespread thanks to distribution.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Tweag
MAY 16, 2023
Today, I am very excited to announce the 1.0 release of Nickel. A bit more than one year ago, we released the very first public version Nickel (0.1). Throughout various write-ups and public talks ( 1 , 2 , 3 ), we’ve been telling the story of our dissatisfaction with the state of configuration management. The need for a New Deal Configuration is everywhere.
databricks
MAY 30, 2023
We are excited to announce that bit.io is joining Databricks. At Databricks, we’ve always been focused on empowering organizations to solve their toughest p.
LinkedIn Engineering
MAY 9, 2023
Co-Authors: Estella Pham and Guanlin Lu At peak, LinkedIn serves over 1.4 million member profiles per second. The number of requests to our storage infrastructure doubles every year. In the past, we addressed latency, throughput and cost issues by migrating off Oracle onto Espresso , an open-source document platform, and adding more nodes. We are now at the point where some of the core components are straining under the increasing load, and we can no longer address scaling concerns with the node
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Expert insights. Personalized for you.
98 
Let's personalize your content