The Pragmatic Engineer

NOVEMBER 26, 2024

It’s been nearly 6 months since our research into which AI tools software engineers use, in the mini-series, AI tooling for software engineers: reality check. At the time, the most popular tools were ChatGPT for LLMs, and GitHub copilot for IDE-integrated tooling. Then this summer, I saw the Cursor IDE becoming popular around when Anthropic’s Sonnet 3.5 model was released, which has superior code generation compared to ChatGPT.

Software Engineer 289

Software Engineer Software Engineering Engineering Coding 289

Tweag

NOVEMBER 20, 2024

Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. A lot of improvements have happened after the initial merge, including but not limited to: Many, many bugfixes in the code generator and runtime, witnessed by the full GHC testsuite for the wasm backend in upstream GHC CI pipelines.

Coding 137

Coding Programming Project Process 137

KDnuggets

NOVEMBER 6, 2024

Build an LLM application by easily picking and dropping components and connecting them, such as a vector store, web search, memory, and custom prompt.

Coding 147

Coding Building 147

Confluent

NOVEMBER 25, 2024

In the final article of this four-part series on Kafka producer and consumer internals, observe the inner workings of brokers as they attempt to serve data up to consumers.

Kafka 133

Kafka Data 133

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Seattle Data Guy

NOVEMBER 18, 2024

There are plenty of statistics about the speed at which we are creating data in today’s modern world. On the flip side of all that data creation is a need to manage all of that data and thats where data teams come in. But leading these data teams is challenging and yet many new data… Read more The post From IC to Data Leader: Key Strategies for Managing and Growing Data Teams appeared first on Seattle Data Guy.

Management 130

Management Data Big Data Data Science 130

Start Data Engineering

NOVEMBER 21, 2024

1. Introduction 2. Analytical databases aggregate large amounts of data 3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1. A compute engine is a system that transforms data 3.1.2. Metadata catalog stores information about datasets 3.1.3. Data platform support for SQL, Dataframe, and Dataset APIs 3.1.4.

Metadata 130

Metadata Datasets SQL Database 130

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction. This counting service, built on top of the TimeSeries Abstraction, enables distributed counting at scale while maintaining similar low latency performance.

Datasets 100

Datasets Computer Science Systems Kafka 100

KDnuggets

NOVEMBER 8, 2024

From learning Python to creating analytical reports, learn about ten easy steps to become a data scientist.

Python 136

Python Data 136

databricks

NOVEMBER 25, 2024

The Databricks Serverless compute infrastructure launches and manages millions of virtual machines (VMs) each day across three major cloud providers, and it is.

Cloud 114

Cloud Management IT Data Engineering 114

Seattle Data Guy

NOVEMBER 19, 2024

Scraping data from PDFs is a right of passage if you work in data. Someone somewhere always needs help getting invoices parsed, contracts read through, or dozens of other use cases. Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python 130

Python Data Data Science Data Engineering 130

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights. Before building your own data architecture from scratch though, why not steal – er, learn from – what industry leaders have already figured out?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

The Pragmatic Engineer

NOVEMBER 29, 2024

The last 12 months, The Pragmatic Engineer covered a variety of deepdives, revealing previously unshared details like: * What Stripe's engineering culture is like * The architecture evolution of Bluesky * How the ChatGPT scaled to meet demand * How Anthropic builds products * How and why hardware startup Oxide built two new computers from

Engineering 136

Engineering Architecture Building Computer Science 136

Confluent

NOVEMBER 21, 2024

Confluent’s CwC partner program introduces bidirectional data streaming for SAP Datasphere, powered by Apache Kafka and Apache Flink; CwC Q4 2024 new entrants.

Programming 106

Programming Kafka Data 106

KDnuggets

NOVEMBER 29, 2024

Working with large datasets is common but challenging. Here are some tips to make working with such large datasets in Python simpler.

Datasets 130

Datasets Python 130

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

databricks

NOVEMBER 21, 2024

We’re excited to announce a new integration between Databricks Notebooks and AI/BI Dashboards, enabling you to effortlessly transform insights from your notebooks into.

BI 114

BI 114

Seattle Data Guy

NOVEMBER 8, 2024

Recently, I’ve encountered a few projects that used AWS DMS, which is almost like an ELT solution. Whether it was moving data from a local database instance to S3 or some other data storage layer. It was interesting to see AWS DMS used in this manner. But it’s not what DMS was built for. As… Read more The post What Is AWS DMS And Why You Shouldn’t Use It As An ELT appeared first on Seattle Data Guy.

AWS 130

AWS IT Data Storage Database 130

Cloudera

NOVEMBER 13, 2024

Large Language Models (LLMs) will be at the core of many groundbreaking AI solutions for enterprise organizations. Here are just a few examples of the benefits of using LLMs in the enterprise for both internal and external use cases: Optimize Costs. LLMs deployed as customer-facing chatbots can respond to frequently asked questions and simple queries.

Datasets 81

Datasets Machine Learning Coding Data Preparation 81

ArcGIS

NOVEMBER 4, 2024

An overview of all the enhancements and improves with ArcGIS Data Interoperability with the latest release of ArcGIS Pro at version 3.4.

Data 101

Data Metadata 101

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

DataKitchen

NOVEMBER 27, 2024

A Guide to the Six Types of Data Quality Dashboards Poor-quality data can derail operations, misguide strategies, and erode the trust of both customers and stakeholders. Data quality dashboards have emerged as indispensable tools, offering a clear window into the health of their data and enabling targeted actionable improvements. However, not all data quality dashboards are created equal.

Banking 87

Banking Data Datasets Retail 87

KDnuggets

NOVEMBER 19, 2024

Interested in data analytics? Here's a list of Python libraries you cannot do without.

Python 130

Python Data Analytics Data 130

databricks

NOVEMBER 11, 2024

Databricks Marketplace is an open marketplace for data, analytics, and AI, powered by the open-source Delta Sharing standard. Since the release of Databricks.

Data Analytics 113

Data Analytics Data 113

Seattle Data Guy

NOVEMBER 13, 2024

Much of the data we have used for analysis in traditional enterprises has been structured data. It’s easy for humans to break down, understand, and, in turn, find insights from it. However, much of the data that is being created and will be created comes in some form of unstructured format. However, the digital era… Read more The post What is Unstructured Data?

Unstructured Data 130

Unstructured Data Process Structured Data Data 130

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Cloudera

NOVEMBER 13, 2024

We are excited to announce the acquisition of Octopai , a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Cloudera’s mission since its inception has been to empower organizations to transform all their data to deliver trusted, valuable, and predictive insights.

Metadata 91

Metadata Management Data Governance Government 91

Knowledge Hut

NOVEMBER 25, 2024

Scrum is a quality-driven process for producing excellent business outcomes. Organizations are looking for professional product owners that grasp this notion and can use it in the real world. Employers use many credentialing services to certify levels of comprehension and application by level, which are referred to as belts. Scrum training sessions, along with resources like a PSPO study guide, assist you in learning PSPO I principles, studying efficiently and effectively to pass your exam, adva

Certification 97

Certification Business Analyst Consulting Education 97

DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. The Medallion architecture is a design pattern that helps data teams organize data processing and storage into three distinct layers, often called Bronze, Silver, and Gold.

Architecture 81

Architecture Raw Data Pipeline-centric Data Ingestion 81

KDnuggets

NOVEMBER 27, 2024

Terrified of calculus but dream of being a data scientist? Breathe easy! Discover the surprising truth about math in data science and how you can succeed without being a math genius.

Data Science 127

Data Science Data 127

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

databricks

NOVEMBER 5, 2024

We’re excited to announce that materialized views (MVs) and streaming tables (STs) are now Generally Available in Databricks SQL on AWS and Azure.

SQL 109

SQL AWS Engineering 109

Precisely

NOVEMBER 18, 2024

Key Takeaways: Data integrity is required for AI initiatives, better decision-making, and more – but data trust is on the decline. Data quality and data governance are the top data integrity challenges, and priorities. A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies.

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Cloudera

NOVEMBER 15, 2024

Today, cyber defenders face an unprecedented set of challenges as they work to secure and protect their organizations. In fact, according to the Identity Theft Resource Center (ITRC) Annual Data Breach Report , there were 2,365 cyber attacks in 2023 with more than 300 million victims, and a 72% increase in data breaches since 2021. The constant barrage of increasingly sophisticated cyberattacks has left many professionals feeling overwhelmed and burned out.

Metadata 85

Metadata Unstructured Data Data Lake Government 85

Snowflake

NOVEMBER 21, 2024

As travelers increasingly expect personalized experiences, brands in the travel and hospitality industry must find innovative ways to leverage data in their marketing and product experiences. That said, managing vast, complex data sets across multiple brands, loyalty programs and guest touchpoints presents unique challenges for companies in this industry.

Hospitality 79

Hospitality Entertainment Data Governance Government 79

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems