Confessions of a Data Guy

DECEMBER 1, 2022

Nothing captures the imagination and heart like a tale of betrayal and heartbreak, and that is a tale I want to bring to you today. It’s a tale of Databricks Workflows and Jobs, version changes, new features, API’s, and insidious little hidden gems that will make you pull your hair out when you find them. […] The post A Tale of Betrayal and Heartbreak – Databricks Workflows and Jobs. appeared first on Confessions of a Data Guy.

Data 130

Data IT Big Data Data Engineering 130

KDnuggets

DECEMBER 1, 2022

In this blog, I shared my story on getting 4 data science job offers including Airbnb, Lyft and Twitter after being laid off. Any data scientist who was laid off due to the pandemic or who is actively looking for a data science position can find something here to which they can relate.

Data Science 160

Data Science Data 160

Confluent

DECEMBER 2, 2022

ksqlDB use case: see how apps can use ksqlDB to ingest, filter, enrich, aggregate, and query data directly with Kafka—no complex architectures or data stores needed.

Kafka 144

Kafka Building Architecture Data 144

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The most expensive part of working with massive data sets is the work of retrieving and processing the files that contain the raw information. FeatureBase (formerly Pilosa) avoids that overhead by converting the data into bitmaps. In this episode Matt Jaffee explains how to model your data as bitmaps and the benefits that this representation provides for fast aggregate computation.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Teradata

NOVEMBER 30, 2022

Teradata has joined the Amazon SageMaker Ready Program which differentiates Teradata as an AWS Partner Network member with a product that works with Amazon SageMaker & fully supports AWS customers.

Programming 97

Programming Designing AWS 97

KDnuggets

DECEMBER 2, 2022

The data science field is full of job opportunities, yet there is still a lot of confusion about what data scientists actually do. This confusion is largely due to the many myths that exist about the role of a data scientist. In this article, we will bust the top 10 myths about data science. By the end of this article, you will have a better understanding of the role of a data scientist and what it takes to be one.

Data Science 159

Data Science Data IT 159

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. The Arrow project is designed to eliminate wasted effort in translating between languages, and Voltron Data was created to help grow and support its technology and community.

Data Process 100

Data Process Process Metadata Business Intelligence 100

DoorDash Engineering

NOVEMBER 29, 2022

As DoorDash’s business grows, engineers strive for a better network infrastructure to ensure more third-party services could be integrated into our system while keeping data securely transmitted. Due to security and compliance concerns, some vendors handling such sensitive data cannot expose services to the public Internet and therefore host their own on-premise data centers.

Cloud 97

Cloud AWS Amazon Web Services Data 97

KDnuggets

NOVEMBER 30, 2022

The best way to learn Data Science is by solving real-world problems with the data and building your own portfolio. In this article, we will discuss three projects that you can work on to build your portfolio and impress interviewers.

Data Science 120

Data Science Project Portfolio Data 120

Confluent

DECEMBER 2, 2022

Major improvements to the Kafka consumer, Streams, and ksqlDB for incremental cooperative rebalancing while maintaining at-least-once and exactly-once guarantees.

Kafka 138

Kafka Process 138

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. For analytic applications to properly leverage a hybrid, multi-cloud ecosystem to support modern data architectures, data observability has become even more important. I spoke to Mark Ramsey of Ramsey International (RI) to dive deeper into that last subject.

Data Lake 84

Data Lake Data Pipeline Analytics Application Data Governance 84

Engineering at Meta

NOVEMBER 30, 2022

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis at scale. Executing SQL queries against our data warehouse is important to the workflows of many engineers and data scientists at Meta for analytics and monitoring use cases

SQL 71

SQL Data Warehouse Coding Metadata 71

KDnuggets

DECEMBER 1, 2022

The latest KDnuggets exclusive cheatsheet covers the essentials of machine learning with Scikit-learn.

Machine Learning 152

Machine Learning 152

Confluent

NOVEMBER 29, 2022

Apache Kafka’s Streams API embeds Machine Learning into any app or microservice (Java, Docker, Kubernetes, etc.) to add business value.

Machine Learning 145

Machine Learning Kafka Java Building 145

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Cloudera

NOVEMBER 30, 2022

What is CDP Operational Database (COD). CDP Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to Getting Started with Cloudera Data Platform Operational Database (COD).

Database 82

Database Datasets NoSQL Big Data 82

Booking.com Engineering

DECEMBER 2, 2022

Booking.com’s mission is to make it easier for everyone to experience the world. To help people discover destinations, we are a leading travel advertiser on Google Pay Per Click (PPC). Booking Holdings, as a whole, spent $4.7 billion in marketing across all brands in the first nine months of 2022[1]. How do we run PPC at our scale, and efficiently? In this article, we want to illustrate our extensive use of the public cloud, specifically Google Cloud Platform (GCP).

Systems 52

Systems Cloud MySQL Relational Database 52

KDnuggets

NOVEMBER 29, 2022

First steps to learning data science & machine learning are the foundations.

Machine Learning 160

Machine Learning Data Science Data 160

Confluent

DECEMBER 2, 2022

Build fast, break nothing. Learn about the unique challenges Confluent's engineering team has faced building ksqlDB and continuously shipping the latest, greatest features.

Building 124

Building Engineering Process 124

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Cloudera

NOVEMBER 30, 2022

What is Cloudera Operational Database (COD). Cloudera Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to our article Getting Started with Cloudera Data Platform Operational Database (COD).

Database 72

Database Java Coding Building 72

Pinterest Engineering

DECEMBER 2, 2022

Grey Skold | (former Android Video Engineer) ; Lin Wang | Android Performance Engineer; Sheng Liu | Android Performance Engineer Pinterest Android App offers a rare experience with a mix of images and videos on a two-column grid. In order to maintain a performant video experience on Android devices, we focused on: Warming up Configurations Pooling players Warming Up In order to reduce the startup latency, we establish a video network connection by sending a dummy HTTP HEAD request during the ear

Media 52

Media Engineering Utilities Architecture 52

KDnuggets

DECEMBER 1, 2022

Introduction to PyTorch Lightning and how it can be used for the model building process. It also provides a brief overview of the PyTorch characteristics and how they are different from TensorFlow.

Building 108

Building Process IT Machine Learning 108

Confluent

DECEMBER 2, 2022

Datadog and Confluent integration brings new monitoring, metrics, and enterprise capabilities for Kafka. Monitor Kafka Connect, ksqlDB, Schema Registry, REST Proxy, and more.

Kafka 119

Kafka 119

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project

U-Next

DECEMBER 2, 2022

Introduction To Summarizing Data In Excel . Data for Excel is a way of summarizing large amounts of data into a few numbers. For example, if you have 3,000 sales at $50 each, you could summarize this by saying that total sales were $150,000. The summarization of data in Excel is doable in many ways. . Approximately 54% of businesses use Excel , which doesn’t include any other spreadsheet programs.

Data 52

Data Datasets Programming Data Analysis 52

Scott Logic

NOVEMBER 30, 2022

An introduction to Markdown Markdown is a brilliant tool for quickly writing up universally accessible documents. Created by John Gruber and Aaron Schwartz in 2004, it stands as one of the most popular and widely used markup languages around. It uses simple and intuitive formatting that can be easily read and understood. “A Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions” John Gruber, creator of

Programming Language 52

Programming Language Utilities Designing Coding 52

KDnuggets

NOVEMBER 29, 2022

What is Chebychev's Theorem and How Does it Apply to Data Science? • How to Select Rows and Columns in Pandas Using [ ],loc, iloc,at and.iat • Linux for Data Science Cheatsheet • How Much Math Do You Need in Data Science? • Git for Data Science Cheatsheet.

Data Science 108

Data Science IT Data 108

Confluent

NOVEMBER 30, 2022

What is stream processing, or complex event processing (CEP), and how does it work? Learn about real-time data and event stream analytics in this tutorial.

Process 126

Process Data IT 126

Speaker: Evelyn Chou

Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.

Business Intelligence

Tweag

NOVEMBER 30, 2022

There is an inherent beauty to be found in simple, pervasive ideas that shift our perspective on familiar objects. Such ideas can help tame the complexity of abstruse abstractions by offering a more intuitive angle from which to understand them. The aim of this post is to present an alternative angle — that of interactive semantics — from which to view one of the fundamental notion of functional programming: higher-order functions.

Programming Language 52

Programming Language Programming Computer Science Coding 52

Ascend.io

DECEMBER 1, 2022

Data migration is one of the most common undertakings for data teams. Yet, many businesses underestimate the process—resulting in extra time and money spent. A data migration process usually takes longer than it should and requires several teams. In addition, it is highly visible to both users and executives. How can you keep from making the same mistake?

Process 52

Process Data Database ETL Tools 52

KDnuggets

NOVEMBER 28, 2022

Everything you need to know to start your career in Data Engineering.

Data Engineering 158

Data Engineering Data Engineer Engineering Data 158

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government