Start Data Engineering

DECEMBER 13, 2023

1. Introduction 2. Setup 3. Ways to uplevel your dbt workflow 3.1. Reproducible environment 3.1.1. A virtual environment with Poetry 3.1.2. Use Docker to run your warehouse locally 3.2. Reduce feedback loop time when developing locally 3.2.1. Run only required dbt objects with selectors 3.2.2. Use prod datasets to build dev models with defer 3.2.3. Parallelize model building by increasing thread count 3.

Datasets 130

Datasets Building 130

Waitingforcode

DECEMBER 12, 2023

One week later than initially announced, but here it is, the second part for Data+AI Summit 2023 retrospective. I don't know how, but I managed to include some streaming-related talks here too!

Data 130

Data Management IT 130

Data Engineering Podcast

DECEMBER 10, 2023

Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Simon Späti

DECEMBER 10, 2023

In my journey, detailed in why Vim is more than an editor , I’ve discovered the profound impact of integrating Vim and its motions into my entire computer workflow. This evolution, from using familiar tools like Notepad++ and SQL Server Management Studio to embracing Vim, represents a significant shift in how I approach tasks in data engineering and writing.

SQL 130

SQL Data Engineer Data Engineering Engineering 130

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Confluent

DECEMBER 12, 2023

Dive into the serverless architecture of Confluent Cloud for Apache Flink and explore its benefits like reduced infrastructure costs, increased reliability, & seamless adoption.

Architecture 126

Architecture Cloud IT 126

Jesse Anderson

DECEMBER 12, 2023

What better year to start the Christmas season than to drop a new episode of Unapologetically Technical! In this episode, I interview Stephane Derosiaux from Conduktor. We talk about his time evolving architectures and creating real-time systems at Auchan (grocery) and Adeo/Leroy Merlin (Home Improvement). We discuss the issues of British food and how to find good food in London.

Food 100

Food Kafka Architecture Systems 100

databricks

DECEMBER 11, 2023

Following the announcements we made last week about Retrieval Augmented Generation (RAG), we're excited to announce major updates to Model Serving. Databricks Model.

Building 120

Building Engineering 120

Confluent

DECEMBER 12, 2023

Telcos use Confluent with event-driven microservices to enable real-time communications with 3rd-party field service providers, fulfilling customer service requests more efficiently.

110

110

Netflix Tech

DECEMBER 14, 2023

Holden Karau Elizabeth Stone Pedro Duarte Chris Stephens Pallavi Phadnis Lee Woodridge Mark Cho Guil Pires Sujay Jain Tristan Reid Senthilnathan Athinarayanan Bharath Mummadisetty Abhinaya Shetty Judit Lantos Amanuel Kahsay Dao Mi Mick Dreeling Chris Colburn and Agata Gryzbek Introduction Earlier this summer Netflix held our first-ever Data Engineering Forum.

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

KDnuggets

DECEMBER 15, 2023

You learned how to generate images using the base model, how to upgrade to the Stable Diffusion XL model to improve image quality, and how to use a custom model to generate high quality portraits.

129

129

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

databricks

DECEMBER 12, 2023

Introduction Databricks Lakehouse Monitoring allows you to monitor all your data pipelines – from data to features to ML models – without additional too.

Data Pipeline 124

Data Pipeline Data Data Science Data Engineering 124

Snowflake

DECEMBER 13, 2023

Our recently released predictions report includes a number of important considerations about the likely trajectory of cybercrime in the coming years, and the strategies and tactics that will evolve in response. Every year, the story is “Attackers are getting more sophisticated, and defenders have to keep up.” As we enter a new era of advanced AI technology, we identify some surprising wrinkles to that perennial trend.

Machine Learning 103

Machine Learning Technology Building IT 103

ArcGIS

DECEMBER 12, 2023

In this Part-1 of a two-part blog series, we will share tips for labeling objects on images for object detection deep learning models.

Deep Learning 115

Deep Learning 115

KDnuggets

DECEMBER 13, 2023

This article is about the less common data science skills that can help you get hired. While these skills are not as common as they are for technical jobs, they are certainly worth developing.

Data Science 128

Data Science Data 128

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

databricks

DECEMBER 14, 2023

As CEO of the North Pole, Santa Claus oversees one of the world’s most complicated supply chain, manufacturing and logistics operations. Every year, S.

Manufacturing 120

Manufacturing Data 120

Precisely

DECEMBER 13, 2023

Companies that excel at targeted messaging will generally outperform their peers both in terms of revenue growth and customer loyalty. Digital marketing is ideally suited for precise targeting and rapid feedback, provided that business users have access to the detailed demographic and geospatial data they need. Most businesses do not tap into the full potential of digital marketing automation tools.

Raw Data 96

Raw Data Entertainment Data Validation Education 96

ArcGIS

DECEMBER 13, 2023

In ArcGIS Pro 3.2, the field map parameter has been redesigned for improved usability and new capabilities.

Management 124

Management 124

KDnuggets

DECEMBER 12, 2023

Whether you're a seasoned ML engineer or a new LLM developer, these tools will help you get more productive and accelerate the development and deployment of your AI projects.

Building 127

Building Project Engineering 127

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

databricks

DECEMBER 13, 2023

Background In an era where Retrieval-Augmented Generation (RAG) is revolutionizing the way we interact with AI-driven applications, ensuring the efficiency and effectiveness of.

Data Science 110

Data Science Engineering Data 110

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Let’s dive into all new releases in September, October and November. Architecture Flexibility Iceberg Tables – public preview While many customers value the simplicity of fully managed storage and a single, mul

Metadata 94

Metadata Python AWS Government 94

DoorDash Engineering

DECEMBER 12, 2023

The DoorDash retail shopping experience mission seeks to combine the best parts of in-person shopping with the power of personalization. While shopping in a physical store has its advantages, a brick-and-mortar store cannot be personalized – the onus is on the consumer to navigate aisles to find what they need. Conversely, a digital shopping experience can be highly personalized.

Retail 93

Retail Architecture Building Process 93

KDnuggets

DECEMBER 11, 2023

Welcome back to the KDnuggets’ "Back to Basics" series. This is the BONUS week and we will dive into learning about deploying to the cloud.

Cloud 125

Cloud Data Engineering Data Engineer Engineering 125

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud

Cloudera

DECEMBER 13, 2023

During Week of Giving Clouderans across the globe took time out of their busy schedules to give back and support causes meaningful to them. For many colleagues, however, giving and volunteering during Week of Giving is just one of the many ways they support the causes meaningful to them. We had the privilege of sitting down with Remus Lim, Regional VP of Sales in APAC who not only volunteered alongside his Singapore-based colleagues during Week of Giving but is dedicating an upcoming trip to phi

IT 89

IT 89

Snowflake

DECEMBER 11, 2023

You need metrics to do your job well as a marketer but getting clear, meaningful metrics is a huge challenge. While digital advertisers and paid media professionals are on the hook to build ample sales pipeline and maximize return on ad spend (ROAS), they’re also expected to deliver personalized advertising content while navigating evolving privacy requirements and adhering to consumer expectations—all while extracting insights from siloed ad platforms.

Building 93

Building Cloud Media Algorithm 93

Towards Data Science

DECEMBER 15, 2023

For a data scientist, there’s no such thing as too much data. But when we take a broader look at the organizational context, we have to balance our goals with other considerations. Photo by Trnava University on Unsplash Data Science vs Security/IT: A Battle for the Ages Acquiring and keeping data is the focus of a huge amount of our mental energy as data scientists.

Machine Learning 89

Machine Learning Data Science Data Security Data Storage 89

KDnuggets

DECEMBER 14, 2023

Looking for the best resources to learn Python programming? Check out these free university courses.

Python 142

Python Programming 142

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Cloudera

DECEMBER 13, 2023

Legal & General Investment Management (LGIM) is one of the largest global asset managers, managing £1.2 trillion on behalf of savers, retirees, and institutions worldwide. LGIM prides itself on being a responsible investor and is at the forefront of global index fund management and pension investment. Its strategies cover a broad array of asset classes and styles, including equities, bonds, property and alternatives, as well as multi-asset funds.

Pipeline-centric 87

Pipeline-centric Professional Services Datasets BI 87

ArcGIS

DECEMBER 14, 2023

Learn about our recently released pretrained deep learning models available in the ArcGIS Living Atlas of the World.

Deep Learning 108

Deep Learning 108

databricks

DECEMBER 14, 2023

Adopting AI is existentially vital for most businesses Machine Learning (ML) and generative AI (GenAI) are revolutionizing the future of work. Organizations understand.

Management 90

Management Machine Learning Data 90

KDnuggets

DECEMBER 14, 2023

This article is a brief overview of the field of Emotion AI, and the potential applications of its technology in intimate roles.

Technology 130

Technology IT 130

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project