KDnuggets

SEPTEMBER 12, 2024

Develop unique yet standing-out data science projects to improve your data portfolio.

Data Science 156

Data Science Project Portfolio Data 156

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge. Picture this: thus, unlike many other extensions that require deep setup and constant coding, […] The post Setup Mage AI with Postgres to Build and Manage Your Data Pipeline appeared first on Analytics Vidhy

Data Pipeline 144

Data Pipeline Building Management Data 144

Confluent

SEPTEMBER 9, 2024

Confluent has acquired WarpStream, an innovative Kafka-compatible streaming solution. Read the full statement by Jay Kreps, co-founder and CEO of Confluent.

Cloud 142

Cloud Kafka 142

databricks

SEPTEMBER 10, 2024

We are excited to share the latest new features and performance improvements that make Databricks SQL simpler, faster and lower cost than ever.

SQL 128

SQL 128

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

KDnuggets

SEPTEMBER 12, 2024

The GitHub repository includes up-to-date learning resources, research papers, guides, popular tools, tutorials, projects, and datasets.

Datasets 151

Datasets Project 151

Engineering at Meta

SEPTEMBER 10, 2024

We’re sharing more about the role that reinforcement learning plays in helping us optimize our data centers’ environmental controls. Our reinforcement learning-based approach has helped us reduce energy consumption and water usage across various weather conditions. Meta is revamping its new data center design to optimize for artificial intelligence and the same methodology will be applicable for future data center optimizations as well.

Data 124

Data Designing Systems Building 124

databricks

SEPTEMBER 9, 2024

Personalization and scale have historically been mutually exclusive. For all the talk of one-to-one marketing and hyper-personalization , the reality has been that.

Building 105

Building Retail 105

KDnuggets

SEPTEMBER 11, 2024

Kickstart your data analyst career with all these free courses.

Data Analytics 151

Data Analytics Data Data Science 151

Christophe Blefari

SEPTEMBER 13, 2024

Back to work ( credits ) Hey you, can you believe it's already September? This year has been flying. It feels like I just blinked, and here we are. In August, I've been focusing mainly on my next big journey—if you follow me on LinkedIn, you might have caught a sneak peek! I'll be making a full announcement next week. I want to take the time to explain my thought process and ideas behind it.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Netflix Tech

SEPTEMBER 10, 2024

By Jose Fernandez , Sebastien Dabdoub , Jason Koch , Artem Tkachuk The Compute and Performance Engineering teams at Netflix regularly investigate performance issues in our multi-tenant environment. The first step is determining whether the problem originates from the application or the underlying infrastructure. One issue that often complicates this process is the "noisy neighbor" problem.

Utilities 100

Utilities Coding Programming Java 100

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

databricks

SEPTEMBER 9, 2024

Imagine giving your business an intelligent bot to talk to customers. Chatbots are commonly used to talk to customers and provide them with.

100

100

KDnuggets

SEPTEMBER 9, 2024

Exploring the not-so-famous data science libraries that can be useful in your data workflow.

Data Science 149

Data Science Python Data Workflow Data 149

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Most importantly, these pipelines enable your team to transform data into actionable insights, demonstrating tangible business value. According to an IBM study, businesses expect that fast data will enable them to “make better informed decisions using insights from analytics (44%), improved data quality and

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Netflix Tech

SEPTEMBER 10, 2024

By Karthik Yagna , Baskar Odayarkoil , and Alex Ellis Pushy is Netflix’s WebSocket server that maintains persistent WebSocket connections with devices running the Netflix application. This allows data to be sent to the device from backend services on demand, without the need for continually polling requests from the device. Over the last few years, Pushy has seen tremendous growth, evolving from its role as a best-effort message delivery service to be an integral part of the Netflix ecosystem.

Metadata 93

Metadata Coding Transportation Systems 93

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Snowflake

SEPTEMBER 12, 2024

The Snowflake World Tour is making 23 stops around the globe, so you can learn about the latest innovations in the AI Data Cloud in a city near you. This tour will cover Snowflake’s latest advancements that can help you accelerate AI and application development in your organization while advancing the data foundation that makes it all possible. This includes new capabilities related to Snowflake Cortex, streaming, Iceberg open table formats and more.

Data Lake 86

Data Lake Government Cloud Machine Learning 86

KDnuggets

SEPTEMBER 10, 2024

Learn about machine learning APIs for datasets, models, web applications, free GPUs, and text, audio, and image generation.

Machine Learning 144

Machine Learning Datasets 144

databricks

SEPTEMBER 9, 2024

We are excited to introduce several powerful new capabilities to Mosaic AI Gateway, designed to help our customers accelerate their AI initiatives with.

Government 84

Government Designing Machine Learning 84

Confluent

SEPTEMBER 10, 2024

Engineers can put generative AI to work to improve the quality of their data, allowing them to build more accurate and trustworthy AI-powered applications.

Data 72

Data Engineering Building 72

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

SEPTEMBER 13, 2024

Snowflake has always been committed to helping customers protect their accounts and data. To further our commitment to protect against cybersecurity threats and to champion the advancement of industry standards for security, Snowflake recently signed the Cybersecurity and Infrastructure Security Agency (CISA) Secure By Design Pledge. In line with CISA’s Secure By Design principles, we recently announced a number of security enhancements in the platform — most notably the general availability of

Consulting 75

Consulting Designing IT Data 75

KDnuggets

SEPTEMBER 13, 2024

Learn how to use the OpenAI o1-preview & o1-mini for decision-making, coding, and building an end-to-end machine learning project from scratch.

Machine Learning 141

Machine Learning Coding Project Building 141

databricks

SEPTEMBER 11, 2024

We are excited to announce that Databricks was named one of the 2024 Fortune Best Workplaces in Technology™. This award reflects our.

Technology 78

Technology 78

Tweag

SEPTEMBER 11, 2024

We’ve all been there: wasting a couple of days on a silly bug. Good news for you: formal methods have never been easier to leverage. In this post, I will discuss the contributions I made during my internship to Liquid Haskell (LH), a tool that makes proving that your Haskell code is correct a piece of cake. LH lets you write contracts for your functions inside your Haskell code.

Coding 70

Coding IT Programming Accessible 70

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Confluent

SEPTEMBER 11, 2024

To make application testing for topics with schemas easier, you can now produce messages that are serialized with schemas using the Confluent Cloud Console UI.

Cloud 69

Cloud 69

KDnuggets

SEPTEMBER 12, 2024

In order for Lakehouse to become a unified data layer for both analytics and AI, it needs to be extended with new capabilities

IT 139

IT Data 139

databricks

SEPTEMBER 10, 2024

We are excited to announce a set of enhanced capabilities for the Databricks Visual Studio Code Extension: Easily set up your projects built.

Coding 72

Coding Project 72

DataKitchen

SEPTEMBER 9, 2024

From Cattle to Clarity: Visualizing Thousands of Data Pipelines with Violin Charts Most data teams work with a dozen or a hundred pipelines in production. What do you do when you have thousands of data pipelines in production? How do you understand what is happening to those pipelines? Is there a way that you can visualize what is happening in production quickly and easily?

Data Pipeline 69

Data Pipeline Data Cloud IT 69

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Towards Data Science

SEPTEMBER 11, 2024

One answer and many best practices for how larger organizations can operationalizing data quality programs for modern data platforms An answer to “who does what” for enterprise data quality. Image courtesy of the author. I’ve spoken with dozens of enterprise data professionals at the world’s largest corporations, and one of the most common data quality questions is, “who does what?

Government 64

Government Data Engineering Data Engineer Data 64

KDnuggets

SEPTEMBER 11, 2024

Access a pre-built Python environment with free GPUs, persistent storage, and large RAM. These Cloud IDEs include AI code assistants and numerous plugins for a fast and efficient development experience.

Cloud 139

Cloud Data Science Python Coding 139

databricks

SEPTEMBER 11, 2024

Personal Access Tokens (PATs) are a convenient way to access services like Azure Databricks or Azure DevOps without logging in with your password.

Accessible 69

Accessible Accessibility Data Science Engineering 69

Precisely

SEPTEMBER 9, 2024

Key Takeaways: • Implement effective data quality management (DQM) to support the data accuracy, trustworthiness, and reliability you need for stronger analytics and decision-making. • Embrace automation to streamline data quality processes like profiling and standardization. • Develop standardized processes to quickly identify and fix data issues, maintaining integrity and compliance.

Management 64

Management High Quality Data Structured Data Data Lake 64

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud