KDnuggets

SEPTEMBER 11, 2024

Kickstart your data analyst career with all these free courses.

Data Analytics 151

Data Analytics Data Data Science 151

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Most importantly, these pipelines enable your team to transform data into actionable insights, demonstrating tangible business value. According to an IBM study, businesses expect that fast data will enable them to “make better informed decisions using insights from analytics (44%), improved data quality and

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

KDnuggets

SEPTEMBER 11, 2024

Access a pre-built Python environment with free GPUs, persistent storage, and large RAM. These Cloud IDEs include AI code assistants and numerous plugins for a fast and efficient development experience.

Cloud 139

Cloud Data Science Python Coding 139

databricks

SEPTEMBER 11, 2024

We are excited to announce that Databricks was named one of the 2024 Fortune Best Workplaces in Technology™. This award reflects our.

Technology 78

Technology 78

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

KDnuggets

SEPTEMBER 11, 2024

Learn how to create a data science pipeline with a complete structure.

Data Science 136

Data Science Data Ingestion Process Data 136

Tweag

SEPTEMBER 11, 2024

We’ve all been there: wasting a couple of days on a silly bug. Good news for you: formal methods have never been easier to leverage. In this post, I will discuss the contributions I made during my internship to Liquid Haskell (LH), a tool that makes proving that your Haskell code is correct a piece of cake. LH lets you write contracts for your functions inside your Haskell code.

Coding 70

Coding IT Programming Accessibility 70

Confluent

SEPTEMBER 11, 2024

To make application testing for topics with schemas easier, you can now produce messages that are serialized with schemas using the Confluent Cloud Console UI.

Cloud 69

Cloud 69

databricks

SEPTEMBER 11, 2024

Personal Access Tokens (PATs) are a convenient way to access services like Azure Databricks or Azure DevOps without logging in with your password.

Accessibility 69

Accessibility Accessible Data Science Engineering 69

Towards Data Science

SEPTEMBER 11, 2024

One answer and many best practices for how larger organizations can operationalizing data quality programs for modern data platforms An answer to “who does what” for enterprise data quality. Image courtesy of the author. I’ve spoken with dozens of enterprise data professionals at the world’s largest corporations, and one of the most common data quality questions is, “who does what?

Government 64

Government Data Engineer Data Engineering Data 64

Scott Logic

SEPTEMBER 11, 2024

In February of this year, Scott Logic announced our proposed Technology Carbon Standard , setting out an approach to describing an organisation’s technology footprint. This standard has proven invaluable in mapping our own carbon footprint, as well as those of clients we’ve worked with. As awareness of the environmental impact of digital infrastructure grows, it has become crucial to understand and manage technology-related emissions.

Technology 52

Technology Accessibility Accessible Project 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Cloudera

SEPTEMBER 11, 2024

Imagine a world where your sensitive data moves effortlessly between clouds – secure, private, and far from the prying eyes of the public internet. Today, we’re making that world a reality with the launch of Cloudera Private Link Network. Organizations are continuously seeking ways to enhance their data security. One of the challenges is ensuring that data remains protected as it traverses different cloud environments.

Pharmaceutical 52

Pharmaceutical Cloud Pipeline-centric Healthcare 52

Hevo

SEPTEMBER 11, 2024

ETL tools have become important in efficiently handling integrated data. In this blog, we will discuss Fivetran vs AWS Glue, two influential ETL tools on the market. This will help you gain a comprehensive understanding of the product’s features, pricing models, and real-world use cases, helping you choose the right solution.

ETL Tools 52

ETL Tools AWS Data 52

Cloudyard

SEPTEMBER 11, 2024

Read Time: 1 Minute, 52 Second In this use case, a financial services company has decided to migrate its data warehouse from Oracle to Snowflake. The migration involves not only migrating the data from Oracle to Snowflake but also replicating all views in Snowflake. After successfully migrating several views, the data engineering team noticed discrepancy between the Oracle view definitions and their Snowflake counterparts.

Data Warehouse 52

Data Warehouse Data Engineer Data Engineering Engineering 52

Hevo

SEPTEMBER 11, 2024

Data pipelines and workflows have become an inherent part of the advancements in data engineering, machine learning, and DevOps processes. With ever-increasing scales and complexity, the need to orchestrate these workflows efficiently arises. That is where Apache Airflow steps in —an open-source platform designed to programmatically author, schedule, and monitor workflows.

Architecture 52

Architecture Data Pipeline Machine Learning Data Engineer 52

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Hevo

SEPTEMBER 11, 2024

1GB of data was referred to as big data in 1999. Nowadays, the term is used for petabytes or even exabytes of data (1024 Petabytes), close to trillions of records from billions of people. In this fast-moving landscape, the key to making a difference is picking up the correct data storage solution for your business.

Data Lake 40

Data Lake Data Warehouse Database Data Storage 40

Hevo

SEPTEMBER 11, 2024

A Data Pipeline is an indispensable part of a data engineering workflow. It enables the extraction, transformation, and storage of data across disparate data sources and ensures that the right data is available at the right time.

Data Pipeline 40

Data Pipeline Python Coding Data Engineer 40

Hevo

SEPTEMBER 11, 2024

We’re excited to announce that Hevo Data has achieved the prestigious Snowflake Ready Technology Validation certification! This recognition solidifies our commitment to delivering top-notch data integration solutions that seamlessly work with Snowflake, a leading AI Data Cloud. What is the Snowflake Ready Technology Validation Program?

Technology 40

Technology Certification Programming Data Integration 40