databricks

APRIL 18, 2024

We are excited to partner with Meta to release the latest state-of-the-art large language model, Meta Llama 3 , on Databricks. With Llama.

Building 143

Building Data Science Data 143

KDnuggets

APRIL 18, 2024

Join us on a journey of becoming a professional MLOps engineer by mastering essential tools, frameworks, key concepts, and processes in the field.

Engineering 136

Engineering Process 136

Waitingforcode

APRIL 18, 2024

Streaming jobs are supposed to run continuously but it applies to the data processing logic. After all, sometimes you may need to release a new job package with upgraded dependencies or improved business logic. What happens then?

Data Process 130

Data Process Process IT Data 130

KDnuggets

APRIL 18, 2024

Learning a new skill can be expensive, but it doesn’t have to be.

IT 126

IT 126

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

Cloudera

APRIL 18, 2024

Artificial intelligence (AI) is something that, by its very nature, can be surrounded by a sea of skepticism but also excitement and optimism when it comes to harnessing its power. With the arrival of the latest AI-powered technologies like large language models (LLMs) and generative AI (GenAI), there’s a vast amount of opportunities for innovation, growth, and improved business outcomes right around the corner.

Metadata 110

Metadata Government Data Governance Data 110

Snowflake

APRIL 18, 2024

Legacy security information and event management (SIEM) solutions, like Splunk, are powerful tools for managing and analyzing machine-generated data. They have become indispensable for organizations worldwide, particularly for security teams. But as much as security operation center (SOC) analysts have come to rely on solutions like Splunk, there is one complaint that comes up for some: Costs can quickly add up.

Data Lake 95

Data Lake Data Ingestion Bytes Cloud Computing 95

Confluent

APRIL 18, 2024

Learn how to use kafka-docker-composer, a simple tool to create a docker-compose.yml file for failover testing, to understand cluster settings like Kraft, and for app development.

Kafka 69

Kafka 69

Scott Logic

APRIL 18, 2024

In this episode, Oliver Cronk, Andrew Carr and David Hope talk about the ever-changing world of data, with conversations moving from data warehouse to data lake, and data mesh to data fabric. They discuss the importance of data ownership and common tooling, and their view that data mesh is an approach rather than an architecture.

Data Lake 59

Data Lake Data Warehouse Data Architecture Architecture 59

Confluent

APRIL 18, 2024

Siemens and Brose revolutionized global manufacturing operations with Confluent’s data streaming, increasing its IoT integration and the future of Industry 4.0.

Manufacturing 59

Manufacturing Data IT 59

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Scott Logic

APRIL 18, 2024

The Internal App Portal Project What do you do when you’re a software consultancy that uses small, internally developed applications, and you need to be able to spin those applications up on demand? You build something! Part of life at a consultancy means stints on the bench, allowing us to participate in internal projects. One of those projects is the ‘Internal Application Portal’ (IAP).

AWS 52

AWS Building Consulting Architecture 52

Cloudyard

APRIL 18, 2024

Read Time: 2 Minute, 59 Second Consider a Scenario where the Client manages a large Customer-Invoice details with constantly changing Invoice data. The company uses Snowflake to manage Invoice data in a table named S_INVOICE. This table receives daily loads(history + Current) to reflect changes in customer and invoice partial payment. The goal is to efficiently merge this dynamic data into a target table named S_INVOICE_TARGET while performing Insert operations using stored procedure.

Utilities 52

Utilities Management IT Data 52

RandomTrees

APRIL 18, 2024

In today’s dynamic business landscape, data integration has become a critical component for enterprises to derive meaningful insights and make informed decisions. Among the various tools available for data integration, Informatica and Talend stand out as popular choices, each with its strengths and capabilities. However, migrating from one platform to another can be a daunting task, especially when it involves converting existing code.

Coding 52

Coding Retail Metadata Python 52

DataKitchen

APRIL 18, 2024

Introducing DataKitchen’s Open Source Data Observability Software Today, we announce that we have open-sourced two complete, feature-rich products that solve the data observability problem: DataOps Observervability and DataOps TestGen. With these two products, you will know if your pipelines are running without error and on time and can finally trust your data.

Data Analytics 49

Data Analytics Data SQL Software Engineer 49

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

Cloudera

APRIL 18, 2024

Regulations often get a bad rap. You may have heard the old idiom “cut the red tape” which means to circumvent obstacles like regulations or bureaucracy. But in many – if not most )– cases the underlying need for regulations outweighs the burden of compliance. In the financial sector, regulations are essential for financial institutions to maintain stability by preventing excessive risk-taking, ensuring adequate capitalization and reducing the likelihood of failures or financial crises.

Government 76

Government Data Security Cloud Banking 76