Precisely

OCTOBER 5, 2023

How much data is your business generating each day? While answers will vary by organization, chances are there’s one commonality: it’s more data than ever before. But what do you do with all that data? According to the 2023 Data Integrity Trends and Insights Report , published in partnership between Precisely and Drexel University’s LeBow College of Business, 77% of data and analytics professionals say data-driven decision-making is the top goal of their data programs.

Raw Data 69

Raw Data Insurance Datasets Telecommunication 69

Analytics Vidhya

OCTOBER 6, 2023

In today’s rapidly evolving digital landscape, seamless data, applications, and device integration are more pressing than ever. Enter Microsoft Fabric, a cutting-edge solution designed to revolutionize how we interact with technology. This article will explore the key features and benefits, identify the ideal users for this solution, and guide you on when and how to […] The post Introduction of Microsoft Fabric appeared first on Analytics Vidhya.

Designing 261

Designing Technology Data Lake Business Intelligence 261

Marc Lamberti

OCTOBER 4, 2023

Are you looking for a way to choose one task or another? Do you want to execute a task based on a condition? Do you have multiple tasks, but only one should be executed if a criterion is valid? You’ve come to the right place! The BranchPythonOperator does precisely what you are looking for. It’s common to have DAGs with different execution flows, and you want to follow only one, depending on a value or a condition.

Python 246

Python Data Pipeline Machine Learning IT 246

Jesse Anderson

OCTOBER 5, 2023

Confluent had their Current Conference (Videos: day one and day two ). There were many announcements that both technologists and investors need to know about. Confluent had two moats (replication and Confluent Cloud), and now they anticipate three moats (replication, Confluent Cloud, serverless Flink). As expected with a vendor conference, there is a lot of marketing from the stage.

Kafka 195

Kafka Finance Cloud Designing 195

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI.

Building 162

Building BI SQL Machine Learning 162

Marc Lamberti

OCTOBER 1, 2023

Airflow Sensors are one of the most common tasks in data pipelines. Why? Because a Sensor waits for a condition to be true to complete. Do you need to wait for a file? Check if an SQL entry exists? Delay the execution of a DAG? That’s the few possibilities of the Airflow Sensors. If you want to make complex and robust data pipelines, you have to understand how Sensors work genuinely.

Data Pipeline 243

Data Pipeline SQL Algorithm Coding 243

Confessions of a Data Guy

OCTOBER 6, 2023

Is there anything more Chad than Apache Airflow … and Rust? I think not you whimp. What two things do I love most? At the moment Rust and Airflow are at least somewhere at the top of that list. I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack.

Data Engineering 148

Data Engineering Data Engineer Engineering Data 148

Waitingforcode

OCTOBER 4, 2023

Do not get the title wrong! Having applyInPandasWithState in the PySpark API is huge! However, due to Python duck typing, some operations are more difficult and more risky to express in the code than in the strongly typed Scala API.

Scala 147

Scala Python Coding 147

KDnuggets

OCTOBER 6, 2023

Imagine you’re allowed to do only three data science projects. Which should you choose to guarantee you get the job? Here’s my choice!

Data Science 149

Data Science Project Data Machine Learning 149

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Ripple Engineering

OCTOBER 5, 2023

Overview In the rippled 1.12.0 release, the AMM amendment stands out as a significant feature in both size and scope. Since September 2022, the RippleX performance team has collaborated closely with the engineering team responsible for the AMM feature implementation. This report presents a thorough overview of our testing approach, findings, and key takeaways.

AWS 144

AWS BI Designing Database 144

ArcGIS

OCTOBER 4, 2023

Learn how the ArcGIS Utility Network is ready to use without spending a significant amount of time configuring or customizing.

Utilities 135

Utilities Data Management Management Data 135

LinkedIn Engineering

OCTOBER 5, 2023

Think of how many times a day you use some type of search functionality across your devices and applications to discover information, find a contact, or a new job opportunity. The truth is we all depend on the ability to search for things online, and finding the right match to the information, organization, or to a job that maps to your skills and interests makes all the difference in our experiences and the knowledge we can gain.

IT 133

IT Metadata Designing Cloud 133

KDnuggets

OCTOBER 5, 2023

Will AI be able to surpass human intelligence? An article going through the current progression, and challenges of AGI.

148

148

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Marc Lamberti

OCTOBER 2, 2023

Airflow Variables are easy to use but easy to misuse as well. In this tutorial, you will learn everything you need about variables in Apache Airflow. What are they, how do they work, define one, get the value, and more. If you followed my course “Apache Airflow: The Hands-On Guide” variables shouldn’t sound unfamiliar. This time, I will give you all I know about variables so that, in the end, you will be ready to use Variables in your DAGs properly.

AWS 130

AWS Google Cloud Database Coding 130

Confessions of a Data Guy

OCTOBER 1, 2023

So perhaps you’re thinking it’s time to use Rust on your next project. You’ll find plenty of primers on how to get your feet wet in the language (and if you somehow made it this far without that much, The Book is that starting point), but maybe you’re feeling a bit lost amidst the seas […] The post Introduction to using Rust Libraries (cargo and crates) appeared first on Confessions of a Data Guy.

Project 130

Project IT Data Data Engineering 130

Snowflake

OCTOBER 6, 2023

At Snowflake, we’re committed to helping customers effectively manage and optimize spend. To this effect, we’re excited to launch the public preview of Budgets on AWS today, which enables customers to set spending limits and receive notifications for Snowflake credit usage for either their entire Snowflake account or for a custom group of resources within an account.

Retail 126

Retail AWS Management Database 126

KDnuggets

OCTOBER 6, 2023

Learn how to present complex information effectively with data visualization.

Data 147

Data Data Science 147

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Marc Lamberti

OCTOBER 2, 2023

Airflow Variables are easy to use but easy to misuse as well. In this tutorial, you will learn everything you need about variables in Apache Airflow. What are they, how do they work, define one, get the value, and more. If you followed my course “Apache Airflow: The Hands-On Guide” variables shouldn’t sound unfamiliar. This time, I will give you all I know about variables so that, in the end, you will be ready to use Variables in your DAGs properly.

AWS 130

AWS Google Cloud Database Coding 130

Engineering at Meta

OCTOBER 5, 2023

Python 3.12 is out! It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 marks a milestone in our efforts to make our work developing and scaling Python for Meta’s use cases more accessible to the broader Python community.

Python 121

Python Coding Programming Language Programming 121

databricks

OCTOBER 5, 2023

Have you ever deployed an AI model, only to discover it's delivering unexpected results in a real-world setting? Monitoring models is as crucial.

IT 118

IT 118

KDnuggets

OCTOBER 1, 2023

Explore the essentials of Google Cloud Platform for data science and ML, from account setup to model deployment, with hands-on project examples.

Google Cloud 145

Google Cloud Cloud Data Science Project 145

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

Snowflake

OCTOBER 2, 2023

Today’s consumer expects a personalized, relevant, end-to-end customer experience. Delivering this level of engagement can drive transformational growth, but it requires a new level of sophistication and a deep understanding of the customer. Data fuels that understanding, and the holy grail for companies is to achieve a holistic view of the customer and their journey.

Cloud 114

Cloud Building Insurance Data Engineering 114

ThoughtSpot

OCTOBER 4, 2023

This blog is part of our ongoing ‘Why I joined ThoughtSpot’ series, where we profile Spotters from around the world to learn who they are and why they chose a career at ThoughtSpot. Jeff Depa recently joined ThoughtSpot as Chief Revenue Officer, and is based out of Austin, Texas. In this role, Jeff will contribute to ThoughtSpot’s strategic growth and revenue goals by maximizing profit through go to market strategies that address the entire customer lifecycle.

Hospitality 111

Hospitality Technology Programming Cloud 111

databricks

OCTOBER 3, 2023

This blog was written in collaboration with Sukh Sekhon, Software Engineer, Cloud Infrastructure and Helen Li, Sr. Director of Engineering at Exai Bio.

Software Engineer 118

Software Engineer Software Engineering Engineering Cloud 118

KDnuggets

OCTOBER 5, 2023

Build an irresistible portfolio that hooks recruiters with these 5 free platforms - you won't believe how easy it is!

Portfolio 144

Portfolio Building Data Science Recruitment 144

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Pinterest Engineering

OCTOBER 2, 2023

Each summer, Pinterest welcomes Software Engineering Pinterns who spend 12 weeks with us creating impact within our product and teams. While Pinterns are fully immersed in their teams throughout the summer, they also get to attend exciting activities and events hosted by the University Recruiting team and within the company. Here’s a quick recap from this summer: Social events were a hit with boba tea making, creating your own vision board, chocolate making and a virtual escape room.

Recruitment 109

Recruitment Software Engineer Software Engineering Engineering 109

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime. Turnkey data pipelines replication and failover Snowflake provides a best-in-class experience for data engineering workloads.

Data Pipeline 101

Data Pipeline Management Data Ingestion Data 101

databricks

OCTOBER 5, 2023

This blog post was written in collaboration with Eric Schwartz, Director of Partnerships at Ribbon Health, and David Kulwin, Director, Databricks Marketplace. Ensuring.

110

110