KDnuggets

NOVEMBER 30, 2023

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

Data Engineering 134

Data Engineering Data Engineer Engineering Data Workflow 134

Analytics Vidhya

NOVEMBER 26, 2023

In this Leading with Data episode, explore the analytics landscape with Dr. Swati Jain, a seasoned leader boasting over two decades of experience. From her unforeseen foray into analytics to steering EXL Analytics’ India business, Dr. Jain imparts invaluable insights into the ever-evolving world of data science. Read on to know more about her career, […] The post Unlocking the Power of Analytics with Dr.

Data Science 213

Data Science Data Analysis Data Data Engineering 213

Confessions of a Data Guy

NOVEMBER 25, 2023

Ok. Get off your high horse. You are human just like the rest of us. Just like your ancient ancestors who were throwing rocks and sticks at each other a thousand years ago … you are looking for a leg up on the competition. Isn’t that the world we live in? At the end of […] The post How to be Better Than Everyone Else appeared first on Confessions of a Data Guy.

Data 148

Data Data Engineering Data Engineer Engineering 148

Jesse Anderson

NOVEMBER 30, 2023

Lately, I’ve been learning how to trade options. Although there’s data and programming involved in options trading, it isn’t as technical as data engineering or software engineering. However, it reflects the current state of learning, whether that’s data engineering or options trading. It gave me a look into learning a skill using videos. Each lesson I learned will directly apply to your learning or skill improvement.

Data Engineering 147

Data Engineering Data Engineer Algorithm Engineering 147

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

KDnuggets

DECEMBER 1, 2023

The blog covers machine learning courses, bootcamps, books, tools, interview questions, cheat sheets, MLOps platforms, and more to master ML and secure your dream job.

Machine Learning 159

Machine Learning 159

databricks

NOVEMBER 28, 2023

Defining what a data culture is can vary by organization. A data culture is the shared values, attitudes, and behaviors that enable organizations.

Building 133

Building Data 133

Confluent

NOVEMBER 27, 2023

The top 7 free online courses, tutorials, get started guides, and examples for the easiest way to learn Apache Kafka.

Kafka 132

Kafka 132

KDnuggets

DECEMBER 1, 2023

Image by Author When you are getting started with machine learning, logistic regression is one of the first algorithms you’ll add to your toolbox.

Python 158

Python Building Machine Learning Algorithm 158

Waitingforcode

NOVEMBER 28, 2023

In March I wrote a blog showing how to use accumulators to know the application of each filter statement. Turns out, the solution may not be perfect as mentioned by Aravind in one of the comments. I bet you already have an idea but if not, keep reading. Everything will be clear in the end!

130

130

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Seattle Data Guy

NOVEMBER 28, 2023

Data warehousing would be easy if all data were structured and formatted in the data source. Maybe we wouldn’t even need to build a data warehouse. But as anyone who has worked with data from more than one source knows, that’s rarely the case. Businesses today need to pull data from a plethora of sources,… Read more The post Finding The Right ETL/ELT Solution – What Is Estuary And Should You Use It?

IT 130

IT Data Warehouse Building Data 130

Confluent

NOVEMBER 28, 2023

Learn how to write code that produces messages via librdkafka, how it will behave during error situations, and how your application should detect and respond to them.

Coding 131

Coding IT Kafka 131

KDnuggets

DECEMBER 1, 2023

Curious about optimizing AI for everyday devices? Dive into the complete overview of MIT's TinyML and Efficient Deep Learning Computing course. Explore strategies to make AI smarter on small devices. Read the full article for an in-depth look!

Deep Learning 154

Deep Learning Machine Learning 154

ArcGIS

NOVEMBER 30, 2023

Learn how to filter coordinate systems based on a spatial extent, GCS, or projection property.

Project 128

Project Systems 128

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Seattle Data Guy

NOVEMBER 27, 2023

If you’re a data engineer, then you’ve likely at least heard of Airflow. Apache Airflow is one of the most popular open-source workflow orchestration solutions that gets used for data pipelines. This is what spurred me to write the article “Should You Use Airflow” because there are plenty of people who don’t enjoy Airflow or… Read more The post Common Pitfalls in Deploying Airflow for Data Teams appeared first on Seattle Data Guy.

Data Pipeline 130

Data Pipeline Data Engineering Data Engineer Data 130

Tweag

NOVEMBER 27, 2023

Sponsored by Antithesis (distributed systems reliability testing experts), I’ve developed a new library to filter local files in Nix which I’d like to introduce! This post requires some familiarity with Nix and its language. So if you don’t know what Nix is yet, take a look first, it’s pretty neat. In this post we’re going to look at what source filtering is, why it’s useful, why a new library was needed for it, and the basics of the new library.

Building 122

Building Systems IT Project 122

KDnuggets

NOVEMBER 29, 2023

From ANI to AGI and Beyond: Deciphering AI's Evolutionary Path.

154

154

ArcGIS

NOVEMBER 28, 2023

New in ArcGIS Pro 3.

Government 123

Government 123

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

databricks

NOVEMBER 30, 2023

We are excited to introduce five new integrations in Databricks Partner Connect—a one-stop portal enabling you to use partner solutions with your Databricks D.

119

119

Cloudera

DECEMBER 1, 2023

Recent Government Initiatives on Public Sector AI Solutions In recent years, governments across the globe have recognized the transformative potential of artificial intelligence (AI) and have embarked on initiatives to harness this technology to drive innovation and serve their citizens more effectively. These government-led efforts have had a profound impact on the development and adoption of AI solutions in the public sector, paving the way for a future where data-driven decision-making and au

Building 109

Building Government Transportation Data Governance 109

KDnuggets

NOVEMBER 29, 2023

Probability is one of the foundational elements of computer science. Some bootcamps will skim over the topic, however, it is integral to your computer science knowledge.

Computer Science 150

Computer Science IT Data Science Data 150

ArcGIS

NOVEMBER 27, 2023

A suite of ArcGIS Solutions to support common workflows in the stormwater industry.

Utilities 120

Utilities Government Architecture Engineering 120

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

databricks

NOVEMBER 29, 2023

Background: Modernizing Data Delivery Today's enterprise data estates are vastly different from 10 years ago. Industries have transitioned their analytics from monolithic data.

Healthcare 118

Healthcare Government Data 118

Lyft Engineering

NOVEMBER 29, 2023

Written by Ritesh Varyani and Jeana Choi at Lyft. Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data.

Kafka 105

Kafka Data Ingestion Datasets Architecture 105

KDnuggets

NOVEMBER 28, 2023

Want to support the behavior of built-in functions and method calls in your Python classes? Magic methods in Python let you do just that! So let’s uncover the method behind the magic.

Python 149

Python 149

ArcGIS

DECEMBER 1, 2023

Learn how to resolve error 00374: Unique numeric IDs are not assigned.

116

116

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

databricks

NOVEMBER 28, 2023

We’re thrilled to share that Databricks has won the AWS ISV Partner of the Year award for North America. This award recognizes top I.

AWS 108

AWS 108

Precisely

NOVEMBER 30, 2023

Today’s customers expect businesses to engage with them on their own terms. They want companies to anticipate their needs, personalize offerings to their individual preferences, and present them with multiple ways to interact with the brand. That requires a deep, contextual understanding of each customer’s behaviors, intentions, and wishes. It often means predicting what the customer wants, even before they go looking for it.

Electronics 103

Electronics Management Technology Government 103

KDnuggets

NOVEMBER 30, 2023

The blog discusses five platforms designed for data scientists with specialized capabilities in managing large datasets, models, workflows, and collaboration beyond what GitHub offers.

Data Science 142

Data Science Project Datasets Data 142