Analytics Vidhya

DECEMBER 30, 2024

What will data engineering look like in 2025? How will generative AI shape the tools and processes Data Engineers rely on today? As the field evolves, Data Engineers are stepping into a future where innovation and efficiency take center stage. GenAI is already transforming how data is managed, analyzed, and utilized, paving the way for […] The post Top 11 GenAI Powered Data Engineering Tools to Follow in 2025 appeared first on Analytics Vidhya.

Data Engineering 143

Data Engineering Data Engineer Engineering Utilities 143

Engineering at Meta

DECEMBER 3, 2024

Meta releases a Request for Proposals (RFP) to identify nuclear energy developers to support AI innovation and clean and renewable energy goals.

Engineering 145

Engineering Data 145

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineering 143

Data Engineering Data Engineer Engineering Project 143

Data Engineering Weekly

DECEMBER 28, 2024

As we approach the new year, it's time to gaze into the crystal ball and ponder the future. In this post, we delve into predictions for 2025, focusing on the transformative role of AI agents, workforce dynamics, and data platforms. Join Ananth Packkildurai, Ashwin Ashish, and Rajesh as they unravel the future and guide us through the fascinating changes ahead.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Jesse Anderson

DECEMBER 25, 2024

We’re down to the last month of the year, but we’re never leaving without a new episode of the Unapologetically Technical! In this episode, I interview Frances Perry, the Head of Engineering at MotherDuck. Frances Perry is an engineering manager who spent many years as a heads-down coder creating various distributed systems used in Google and Google Cloud.

Google Cloud 130

Google Cloud Cloud Database Data Solutions 130

Confluent

DECEMBER 19, 2024

Queue support in Apache Kafka 4.0, enabled by share groups, lets you accommodate traditional queue-type workloads through cooperative consumption.

Kafka 131

Kafka Process 131

Seattle Data Guy

DECEMBER 12, 2024

Document Intelligence Studio is a data extraction tool that can pull unstructured data from diverse documents, including invoices, contracts, bank statements, pay stubs, and health insurance cards. The cloud-based tool from Microsoft Azure comes with several prebuilt models designed to extract data from popular document types. However, you can also use labeled datasets to train… Read more The post Alternatives to Azure Document Intelligence Studio: Exploring Powerful Document Analysis Tool

Insurance 130

Insurance Unstructured Data Banking Datasets 130

KDnuggets

DECEMBER 25, 2024

2024 has been yet another groundbreaking year for AI, with major breakthroughs, industry shifts, and ethical challenges shaping its future. Let's uncover together the key moments that defined AI this year about to finalize.

IT 127

IT 127

ArcGIS

DECEMBER 31, 2024

This blog provides step-by-step guidance to determine and use a silent install when configuring a driver to use Excel files in ArcGIS Pro. Learn More.

Data Management 119

Data Management Management Data 119

databricks

DECEMBER 20, 2024

We're excited to announce the Public Preview of credential vending for Unity Catalogs open APIs, allowing external clients to securely access Unity Catalog.

Accessible 113

Accessible Accessibility 113

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Confluent

DECEMBER 18, 2024

Real-time data streaming and messaging are essential for building scalable, resilient, event-driven microservices. Explore integrating the Micronaut framework with Confluent Cloud.

Cloud 115

Cloud Building Data 115

Netflix Tech

DECEMBER 19, 2024

David J. Berg * , David Casler ^, Romain Cledat * , Qian Huang * , Rui Lin * , Nissan Pow * , Nurcan Sonmez * , Shashank Srikanth * , Chaoying Wang * , Regina Wang * , Darin Yu * *: Model Development Team, Machine Learning Platform ^: Content Demand ModelingTeam A month ago at QConSF, we showcased how Netflix utilizes Metaflow to power a diverse set of ML and AI use cases , managing thousands of unique Metaflow flows.

Machine Learning 82

Machine Learning Data Warehouse Project Coding 82

Seattle Data Guy

DECEMBER 23, 2024

If you’re looking to pass hundreds of GBs of data quickly, you’re likely not going to use a REST API. That’s why every day, companies share data sets of users, patient claims, financial transactions, and more via SFTP. If youve been in the industry for a while, youve probably come across automated SFTP jobs that… Read more The post The Basics of SFTP: Authentication, Encryption, and File Management appeared first on Seattle Data Guy.

Management 130

Management Data Data Engineering Data Engineer 130

KDnuggets

DECEMBER 2, 2024

Learn reinforcement learning using free resources, including books, frameworks, courses, tutorials, example code, and projects.

Coding 134

Coding Project 134

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

ArcGIS

DECEMBER 9, 2024

Learn how to build a custom geoprocessing tool and about some new features, like Enable Undo for Script and Model tools, in ArcGIS Pro 3.

Building 115

Building Python 115

databricks

DECEMBER 23, 2024

AI remains at the forefront of every business leaders plans for 2025. Overall, 70% of businesses continue to believe AI is critical to.

Data 111

Data 111

Confluent

DECEMBER 23, 2024

Discover how data fuels Generative AI and why streaming data is key to success. Learn the fundamentals to unlock AIs true potential for your business.

Building 111

Building Engineering Data Process 111

Confessions of a Data Guy

DECEMBER 13, 2024

Today we talk about what is really going on with Data Contracts, they came in like a rocket a few years ago, but then died on the vine. What’s the deal? The post Data Contracts were a LIE! appeared first on Confessions of a Data Guy.

Data 100

Data Big Data Data Engineering Data Engineer 100

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Christophe Blefari

DECEMBER 13, 2024

Hey, it's been a few weeks since something has been published here—I hope you haven’t forgotten about me 😊 In the last weeks I've been all over the place and worked on a lot of topics except this newsletter, I've decided to take a break from the newsletter to catchup the rhythm in January! The Forward Data Conference was a huge success and I want to thanks again all the attendees, speakers, sponsors and my co-organisers.

Data 100

Data Building IT 100

KDnuggets

DECEMBER 26, 2024

A beginners guide to getting started with image captioning models with HuggingFace.

123

123

ArcGIS

DECEMBER 10, 2024

Explore Density tools in ArcGIS Pro for spatial data analysis to reveal hidden patterns and effective visualization to aid in informed decision-making.

Data Analysis 110

Data Analysis Data 110

databricks

DECEMBER 9, 2024

Our customers continue to shift from monolithic prompts with general-purpose models to specialized agent systems to achieve the quality needed to drive ROI.

Systems 113

Systems Data 113

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Scott Logic

DECEMBER 14, 2024

Advent of Code (AoC) is an annual, christmas-themed, coding competition that has been running for the past years and is something that I participate in at times. This year, while ~~subjecting myself to~~ learning Rust, I decided to see how OpenAIs latest model faired at the challenge. I quickly knocked together a script, and to my astonishment, found that o1-mini gave correct answers to all but one part of the first six days.

Coding 98

Coding Datasets Software Engineer Software Engineering 98

Snowflake

DECEMBER 12, 2024

As organizations mature in their execution of data and AI initiatives, a burning question remains: How do we measure the effectiveness of our teams and our impact on the business? This isnt the perennial Whats my data worth? dilemma often asked rhetorically and answered theoretically. Todays challenge is concrete: to define and track the metrics used to justify continued investment in data and AI innovation.

Pharmaceutical 91

Pharmaceutical Food Banking Insurance 91

Confessions of a Data Guy

DECEMBER 6, 2024

Well, everyone is abuzz with the recently announced S3 Tables that came out of AWS reinvent this year. I’m going to call fools gold on this one right out of the gate. I tried them out, in real life that is, not just some marketing buzz, and it will leave most people, not all, be […] The post AWS S3 Tables. Technical Introduction. appeared first on Confessions of a Data Guy.

AWS 130

AWS Data IT Big Data 130

KDnuggets

DECEMBER 2, 2024

In this article, we’ll go over Python libraries for tasks like logging, unit testing, data handling, and more — each with features that can simplify your application development.

Python 125

Python Data 125

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

ArcGIS

DECEMBER 9, 2024

Spatial Analytics and Data Science capabilities across ArcGIS have been enhanced this fall with new tools and optimized experiences.

Data Science 109

Data Science Data 109

databricks

DECEMBER 17, 2024

Were excited to announce the Public Preview of Query Git integration as part of the new SQL Editor. Git support for queries.

SQL 109

SQL 109

Seattle Data Guy

DECEMBER 3, 2024

When I broke into the data world, everyone wanted to hire data scientists that would let their companies become more data driven. There were statistics about the exabytes of data that we were creating and the value it would provide. However, a few years into my career, the data world started to make a pivot… Read more The post Preparing Your Data Infrastructure for 2025: Lessons from the Past, Strategies for the Future appeared first on Seattle Data Guy.

Data 130

Data IT 130

Cloudera

DECEMBER 19, 2024

AWS re:Invent is one of my favorite trade shows. It is one of the biggest technology conferences of the year and is an opportunity to have hundreds of conversations with customers and prospects, listen to their priorities and challenges, hopes, and give them a Cloudera tote bag or a pair of orange sunglasses. What follows is a collection of just a few things I learned and observed during my week in Las Vegas.

AWS 75

AWS Metadata Government Machine Learning 75

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems