This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookie Settings
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookie Settings
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Strictly Necessary: Used for the proper function of the website
Performance/Analytics: Used for monitoring website traffic and interactions
Since I started working in tech, one goal that kept coming up was workflow automation. Whether automating a report or setting up retraining pipelines for machine learning models, the idea was always the same: do less manual work and get more consistent results. But automation isnt just for analytics. RevOps teams want to streamline processes… Read more The post Best Automation Tools In 2025 for Data Pipelines, Integrations, and More appeared first on Seattle Data Guy.
Despite the best efforts of many ML teams, most models still never make it to production due to disparate tooling, which often leads to fragmented data and ML pipelines and complex infrastructure management. Snowflake has continuously focused on making it easier and faster for customers to bring advanced models into production. In 2024, we launched over 200 AI features, including a full suite of end-to-end ML features in Snowflake ML , our integrated set of capabilities for machine learning mode
Open source has played an essential role in the tech industry and beyond. Whether in the AI/ML, web, or mobile space, our open source community grew and evolved while connecting people worldwide. At Meta Open Source , 2024 was a year of growth and transformation. Our open source initiatives addressed the evolving needs and challenges of developerspowering breakthroughs in AI and enabling the creation of innovative, user-focused applications and experiences.
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
While the technology continues to be a male-dominated industry, more women are pursuing careers in the space, driving meaningful change and innovation. At Precisely, recognizing the impact that women have in tech and championing their contributions is a top priority. To support this, the Precisely Women in Technology (PWIT) network, was created as a dedicated place for women to connect, share experiences, and learn from one another.
Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew. The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems.
Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew. The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems.
Introducing Apache Airflow® 3.0 Be among the first to see Airflow 3.0 in action and get your questions answered directly by the Astronomer team. You won't want to miss this live event on April 23rd! Save Your Spot → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community.
During the InferESG project we made a pivotal decision to create an alternative architecture, one that sits parallel to the agentic framework used for the conversational part of the system. This decision came about from discussions with the client, and their needs to analyse and process company sustainability reports, evaluate them and compare them to relevant materiality topics.
Read Time: 3 Minute, 9 Second Snowpark Magic: Auto-Create Tables from S3 Folders In modern data lakes, its common for departments like Finance, Marketing, Sales, etc., to continuously drop data files into their respective folders within an S3 bucket. These files often arrive in CSV format, and over time, teams request new folders or refresh their data.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Let's discover what small language models (SLMs) are, how they can be used in RAG systems and applications, and when to use them over their large language counterparts.
What is Steganography? It is the practice of concealing information within ordinary files or media, making the hidden data undetectable to anyone unaware of its presence. From ancient techniques like invisible ink to modern digital methods that embed messages in images, audio, or network traffic, it has evolved significantly. While it is widely used for data protection and digital watermarking, cybercriminals also exploit it to hide malware and evade detection.
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Application engineers situated at the beginning of a data pipeline ("left side") should apply data contracts and products as rigorously as the data engineers further down the line.
Mobile GraphQL is a framework used at Meta for fetching data in mobile applications using GraphQL , a strongly-typed, declarative query language. At Meta it handles data fetching for apps like Facebook and Instagram. Sabrina, a software engineer on Metas Mobile GraphQL Platform Team, joins Pascal Hartig on the Meta Tech podcast to discuss the evolution and future of GraphQL.
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Improving data quality can feel overwhelming. There are so many things to fix and so many processes to improve. Where do you even start? Surprisingly, the answer may come from an unusual placeflossing your teeth. Start Small to Build Better Habits In the book Tiny Habits by B.J. Fogg, the author suggests a simple way to build habits. If you want to start flossing your teeth, dont aim for a perfect routine right away.
To solve the AI silo problem, enterprises need a shared communication layer for AI agentsa real-time, event-driven approach that lets agents share intelligence and take coordinated action.
Discover how Snowflake's ABM team achieved a 2.3x lift in meetings booked and a 54% increase in CTR by using Snowflake AI for targeted campaigns and more personalized messaging while optimizing both budget and engagement. Weve seen how Snowflake AI tools are transforming outcomes for our customers. From saving 4,000 hours a year on manual email intake to treating more patients in emergency rooms to saving 75% of costs , AI in Snowflake is making a real impact on businesses around the world.
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Since our launch on Google Cloud Platform (GCP) in 2021, Databricks on Google Cloud has provided more than 1,500 joint customers with a tightly integrated.
Artificial intelligence (AI) and machine learning (ML) are transforming the way the world works by enabling smarter, faster, and more automated decision-making. However, one of the challenges that have emerged as AI systems evolve is the issue of AI/ML hallucinationsoutputs generated by models that are plausible but incorrect, which can undermine the reliability of AI systems.
Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.
For many organizations across industries, the era of experimental AI has given way to the era of practical implementation. Even those companies still testing and evaluating AI solutions are shifting away from the art of the possible to focus more closely on what will soon produce measurable ROI. It will no longer be enough for your organization to merely use AI to win the approval of company leadership, says Samuel Lee, Product Marketing Director for Financial Services at Snowflake.
Willis Towers Watson (WTW) is a multinational company that provides a wide range of services in commercial insurance brokerage, risk management, employee benefits, and actuarial.
As organizations generate and manage more digital content, having a strong digital archiving strategy is essential. Regulations continue to change, customer expectations continue to grow, and businesses must balance accessibility with security. A poorly managed archiving system can lead to compliance risks, data silos, and inefficiencies that slow down operations.
Generative AI is changing how we generate content, solve problems, and engage with technology. Unlike typical AI models, which classify or forecast based on incoming data, generative AI generates new content—whether text, images, music, or even code. This technology generates human-like outputs by utilizing advanced machine learning techniques such as deep learning and neural networks.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
We organize all of the trending information in your field so you don't have to. Join 37,000+ users and stay up to date on the latest articles your peers are reading.
You know about us, now we want to get to know you!
Let's personalize your content
Let's get even more personalized
We recognize your account from another site in our network, please click 'Send Email' below to continue with verifying your account and setting a password.
130 
Let's personalize your content