Start Data Engineering
JULY 26, 2024
1. Introduction 2. Project overview 3. Check your data before making it available to end-users; Write-Audit-Publish(WAP) pattern 4. TL;DR: How the greatexpectations library works 4.1. greatexpectations quick setup 5. From an implementation perspective, there are four types of tests 5.1. Running checks on one dataset 5.2. Checks involving the current dataset and its historical data 5.3.
Confessions of a Data Guy
JULY 24, 2024
I’ve had something rattling around in the old noggin for a while; it’s just another strange idea that I can’t quite shake out. We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format. So, […] The post PyArrow vs Polars (vs DuckDB) for Data Pipelines. appeared first on Confessions of a Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Christophe Blefari
JULY 26, 2024
Tallinn ( credits ) Dear members, it's Summer Data News, the only news you can consume by the pool, the beach or at the office—if you're not lucky. This week, I'm writing from the Baltics, nomading a bit in Eastern and Northern Europe. I'm pleased to announce that we have successfully closed the CfP for Forward Data Conf, we received nearly 100 submissions and the program committee is currently reviewing all submissions.
databricks
JULY 23, 2024
In this blog, we are excited to share Databricks's journey in migrating to Unity Catalog for enhanced data governance. We'll discuss our high-level strategy and the tools we developed to facilitate the migration. Our goal is to highlight the benefits of Unity Catalog and make you feel confident about transitioning to it.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
KDnuggets
JULY 26, 2024
Discover how Bayesian thinking transforms decision-making with its unique approach to updating initial beliefs with new evidence.
Data Engineering Weekly
JULY 21, 2024
Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. But how can we choose a tool for the specific need? The traditional evaluation of running PoC on all the selected vendor tools is time-consuming and practically unviable for growth-driven companies. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
databricks
JULY 23, 2024
We are excited to partner with Meta to release the Llama 3.1 series of models on Databricks, further advancing the standard of powerful.
KDnuggets
JULY 24, 2024
Setup the environment, load the data, perform data analysis and visualization, and create the data pipeline all using Julia programming language.
Robinhood
JULY 25, 2024
Today, we are excited to launch joint investing accounts, which allow customers to seamlessly manage investments with their partner while keeping their shared assets in one place. Joint accounts make investing more collaborative for families and loved ones, providing shared access for account holders that allows them to combine funds and increase their investment power as they work towards their financial goals.
Cloudera
JULY 24, 2024
Cloudera recently unveiled the latest version of Cloudera Private Cloud Base with the Zero Downtime Upgrade (ZDU) feature to enhance your user experience. The goal of ZDU is to make upgrades simpler for you and your stakeholders by increasing the availability of Cloudera’s services. How Do You Keep IT Infrastructure (and Buses) Running and Avoid Downtime?
Speaker: Christophe Louvion, Chief Product & Technology Officer of NRC Health and Tony Karrer, CTO at Aggregage
Christophe Louvion, Chief Product & Technology Officer of NRC Health, is here to take us through how he guided his company's recent experience of getting from concept to launch and sales of products within 90 days. In this exclusive webinar, Christophe will cover key aspects of his journey, including: LLM Development & Quick Wins 🤖 Understand how LLMs differ from traditional software, identifying opportunities for rapid development and deployment.
databricks
JULY 22, 2024
Evaluating long-form LLM outputs quickly and accurately is critical for rapid AI development. As a result, many developers wish to deploy LLM-as-judge methods.
KDnuggets
JULY 23, 2024
This collection of tutorials from our sister site Statology center on data visualization. Learn more about visualizing your data right here.
Confluent
JULY 23, 2024
CC 2024 Q2 adds Flink Private Networking (AWS), Flink SQL Interactive Tables; Enterprise:Connect w/Confluent, Connector Custom Offsets; SI: Build w/Confluent, etc.
Uber Engineering
JULY 26, 2024
From WWDC reveal to delivery, discover how we tackled new tech, design challenges, and tight timelines to enhance rider & driver experiences with Live Activity® from Apple.
Speaker: Jamie Bernard
Success in product management goes beyond delivering great features - it’s about achieving measurable financial outcomes that resonate across the organization. By connecting your product’s journey with the company’s financial success, you’ll ensure that every feature, release, and innovation contributes to the bottom line, driving both customer satisfaction and business growth.
databricks
JULY 22, 2024
Today, we're thrilled to announce that Mosaic AI Model Training's support for fine-tuning GenAI models is now available in Public Preview. At Databricks.
KDnuggets
JULY 24, 2024
From the soft tools to the hard tools, these are what make a data scientist successful.
Precisely
JULY 26, 2024
Mainframes have one of the longest histories of any kind of computing technology that is still used today. In fact, mainframe history shows the fast-evolving landscape of technology, few innovations have left as profound a mark as mainframe computers. From their inception to the sophisticated systems of today, mainframes have continuously adapted to meet the ever-growing demands of business operations.
Hevo
JULY 25, 2024
A data lake is a central storage place for an organization’s data in its original format. Unlike data warehouses, data lakes can handle all kinds of data, including unstructured and semi-structured data like images, video, audio, and documents.
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
databricks
JULY 26, 2024
Introduction Time series forecasting serves as the foundation for inventory and demand management in most enterprises. Using data from past periods along with.
KDnuggets
JULY 23, 2024
Tired of staring at bland dataframes? Discover how conditional formatting in Pandas can transform your data visualization experience!
Precisely
JULY 24, 2024
Oracle’s recent decision to shut down its advertising business marks a significant shift in the ad tech landscape. This move, driven by declining revenues and increasing regulatory pressures, leaves many advertisers seeking alternative solutions to maintain their marketing momentum while leveraging data that adheres to privacy requirements and aligns with consumers’ expectations.
Hevo
JULY 25, 2024
In recent years, Apache Iceberg has seen considerable advancements that highlights its growing importance. Major tech companies like Google, Snowflake, and Databricks have increasingly embraced this table format. This trend, driven by major tech companies, highlights a transformative shift in the data warehousing landscape as Iceberg gains traction.
Advertisement
The complexity of financial data, the need for real-time insight, and the demand for user-friendly visualizations can seem daunting when it comes to analytics - but there is an easier way. With Logi Symphony, we aim to turn these challenges into opportunities. Our platform empowers you to seamlessly integrate advanced data analytics, generative AI, data visualization, and pixel-perfect reporting into your applications, transforming raw data into actionable insights.
databricks
JULY 24, 2024
Dataricks is thrilled to announce the General Availability (GA) of Primary Key (PK) and Foreign Key (FK) constraints, starting in Databricks Runtime 15.2.
KDnuggets
JULY 22, 2024
Let's learn to use Pandas pivot_table in Python to perform advance data summarization
phData: Data Engineering
JULY 24, 2024
Effective marketing is crucial for business growth, yet achieving cost-effective and impactful results from marketing can be challenging for companies of all sizes. Marketing leaders are tasked with driving results and determining the best course of action for their team by asking questions like: How much should we spend on this new campaign? Should we focus on retaining our customers or trying to find new ones?
Hevo
JULY 24, 2024
Searching for data manually in Snowflake can be very challenging, time-consuming and sometimes frustrating. Snowflake identifies these problems and has developed Universal Search to change the way we search for data. The universal search, built on a powerful snowflake cortex, is designed to make finding data straightforward.
Speaker: Maher Hanafi, VP of Engineering at Betterworks & Tony Karrer, CTO at Aggregage
Executive leaders and board members are pushing their teams to adopt Generative AI to gain a competitive edge, save money, and otherwise take advantage of the promise of this new era of artificial intelligence. There's no question that it is challenging to figure out where to focus and how to advance when it’s a new field that is evolving everyday. 💡 This new webinar featuring Maher Hanafi, VP of Engineering at Betterworks, will explore a practical framework to transform Generative AI pr
databricks
JULY 23, 2024
The communications industry is experiencing immense change due to rapid technological advancements and evolving market trends. Communications service providers (CSP) build various solutions.
KDnuggets
JULY 22, 2024
Transfer learning can improve model performance by leveraging pre-trained models and adapting them to new, related tasks.
Striim
JULY 23, 2024
Real-time data and machine learning are revolutionizing how hospitals operate and deliver care. By adopting a data-driven approach to hospital optimization, healthcare professionals’ jobs become more efficient, allowing them to focus more on what truly matters: Patient health. Not to mention, hospital operation optimization reduces costs. Here’s everything you need to know about how hospitals can leverage advancements of machine learning in healthcare to streamline operations and moderniz
Hevo
JULY 24, 2024
While working with huge amounts of data, Data serialization plays an important role in the performance of the system. Data Serialization converts complex data structures, such as graphs, trees, etc., into a format that can be easily stored or transmitted over the network or across different distributed systems and programming languages.
Speaker: Conrado Morlan
In this engaging and witty talk, industry expert Conrado Morlan will explore how artificial intelligence can transform the daily tasks of product managers into streamlined, efficient processes. Using the lens of a superhero narrative, he’ll uncover how AI can be the ultimate sidekick, aiding in data management and reporting, enhancing productivity, and boosting innovation.
Expert insights. Personalized for you.
208 
Let's personalize your content