Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration for everyone involved due to complex workflows and hard-to-understand dashboards. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Start Data Engineering

MAY 28, 2024

1. Introduction 2. Project demo 3. TL;DR 4. Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline 219

Data Pipeline Python Building Data 219

The Pragmatic Engineer

MAY 1, 2024

This article was published on 14 March 2024 in The Pragmatic Engineer, for subscribers. I'm sharing this piece in public more than a month later, as it provides important context and analysis for the AI dev tools space. Subscribe to The Pragmatic Engineer to stay up-to-date on what is happening with software engineering, Big Tech, and startups.

Software Engineer 196

Software Engineer Software Engineering Programming Language Media 196

Robinhood

MAY 30, 2024

Robinhood Crypto customers in the United States can now use our API to view crypto market data, manage portfolios and account information, and place crypto orders programmatically Today, we are excited to announce the Robinhood Crypto trading API , ushering in a new era of convenience, efficiency, and strategy for our most seasoned crypto traders. Robinhood Crypto customers in the United States can use our new trading API to set up advanced and automated trading strategies that allow them to st

Insurance 141

Insurance Portfolio Algorithm Coding 141

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

KDnuggets

MAY 22, 2024

We are all looking for the right opportunities in our career. In the landscape of data-related careers, the roles can be grouped into classes, and future opportunities tend to follow natural migration paths between the class groups.

Data 151

Data 151

Rock the JVM

MAY 22, 2024

by Herbert Kateu 1. Introduction This article is a follow-up to the websocket article that was published previously. To recap, we created an in-memory chat application using WebSockets with the help of the Http4s library. The chat application had a variety of features implemented through commands directly in the chat window such as the ability to create users, create chat rooms, and switch between chat rooms.

Scala 136

Scala PostgreSQL Database SQL 136

Start Data Engineering

MAY 21, 2024

1. Introduction 2. TL;DR 3. Enabling Stakeholder data access with RAGs 3.1. Set up 3.1.1. Pre-requisite 3.1.2. Demo 3.1.3. Key terminology 3.2. Loading: Read raw data and convert them into LlamaIndex data structures 3.2.1. Read data from structured and unstructured sources 3.2.2. Transform data into LlamaIndex data structures 3.3. Indexing: Generate & store numerical representation of your data 3.

Accessibility 214

Accessibility Accessible SQL Raw Data 214

databricks

MAY 30, 2024

Salesforce and Databricks are excited to announce an expanded strategic partnership that delivers a powerful new integration - Salesforce Bring Your Own Model.

138

138

Confluent

MAY 1, 2024

Confluent Cloud Freight clusters are now available in Early Access. In this blog, learn how Freight clusters can save you up to 90% at GBps+ scale.

Cloud 145

Cloud Accessibility Accessible 145

KDnuggets

MAY 14, 2024

Want to learn machine learning from the best of resources? Check out these free machine learning courses from the top universities of the world.

Machine Learning 151

Machine Learning 151

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

MAY 22, 2024

Accelerating enterprise AI use cases into production is now a board-level priority for most companies. However, one of the key challenges in AI today is ensuring that those use cases are ready for real-life use and continue to perform at a high level in production. Not only must enterprises ensure accurate, reliable, and valuable results they must also address and mitigate critical issues like bias, hallucinations, and toxicity.

Cloud 125

Cloud Data Governance Technology Machine Learning 125

ArcGIS

MAY 16, 2024

A modern take on the London tube map with updated accessible colours, a re-classification of lines by type, and line symbols scaled by frequency

Designing 134

Designing Accessibility Accessible 134

Data Engineering Weekly

MAY 3, 2024

The tweet received strong reactions on LinkedIn and Twitter. To clarify, I quoted it as a Notebook-style development, but it is not exactly a Notebook. There is a lot of context missing in that tweet, so I decided to write a blog about it. People have reservations about using tools like Jupytor Notebook for the production pipeline for a good reason.

Programming Language 117

Programming Language ETL Tools Data Pipeline Coding 117

databricks

MAY 22, 2024

We’re excited to announce the General Availability of Delta Lake Liquid Clustering in the Databricks Data Intelligence Platform. Liquid Clustering is an innovative.

Data 131

Data 131

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Tweag

MAY 1, 2024

tl;dr You may not believe it, but Nix documentation is getting better. Nixpkgs and NixOS still need more time. Table of contents Overview Motivation Statistics Retrospective Thoughts on future work Acknowledgements Overview This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024.

Architecture 110

Architecture Project Coding Designing 110

KDnuggets

MAY 28, 2024

Learning math is super important for data science. Check out these free courses from MIT to learn linear algebra, statistics, and more.

Data Science 150

Data Science Data 150

Snowflake

MAY 21, 2024

Today we’re excited to announce an expansion of our partnership with Microsoft to deliver a seamless and efficient interoperability experience between Snowflake and Microsoft Fabric OneLake, in preview later this year. This will enable our joint customers to experience bidirectional data access between Snowflake and Microsoft Fabric, with a single copy of data with OneLake in Fabric.

Metadata 125

Metadata Cloud Accessibility Accessible 125

ArcGIS

MAY 29, 2024

The latest release of ArcGIS Roads and Highways and ArcGIS Pipeline Referencing includes a variety of new and enhanced features.

Data Management 134

Data Management Management Data 134

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Engineering at Meta

MAY 22, 2024

Today, the internet (like most digital infrastructure in general) relies heavily on the security offered by public-key cryptosystems such as RSA, Diffie-Hellman (DH), and elliptic curve cryptography (ECC). But the advent of quantum computers has raised real questions about the long-term privacy of data exchanged over the internet. In the future, significant advances in quantum computing will make it possible for adversaries to decrypt stored data that was encrypted using today’s cryptosystems.

Bytes 105

Bytes Algorithm Coding Systems 105

databricks

MAY 3, 2024

Moving generative AI applications from the proof of concept stage into production requires control, reliability and data governance. Organizations are turning to open.

Data Governance 128

Data Governance Government Data 128

Cloudera

MAY 10, 2024

Apache Iceberg is vital to the work we do and the experience that the Cloudera platform delivers to our customers. Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time.

Government 102

Government Data Governance Data Pipeline Big Data 102

KDnuggets

MAY 31, 2024

SQL seems like a data science underdog compared to Python and R. However, it’s far from it. I’ll show you here how you can use it as a data scientist.

SQL 140

SQL Data Science Python Data 140

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Snowflake

MAY 2, 2024

To accurately answer business questions using LLMs, companies must augment models with their data. Retrieval Augmented Generation (RAG) is a popular solution to this problem, as it integrates the organization’s factual, real-time data into the prompt for the LLM. While the adoption of RAG has increased, an open question remains: How do enterprises know how effective their system is?

Cloud 118

Cloud Systems Building IT 118

Robinhood

MAY 6, 2024

Robinhood Markets,Inc.(Nasdaq:HOOD) today announced that RHC has received a Wells Notice from the SEC Staff Earlier today we announced Robinhood Crypto (RHC) has received a Wells Notice from the U.S. Securities and Exchange Commission staff indicating they will recommend that the Commission file an enforcement action. “After years of good faith attempts to work with the SEC for regulatory clarity including our well-known attempt to ‘come in and register,’ we are disappointed that the agency has

102

102

Knowledge Hut

MAY 30, 2024

Everybody sees a dream—aspiring to become a doctor, astronaut, or anything that fits your imagination. If you were someone who had a keen interest in looking for answers and knowing the “why” behind things, you might be a good fit for research. Further, if this interest revolved around computers and tech, you would be an excellent computer researcher!

Computer Science 98

Computer Science Data Mining Algorithm Machine Learning 98

databricks

MAY 19, 2024

We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing.

127

127

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project

Cloudera

MAY 9, 2024

The Gartner Data and Analytics Summit in London is quickly approaching on May 13 th to 15 th , and the Cloudera team is ready to hit the show floor! The theme of this year’s summit, “Generating Value Together: Creating Synergies between Data, Analytics & AI,” could not have come at a better time as we push forward on our AI and analytics journey together.

Banking 100

Banking Data Storage Data Analytics Cloud 100

KDnuggets

MAY 8, 2024

The goal of this article is to help demystify the process of selecting the proper machine learning algorithm, concentrating on "traditional" algorithms and offering some guidelines for choosing the best one for your application.

Machine Learning 128

Machine Learning Algorithm Process 128

Snowflake

MAY 7, 2024

Snowflake Cortex is a fully-managed service that enables access to industry-leading large language models (LLMs) is now generally available. You can use these LLMs in select regions directly via LLM Functions on Cortex so you can bring generative AI securely to your governed data. Your team can focus on building AI applications, while we handle model optimization and GPU infrastructure to deliver cost-effective performance.

Government 112

Government SQL Data Security Accessible 112

ArcGIS

MAY 31, 2024

Learn about the new Export Attachments geoprocessing tool in ArcGIS Pro 3.3 and how it simplifies the process of exporting attachments.

Process 113

Process IT Data Management Management 113

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud