The Pragmatic Engineer
OCTOBER 18, 2024
The below was originally published in The Pragmatic Engineer. To get timely analysis on the tech industry like this, on a weekly basis: sign up to The Pragmatic Engineer Newsletter. If you are into podcasts, check out The Pragmatic Engineer Podcast. Imagine Apple decided Spotify was a big enough business threat that it had to take unfair measures to limit Spotify’s growth on the App Store.
Confessions of a Data Guy
OCTOBER 24, 2024
There are some things you don’t need until you need them. I ran into that situation recently with needing to process some CSV / Flatfiles on short notice. At first, it appeared to be easy, but then I realized, as usual, there was a little monkey wrench thrown into the middle of it. It is […] The post Skip Lines of CSV files with DuckDB and Polars appeared first on Confessions of a Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Analytics Vidhya
OCTOBER 7, 2024
Introduction Today, data systems evolve quickly, demanding efficient monitoring and response. Real-time change detection is essential to keeping systems stable, preventing failures, and ensuring business continuity. Microsoft’s open-source tool, Drasi, addresses this need by effortlessly detecting, monitoring, and responding to data changes across platforms, including relational and graph databases.
Start Data Engineering
OCTOBER 17, 2024
Introduction Setup SQL tips 1. Handy functions for common data processing scenarios 1.1. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3. STRUCT data types are sorted based on their keys from left to right 1.4. Get the first/last element with ROW_NUMBER() + QUALIFY 1.5.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Simon Späti
OCTOBER 16, 2024
Data stacks have come a long way, evolving from monolithic, one-fits-all systems like Oracle/SAP to today’s modular open data stacks. This begs the question, what’s next? Or why is the current not meeting our needs? As we see more analytics engineering and software best practices, embracing codeful, Git-based, and more CLI-based workflows, the future looks more code-first.
KDnuggets
OCTOBER 16, 2024
Where can you find projects dealing with advanced ML topics? GitHub is a perfect source with its many repositories. I’ve selected ten to talk about in this article.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Yelp Engineering
OCTOBER 6, 2024
The Yelp Reservations service (yelp_res) is the service that powers reservations on Yelp. It was acquired along with Seatme in 2013, and is a Django service and webapp. It powers the reservation backend and logic for Yelp Guest Manager, our iPad app for restaurants, and handles diner and partner flows that create reservations. Along with that, it serves a web UI and backend API for our Yelp Reservations app, which has been superseded by Yelp Guest Manager but is still used by many of our restaur
databricks
OCTOBER 8, 2024
Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The.
Start Data Engineering
OCTOBER 14, 2024
1. Introduction 2. Code & Data 3. Using nested data types effectively 3.1. Use STRUCT for one-to-one & hierarchical relationships 3.2. Use ARRAY[STRUCT] for one-to-many relationships 3.3. Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Nested data types can be sorted 3.3.3.
Simon Späti
OCTOBER 16, 2024
DuckDB has a significant share and is frequently featured in the latest data engineering news. However, it’s still in its early adopter phase and has yet to be adopted by larger enterprises. Sure, all data creators and startups have used and potentially grown to love DuckDB, but is it also suitable for enterprises? What about scaling out and sharing it with others in the organization?
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
KDnuggets
OCTOBER 30, 2024
Each project, from beginner tasks like Image Classification to advanced ones like Anomaly Detection, includes a link to the dataset and source code for easy access and implementation.
The Pragmatic Engineer
OCTOBER 17, 2024
Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one section from this week’s from last week’s The Pulse issue. To get full issues twice a week, subscribe here.
ArcGIS
OCTOBER 21, 2024
By 2050, projected atmospheric carbon dioxide levels could nearly double, causing a 4.4°C temperature increase by the end of the century. Our study projected over 53% of Türkiye's Aegean olive-growing regions may become unsuitable for cultivation. Using GIS and ArcGIS Living Atlas Layers, we can identify vulnerable areas in future conditions and assess climate change impacts on Türkiye's significant olive production for conservation and land management.
Robinhood
OCTOBER 10, 2024
When Robinhood was founded, we set out to build a platform that gives everyone access to the financial markets. Over the last decade, we’ve disrupted and changed the industry for the better, becoming the first U.S. retail broker to offer commission-free trading, and saving investors billions in the process. In recent years, we’ve expanded our offering, ushering in a number of new cutting-edge products and services that help everyone – regardless of income – trade, invest, and earn.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
In an era where efficiency is king, are you leveraging the right tools to transform your manufacturing processes? A Manufacturing Execution System (MES) is critical for enhancing operational efficiency, reducing waste, and optimizing energy usage—key factors for improving your bottom line and lowering your carbon footprint. Join Nikhil Joshi, a manufacturing technology expert with 18+ years of hands-on experience, in this new webinar as he uncovers the secrets of MES and how to best utilize thes
Engineering at Meta
OCTOBER 15, 2024
At the Open Compute Project (OCP) Global Summit 2024, we’re showcasing our latest open AI hardware designs with the OCP community. These innovations include a new AI platform, cutting-edge open rack designs, and advanced network fabrics and components. By sharing our designs, we hope to inspire collaboration and foster innovation. If you’re passionate about building the future of AI, we invite you to engage with us and OCP to help shape the next generation of open hardware for AI.
databricks
OCTOBER 8, 2024
Summary Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS.
KDnuggets
OCTOBER 25, 2024
The richness of Python’s ecosystem has one downside: it makes it difficult to decide which libraries are the best for your needs. This article is an attempt to amend this by suggesting ten (and some more, as a bonus) libraries that are an absolute must in data science.
Data Engineering Weekly
OCTOBER 11, 2024
The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent. As we grapple with these, another seismic shift is upon us—the rise of Large Language Models (LLMs).
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Precisely
OCTOBER 2, 2024
Recognizing and supporting women in technology is a top priority at Precisely. Whether it’s hosting virtual events for women to connect, or encouraging mentoring opportunities, the Precisely Women in Technology (PWIT) program goes above and beyond to ensure that women in the organization have a great network to lean on. Each month, a PWIT member is featured to share her experience navigating the tech industry.
Robinhood
OCTOBER 21, 2024
Our competitive rates for UK customers range from 5.2% to 6.25% At Robinhood, we’re empowering our customers with the tools they need to navigate the financial markets. Today, we’re excited to build upon that effort for customers in the UK by announcing the launch of margin investing, with some of the most competitive rates in the industry. Margin investing allows customers to borrow money from Robinhood, leveraging their existing holdings to purchase additional securities in order to expa
Engineering at Meta
OCTOBER 2, 2024
At Meta, React and React Native are more than just tools; they are integral to our product development and innovation. With over five thousand people at Meta building products and experiences with React every month, these technologies are fundamental to our engineering culture and our ability to quickly build and ship high quality products. In this post, we will dive into the development experiences of some of the product teams who leveraged React and React Native to deliver exciting projects sh
databricks
OCTOBER 22, 2024
Over the years, organizations have amassed a vast amount of unstructured text data—documents, reports, and emails—but extracting meaningful insights has remained a challenge.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
KDnuggets
OCTOBER 15, 2024
This guide emphasizes the growing significance of GenAI but also highlights the crucial role that data scientists play in harnessing this technology to solve real-world problems.
Yelp Engineering
OCTOBER 7, 2024
In 2019, Yelp’s Core Android team led an effort to boost navigation performance in Yelp’s Consumer app. We switched from building screens with multiple separate activities to using fragments inside a single activity. In this blog post, we’ll cover our solution, how we approached the migration and share learnings from along the way as well as performance wins.
ArcGIS
OCTOBER 8, 2024
The response digital twin to assist disaster management of North Rhine-Westphalia illustrates how to create and maintain 3D mesh data.
Robinhood
OCTOBER 1, 2024
Robinhood Crypto customers in Europe can now deposit and withdraw 20+ cryptocurrencies, and will earn a 1% deposit match for a limited time Robinhood Crypto has launched crypto transfers for customers in Europe, which is one of the most requested features in the region. Crypto transfers enable customers to deposit and withdraw more than 20 cryptocurrencies, including Bitcoin (BTC), Ethereum (ETH), Solana (SOL), USD Coin (USDC), and others, giving them greater flexibility and control over their d
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Engineering at Meta
OCTOBER 15, 2024
At Open Compute Project Summit (OCP) 2024, we’re sharing details about our next-generation network fabric for our AI training clusters. We’ve expanded our network hardware portfolio and are contributing two new disaggregated network fabrics and a new NIC to OCP. We look forward to continued collaboration with OCP to open designs for racks, servers, storage boxes, and motherboards to benefit companies of all sizes across the industry.
databricks
OCTOBER 27, 2024
We're excited to announce the General Availability of Publish to Microsoft Power BI Service from Unity Catalog, an integration that makes it easy.
KDnuggets
OCTOBER 3, 2024
Learn the data engineering tools for data orchestration, database management, batch processing, ETL (Extract, Transform, Load), data transformation, data visualization, and data streaming.
Snowflake
OCTOBER 17, 2024
Predictive machine learning continues to be a cornerstone of data-driven decision-making. However, as organizations accumulate more data in a wide variety of forms, and as modeling techniques continue to advance, the tasks of a data scientist and ML engineer are becoming increasingly complex. Oftentimes, more effort is spent on managing infrastructure, jumping through package management hurdles, and dealing with scalability issues than on actual model development.
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Expert insights. Personalized for you.
315 
Let's personalize your content