The Pragmatic Engineer
SEPTEMBER 19, 2024
This is a follow-up to the article Paying down tech debt , written by industry veteran Lou Franco. Lou has been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app. Later this year, he’s publishing a book on tech debt.
Seattle Data Guy
SEPTEMBER 20, 2024
We are still in the early days of data and the value it can add to companies. You’ll read plenty of statistics about how much value data can drive and how far behind companies that aren’t using data are. And as a data consultant, I have helped companies find that value in their data. It… Read more The post How To Modernize Your Data Strategy And Infrastructure For 2025 appeared first on Seattle Data Guy.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
SEPTEMBER 18, 2024
1. Introduction 2. Setup 3. Parts of data engineering 3.1. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4. Define checks to ensure the output dataset is usable 3.2. Identify what tool to use to process data 3.3. Data flow architecture 3.
Confessions of a Data Guy
SEPTEMBER 19, 2024
It is a Brave New World out there these days. The new tools and features come out faster than your mom on Sunday morning getting you ready for church. The same goes for the context and advice being produced on a myriad of platforms, the ole’ Like and Subscribe, and all that bit. It does […] The post Data Modeling in the Brave New Lakehouse World appeared first on Confessions of a Data Guy.
Speaker: Evelyn Chou
Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.
KDnuggets
SEPTEMBER 19, 2024
In Python, functions often require multiple arguments, and you may find yourself repeatedly passing the same values for certain parameters. This is where partial functions can help. Python’s built-in functools module allows you to create partial functions.
databricks
SEPTEMBER 18, 2024
The Databricks Generative AI Startup Challenge offers $1M+ in prizes for innovative startups building Generative AI use cases on Databricks. Apply by November 1, 2024!
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Cloudera
SEPTEMBER 18, 2024
According to recent survey data from Cloudera, 88% of companies are already utilizing AI for the tasks of enhancing efficiency in IT processes, improving customer support with chatbots, and leveraging analytics for better decision-making. More and more enterprises are leveraging pre-trained models for various applications, from natural language processing to computer vision.
KDnuggets
SEPTEMBER 20, 2024
Rapid development is happening around us, and one of the most interesting aspects of this evolution is artificial intelligence's ability to communicate through natural language with humans. Suppose you want to communicate with some LLM running on your computer without switching between applications or windows, just by using a voice hotkey. This is exactly what.
databricks
SEPTEMBER 19, 2024
Mosaic AI Model Training now supports fine-tuning up to 131K context length for Llama 3.1 models. More efficient training at long sequence lengths is made possible by several optimizations highlighted in this post.
Snowflake
SEPTEMBER 18, 2024
As the frenzied hype around generative AI cools off and as we get into the year of ideation, earlier adopters of AI are starting to see the results of initial experimentation. And these conversations are increasingly shifting to a more problem-oriented mentality. A lot of people were understandably swept up in the excitement of all that AI can do, only to find that some use cases were too risky or that those problems could be solved with traditional methods that were less costly.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Zalando Engineering
SEPTEMBER 16, 2024
Introduction Zalando Marketing Services (ZMS) is Zalando's advertising platform. It helps brands create and manage campaigns on Zalando, increasing their visibility and improving performance at every stage of the marketing funnel, from awareness to purchase, within the Zalando marketplace. At ZMS, we're constantly innovating to optimize the advertising experience on Zalando homepage.
KDnuggets
SEPTEMBER 16, 2024
Image by Editor | Ideogram Let’s learn how to perform time series data aggregation in Pandas. Preparation We would need the Pandas and Numpy packages installed, so we can install them using the following code: pip install pandas numpy With the packages installed, let’s jump into the article. Time Series.
databricks
SEPTEMBER 19, 2024
The value of Generative AI, the deepened investment Databricks has made in the space, and how customers have benefited from the certification.
Snowflake
SEPTEMBER 16, 2024
As one of the most widely used libraries in the Python ecosystem, pandas helps developers analyze, load and transform data across data science, data engineering and machine learning. The flexibility and ease of use of the pandas API have driven rapid growth in popularity, with pandas being used by one in every five developers , according to the StackOverflow 2024 Developer Survey.
Speaker: Christophe Louvion, Chief Product & Technology Officer of NRC Health and Tony Karrer, CTO at Aggregage
Christophe Louvion, Chief Product & Technology Officer of NRC Health, is here to take us through how he guided his company's recent experience of getting from concept to launch and sales of products within 90 days. In this exclusive webinar, Christophe will cover key aspects of his journey, including: LLM Development & Quick Wins 🤖 Understand how LLMs differ from traditional software, identifying opportunities for rapid development and deployment.
ArcGIS
SEPTEMBER 18, 2024
Learn the difference between ArcGIS Data Interoperability and FME technology and how they relate to one another.
KDnuggets
SEPTEMBER 20, 2024
Data come from everywhere, and the number of origins, sources, and formats under which valuable data may appear underscores the need for database management tools capable of loading data from multiple sources. This tutorial illustrates how to load datasets from different formats and sources into Google BigQuery. All the prerequisites we need are having registered.
databricks
SEPTEMBER 18, 2024
Transformer models, the backbone of modern language AI, rely on the attention mechanism to process context when generating output. During inference, the attention.
Snowflake
SEPTEMBER 17, 2024
Why now is the time for data leaders in financial services to address the challenges of tick data analysis — and how Snowflake can help The financial services industry has been facing plenty of challenges lately. The rising cost of capital means leaders need to be smart about finding places to reduce total cost of ownership and scale technology. The number of market and technology regulations around data, infrastructure and reporting, like DORA and GDPR , can be overwhelming; complying to them m
Advertisement
Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.
Precisely
SEPTEMBER 18, 2024
Businesses around the world are facing major challenges due to higher manufacturing costs, disruptive new technologies like artificial intelligence (AI), and tougher global competition. This means it’s more important than ever to make data-driven decisions, cut costs, and improve efficiency. But this is all easier said than done, as evidenced by key findings from this year’s 2025 Outlook: Data Integrity Trends and Insights report, published in partnership between Precisely and the Center for App
KDnuggets
SEPTEMBER 18, 2024
Medical imaging has been revolutionized by the adoption of deep learning techniques. The use of this branch of machine learning has ushered in a new era of precision and efficiency in medical image segmentation, a central analytical process in modern healthcare diagnostics and treatment planning. By harnessing neural networks, deep learning algorithms are able.
databricks
SEPTEMBER 20, 2024
Introduction The bin packing problem is a classic optimization challenge that has far-reaching implications for enterprise organizations across industries. At its core, the.
Data Engineering Weekly
SEPTEMBER 15, 2024
Uber: DataMesh - How Uber laid the foundations for the Data Lake Cloud migration Many companies are slowly adopting DataMesh, and Uber writes about adopting the data mesh principle. Whether or not Data Mesh is a separate product is debatable, but it is certainly an impactful framework for scaling data platforms. I expect more and more frameworks to be built on top of the existing compute and orchestration layer.
Speaker: Jamie Bernard
Success in product management goes beyond delivering great features - it’s about achieving measurable financial outcomes that resonate across the organization. By connecting your product’s journey with the company’s financial success, you’ll ensure that every feature, release, and innovation contributes to the bottom line, driving both customer satisfaction and business growth.
Precisely
SEPTEMBER 19, 2024
Key Takeaways: Data integrity is essential for AI success and reliability – helping you prevent harmful biases and inaccuracies in AI models. Robust data governance for AI ensures data privacy, compliance, and ethical AI use. Proactive data quality measures are critical, especially in AI applications. Using AI systems to analyze and improve data quality both benefits and contributes to the generation of high-quality data.
KDnuggets
SEPTEMBER 16, 2024
Image by Author If you’re in the tech industry (or are attempting to transition into the field), LLMs are a must-learn. Companies have started integrating language models into their workflows to improve efficiencies and cut costs. Due to this, there have been a number of new AI job openings. New roles have begun to.
databricks
SEPTEMBER 17, 2024
Today, we are excited to announce the support for named parameter markers in the SQL editor. This feature allows you to write parameterized.
Confluent
SEPTEMBER 17, 2024
Confluent for VS Code streamlines workflows, accelerates development cycles, and enhances real-time data processing, all within a unified environment.
Advertisement
The complexity of financial data, the need for real-time insight, and the demand for user-friendly visualizations can seem daunting when it comes to analytics - but there is an easier way. With Logi Symphony, we aim to turn these challenges into opportunities. Our platform empowers you to seamlessly integrate advanced data analytics, generative AI, data visualization, and pixel-perfect reporting into your applications, transforming raw data into actionable insights.
Yelp Engineering
SEPTEMBER 18, 2024
Machine Learning Feature Stores ML Feature Store at Yelp Many of Yelp’s core capabilities such as business search, ads, and reviews are powered by Machine Learning (ML). In order to ensure these capabilities are well supported, we have built a dedicated ML platform. One of the pillars of this infrastructure is the Feature Store, which is a centralized data store for ML Features that are the input of ML models.
KDnuggets
SEPTEMBER 17, 2024
Let’s learn how to handle large text inputs in the Large Language Model (LLM). Preparation Ensure you have the Transformers and datasets package from Hugging Face installed in your environment. If not, you can install them via pip using the following code: pip install transformers datasets Additionally, you should install the.
databricks
SEPTEMBER 18, 2024
At Databricks, we know that data is one of your most valuable assets. Our product and security teams work together to deliver an enterprise-grade Data Intelligence Platform that enables you to defend against security risks and meet your compliance obligations. In this blog, we'll explain how you can leverage our platform's security features to establish a robust defense-in-depth posture that protects your data and AI assets from risks.
DataKitchen
SEPTEMBER 18, 2024
A DataOps Approach to Data Quality The Growing Complexity of Data Quality Data quality issues are widespread, affecting organizations across industries, from manufacturing to healthcare and financial services. According to DataKitchen’s 2024 market research, conducted with over three dozen data quality leaders, the complexity of data quality problems stems from the diverse nature of data sources, the increasing scale of data, and the fragmented nature of data systems.
Speaker: Maher Hanafi, VP of Engineering at Betterworks & Tony Karrer, CTO at Aggregage
Executive leaders and board members are pushing their teams to adopt Generative AI to gain a competitive edge, save money, and otherwise take advantage of the promise of this new era of artificial intelligence. There's no question that it is challenging to figure out where to focus and how to advance when it’s a new field that is evolving everyday. 💡 This new webinar featuring Maher Hanafi, VP of Engineering at Betterworks, will explore a practical framework to transform Generative AI pr
Expert insights. Personalized for you.
166 
Let's personalize your content