Christophe Blefari
OCTOBER 19, 2023
cross-project dependencies ( credits ) Over the last few years, dbt has become a de facto standard enabling companies to collaborate easily on data transformations. With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier. So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries.
Data Engineering Podcast
OCTOBER 15, 2023
Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Marc Lamberti
OCTOBER 20, 2023
Templates and Macros in Apache Airflow allow passing data to your DAGs at runtime. Imagine that you want to execute an SQL request with the execution date of your DAG. How can you do that? How can you use the DAG ID when you send notifications to know which DAG to look at? Or what if you need to know when the next DAG run will be? Well, macros and templates answer these questions.
Waitingforcode
OCTOBER 17, 2023
I've already written about watermarks in a few places in the blog but despite that, I still find things to refresh. One of them is the watermark used to filter out the late data, which will be the topic of this blog post.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Christophe Blefari
OCTOBER 20, 2023
Writing about dbt like a sheep ( credits ) Hey, this week Coalesce—the dbt Labs annual conference—took place. During 3 days, people shared how they used dbt around the world. I'll, as usual, write a takeaway post after binge watching all keynotes, but this is for next week. Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product.
Scott Logic
OCTOBER 18, 2023
The State of WebAssembly 2023 survey has closed, the results are in … and they are fascinating! If you want the TL;DR; here are the highlights: Rust and JavaScript usage is continuing to increase, but some more notable changes are happening a little further down - with both Swift and Zig seeing a significant increase in adoption. When it comes to which languages developers ‘desire’, with Zig, Kotlin and C# we see that desirability exceeds current usage WebAssembly is still most often used for we
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
DoorDash Engineering
OCTOBER 19, 2023
As DoorDash’s microservices architecture has grown, so too has the volume of interservice traffic. Each team manages their own data and exposes access through gRPC services, an open-source remote procedure call framework used to build scalable APIs. Most business logic is I/O-bound because of calls to downstream services. Caching has long been a go-to strategy to improve performance and reduce costs.
Christophe Blefari
OCTOBER 14, 2023
( credits ) Hello, dear Data News reader, I hope you'll enjoy this new edition. It's amazing how quickly time flies and this summer I passed the 3-year mark since I started my freelance adventure. I'm so happy with what it's brought me. But I've got this internal alarm that goes off every 3 years asking me for new things. It's time for me to search for my future paths.
LinkedIn Engineering
OCTOBER 19, 2023
Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers. This robust framework empowers near real-time data processing for critical services and platforms, ranging from machine learning and notifications to anti-abuse AI modeling.
KDnuggets
OCTOBER 16, 2023
Want to break into data science? Check this list of free books for learning Python, statistics, linear algebra, machine learning and deep learning.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
ArcGIS
OCTOBER 17, 2023
The new U.S. datums of 2022 will soon be released. This article covers what is coming and how you should prepare your data.
Engineering at Meta
OCTOBER 17, 2023
Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, and those products comprise thousands of individual features.
databricks
OCTOBER 19, 2023
Machine learning (ML) is more than just developing models; it's about bringing them to life in real-world, production systems. But transitioning from prototype.
KDnuggets
OCTOBER 18, 2023
Large Language Models (LLMs) have unlocked a new era in natural language processing. So why not learn more about them? Go from learning what large language models are to building and deploying LLM apps in 7 easy steps with this guide.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Tweag
OCTOBER 18, 2023
At Tweag we have been cooking up a JSON Schema to Nickel contract converter , that we’re excited to announce! Background Nickel is a configuration language being developed at Tweag. You can get some deep dives into its design from previous blog posts. I’ll summarize it here as JSON, plus functions, plus types and contracts. One of its main use-cases is generating JSON configurations for other programs (Terraform, GitHub actions, etc).
Engineering at Meta
OCTOBER 18, 2023
With the recent launches of MTIA v1 , Meta’s first-generation AI inference accelerator, and Llama 2 , the next generation of Meta’s publicly available large language model, it’s clear that Meta is focused on advancing AI for a more connected world. Fueling the success of these products are world-class infrastructure teams, including Meta’s custom AI silicon team, led by Olivia Wu, a leader in the silicon industry for 30 years.
InData Labs
OCTOBER 17, 2023
Big data is central to the efficient running of all modern organizations, but to be of use, raw data must be suitably organized. The way that businesses organize data assets is commonly known as data architecture, with the benefits of modern data architecture enabling teams to respond to changing demands with improved agility when compared. Запись The benefits of modern data architecture впервые появилась InData Labs.
KDnuggets
OCTOBER 19, 2023
Looking to understand the semantic layer and how it can improve the AI-powered data experience? Read more to learn why a semantic layer can be the backbone of LLMs and reduce hallucinations.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Pinterest Engineering
OCTOBER 17, 2023
Charles Wu | Software Engineer; Isabel Tallam | Software Engineer; Kapil Bajaj | Engineering Manager Overview In this blog, we present a pragmatic way of integrating analytics, written in Python, with our distributed anomaly detection platform, written in Java. The approach here could be generalized to integrate processing done in one language/paradigm into a platform in another language/paradigm.
ArcGIS
OCTOBER 17, 2023
The Living Atlas of the World just updated the tiled, hosted image service featuring transportation noise, from the USDOT.
Ripple Engineering
OCTOBER 19, 2023
RippleX has enabled its validator to vote in support of the XLS-30 amendment, introducing innovative AMM capabilities to the XRPL. We, at RippleX, place great emphasis on the strength that collaborative effort and shared responsibility bring to the enhancement and security of the XRPL. Today, we earnestly request the community's consideration of the XLS-30 amendment —a proposal poised to offer numerous advantages by bolstering liquidity, offering yield opportunities for liquidity pro
KDnuggets
OCTOBER 17, 2023
Large language models (LLMs) are transforming the way we process and produce information. But, before considering either one of these models as a one-stop-solution, one must consider their key differences.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
databricks
OCTOBER 17, 2023
Introduction Large Language Models (LLMs) have given us a way to generate text, extract information, and identify patterns in industries from healthcare to.
ArcGIS
OCTOBER 16, 2023
Streamline GIS workflows with ArcGIS Reality for ArcGIS Pro. Automate reality mapping, generate accurate geospatial products.
Snowflake
OCTOBER 19, 2023
Welcome to Snowflake’s Startup Spotlight, where we learn about amazing companies building their businesses on Snowflake. In this edition, Pave.dev President and Co-Founder Ema Rouf talks about breaking down barriers to accessible credit and financial lending, how running a startup is like climbing a mountain, and how building on Snowflake gives Pave the data sharing capabilities it needs to show financial institutions a better way to identify more creditworthy borrowers.
KDnuggets
OCTOBER 19, 2023
Gradient descent is an optimization technique used to minimise errors in machine learning models. By iteratively adjusting parameters in the steepest direction of decrease, it seeks the lowest error value.
Speaker: Evelyn Chou
Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.
databricks
OCTOBER 20, 2023
Check out our Nearest Neighborhood Search Solution Accelerator to get started quickly. The Member Experience An insured member typically experiences their healthcare in.
Scott Logic
OCTOBER 18, 2023
Introduction In my previous blog post I discussed how migrating to the Cloud could help your organisation reach its Net Zero goals. I discussed how shifting your workloads away from on-premises data centres can reduce emissions by allowing you to leverage the expertise of cloud providers and their greater efficiency of scale. It should be noted this isn’t always clear cut - do consider how energy efficient your current hosting is and the embodied carbon of any hardware you’d be decommissioning.
Snowflake
OCTOBER 18, 2023
Product-led growth (PLG) is a business model that emerged in the last decade with the enormous success of vendors like Slack and Datadog. Unlike traditional sales-led models, PLG models cut out the middlemen (sales reps, for example) and let customers just download and use the product without third-party onboarding. The relative novelty of the pricing model and its demonstrably successful application in growing these companies attracted a lot of attention.
KDnuggets
OCTOBER 16, 2023
This article has outlined how you can fine tune your GPT 3.5 Turbo models. You can do this by preparing your data, uploading your files, and then setting up a custom OpenAI session to handle the fine tuning.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Expert insights. Personalized for you.
264 
Let's personalize your content