The Pragmatic Engineer
JUNE 22, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of five topics from today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here.
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing. Traditional data engineering solutions, such as Apache Airflow, have played an important role in orchestrating and controlling data operations in order to tackle these difficulties.
KDnuggets
JUNE 23, 2023
Fast forward your EDA process using this ChatGPT plugin.
LinkedIn Engineering
JUNE 20, 2023
Co-authors: Shivansh Mundra , Gonzalo Aniano Porcile , Smit Marvaniya , Hany Farid A core part of what we do on the Trust Data Team at LinkedIn is create, deploy, and maintain models that detect and prevent many types of abuse. This spans the detection and prevention of fake accounts, account takeovers, and policy-violating content. We are constantly working to improve and increase the effectiveness of our anti-abuse defenses to protect the experiences of our members and customers.
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Waitingforcode
JUNE 23, 2023
Shuffle is a permanent point in the What's new in Apache Spark series. Why? It's often one the most time consuming part of the jobs and knowing the improvement simply helps writing better pipelines.
Confessions of a Data Guy
JUNE 22, 2023
The post Conceptual Introduction to Delta Lake. appeared first on Confessions of a Data Guy.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
JUNE 17, 2023
Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design.
databricks
JUNE 20, 2023
This blog is the first of a 3-part series chronicling TD Bank's Data Platform transformation and the enablement of their Data as a.
Confessions of a Data Guy
JUNE 20, 2023
Sometimes I think Data Engineering is the same as it was 10+ years ago when I started doing it, and sometimes I think everything has changed. It’s probably both. In some ways, the underlying concepts have not moved an inch, some certain truths and axioms still rule over us all like some distant landlord, requiring […] The post Old Dog Learn New Tricks?
KDnuggets
JUNE 20, 2023
Best practices, techniques, and tools to fully understand your data.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Netflix Tech
JUNE 20, 2023
Avneesh Saluja , Andy Yao , Hossein Taghavi Introduction When watching a movie or an episode of a TV show, we experience a cohesive narrative that unfolds before us, often without giving much thought to the underlying structure that makes it all possible. However, movies and episodes are not atomic units, but rather composed of smaller elements such as frames, shots, scenes, sequences, and acts.
databricks
JUNE 22, 2023
This blog is authored by Sai Ravuru Senior Manager of Data Science & Analytics at JetBlue The role of data in the aviation.
ThoughtSpot
JUNE 23, 2023
During Beyond 2023, we officially launched ThoughtSpot Sage , our new search experience that combines the power of GPT’s natural language processing and generative AI capabilities with the accuracy and security of our patented self-service analytics platform. As the experience layer of the modern data stack and an Elite Partner of Snowflake, we are constantly thinking about how our product innovation unlocks value for our shared customers.
KDnuggets
JUNE 21, 2023
Learn everything about the most popular Machine Learning algorithm, Linear Regression, with its Mathematical Intuition and Python implementation.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
ArcGIS
JUNE 21, 2023
Check out exciting sessions, special interest groups and activities on BIM, CAD, and GIS integrations featured at Esri UC 2023.
databricks
JUNE 23, 2023
This is a collaborative post from Databricks and Amazon Web Services (AWS). We thank Venkat Viswanathan, Data and Analytics Strategy Leader, Partner Solutions.
DataKitchen
JUNE 19, 2023
The need to balance data safety with new data initiatives, deliver business value, and change company culture around data tops this year's list of data and analytics management challenges. To help bring new resources and innovation to light, each year, Database Trends and Applications magazine presents the DBTA 100, a list of forward-thinking companies seeking to expand what's possible with data for their customers.
KDnuggets
JUNE 20, 2023
In this article, we’ll learn to adapt pre-trained models to custom classification tasks using a technique called transfer learning. We will demonstrate it for an image classification task using PyTorch, and compare transfer learning on 3 pre-trained models, Vgg16, ResNet50, and ResNet152.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
LinkedIn Engineering
JUNE 23, 2023
In a world where superheroes captivate our imaginations, it's sometimes hard to recognize the real-life superheroes among us like intelligence analysts, forensic scientists, and cybersecurity professionals. Yes, cybersecurity professionals! Though we may not wear capes or possess extraordinary powers, our role, especially here at LinkedIn, is crucial in safeguarding our members, customers, and employees from the ever-present threat of cyberattacks.
databricks
JUNE 22, 2023
We are excited to announce the public preview of Unity Catalog support for Delta Live Tables (DLT). With this preview, any data team.
ArcGIS
JUNE 22, 2023
Explore a curated agenda for topographic mapping at the 2023 Esri User Conference.
KDnuggets
JUNE 21, 2023
This article elaborates on the importance of Explainable AI (XAI), what the challenges in building interpretable AI models are, and some practical guidelines for companies to build XAI models.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Towards Data Science
JUNE 19, 2023
Photo by CHUTTERSNAP on Unsplash This article is about Meerschaum Compose , a tool for defining ETL pipelines in YAML and a plugin for the data engineering framework Meerschaum. Docker was a game-changer, revolutionizing the way we design, build, and run our cloud applications. Pretty early on, however, developers realized its flexibility made collaboration difficult, so docker-compose became to the tool of choice for managing environments and multi-container projects.
databricks
JUNE 21, 2023
The annual Data Team Awards showcase how different enterprise data teams are delivering solutions to some of the world’s toughest problems. Nearly 300 n.
DoorDash Engineering
JUNE 20, 2023
In real-world forecasting applications , it is a challenge to balance accuracy and speed. We can achieve high accuracy by running numerous models and configuration combinations and we gain speed through running fast, computationally inexpensive models. We explore a number of models and configuration combinations at DoorDash to forecast demand on our platform.
KDnuggets
JUNE 20, 2023
Orca is a 13B parameter model that learns to imitate the reasoning processes of LFMs. It uses progressive learning and teacher assistance from ChatGPT to overcome capacity gaps. By leveraging rich signals from GPT-4, Orca enhances its capabilities and improves imitation learning performance.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Uber Engineering
JUNE 22, 2023
We are excited to release Cadence 1.0! Used by many major companies, at Uber it powers over 1,000 services with 100K+ updates a second. Learn how Cadence makes it easy to build complex distributed systems.
databricks
JUNE 19, 2023
The countdown is on for the highly anticipated Data + AI Summit! Whether you're joining us in person or virtually, get ready for.
Cloudera
JUNE 22, 2023
In spite of diligent digital transformation efforts, most financial services institutions still support a loose patchwork of siloed systems and repositories. These dis-integrated resources are “data platforms” in name only: in addition to their high maintenance costs, their lack of interoperability with other critical systems makes it difficult to respond to business change.
KDnuggets
JUNE 19, 2023
Large language models (LLMs) currently have the AI world in a chokehold. It is essential to understand why vector databases are important to LLMs.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Expert insights. Personalized for you.
251 
Let's personalize your content