The Pragmatic Engineer
JUNE 22, 2023
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of five topics from today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here.
Analytics Vidhya
JUNE 20, 2023
Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing. Traditional data engineering solutions, such as Apache Airflow, have played an important role in orchestrating and controlling data operations in order to tackle these difficulties.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Waitingforcode
JUNE 23, 2023
Shuffle is a permanent point in the What's new in Apache Spark series. Why? It's often one the most time consuming part of the jobs and knowing the improvement simply helps writing better pipelines.
Confessions of a Data Guy
JUNE 20, 2023
Sometimes I think Data Engineering is the same as it was 10+ years ago when I started doing it, and sometimes I think everything has changed. It’s probably both. In some ways, the underlying concepts have not moved an inch, some certain truths and axioms still rule over us all like some distant landlord, requiring […] The post Old Dog Learn New Tricks?
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Data Engineering Podcast
JUNE 17, 2023
Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design.
LinkedIn Engineering
JUNE 20, 2023
Co-authors: Shivansh Mundra , Gonzalo Aniano Porcile , Smit Marvaniya , Hany Farid A core part of what we do on the Trust Data Team at LinkedIn is create, deploy, and maintain models that detect and prevent many types of abuse. This spans the detection and prevention of fake accounts, account takeovers, and policy-violating content. We are constantly working to improve and increase the effectiveness of our anti-abuse defenses to protect the experiences of our members and customers.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Confessions of a Data Guy
JUNE 22, 2023
The post Conceptual Introduction to Delta Lake. appeared first on Confessions of a Data Guy.
ThoughtSpot
JUNE 23, 2023
During Beyond 2023, we officially launched ThoughtSpot Sage , our new search experience that combines the power of GPT’s natural language processing and generative AI capabilities with the accuracy and security of our patented self-service analytics platform. As the experience layer of the modern data stack and an Elite Partner of Snowflake, we are constantly thinking about how our product innovation unlocks value for our shared customers.
Netflix Tech
JUNE 20, 2023
Avneesh Saluja , Andy Yao , Hossein Taghavi Introduction When watching a movie or an episode of a TV show, we experience a cohesive narrative that unfolds before us, often without giving much thought to the underlying structure that makes it all possible. However, movies and episodes are not atomic units, but rather composed of smaller elements such as frames, shots, scenes, sequences, and acts.
KDnuggets
JUNE 21, 2023
This article elaborates on the importance of Explainable AI (XAI), what the challenges in building interpretable AI models are, and some practical guidelines for companies to build XAI models.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
DataKitchen
JUNE 19, 2023
The need to balance data safety with new data initiatives, deliver business value, and change company culture around data tops this year's list of data and analytics management challenges. To help bring new resources and innovation to light, each year, Database Trends and Applications magazine presents the DBTA 100, a list of forward-thinking companies seeking to expand what's possible with data for their customers.
databricks
JUNE 20, 2023
This blog is the first of a 3-part series chronicling TD Bank's Data Platform transformation and the enablement of their Data as a.
LinkedIn Engineering
JUNE 23, 2023
In a world where superheroes captivate our imaginations, it's sometimes hard to recognize the real-life superheroes among us like intelligence analysts, forensic scientists, and cybersecurity professionals. Yes, cybersecurity professionals! Though we may not wear capes or possess extraordinary powers, our role, especially here at LinkedIn, is crucial in safeguarding our members, customers, and employees from the ever-present threat of cyberattacks.
KDnuggets
JUNE 20, 2023
In this article, we’ll learn to adapt pre-trained models to custom classification tasks using a technique called transfer learning. We will demonstrate it for an image classification task using PyTorch, and compare transfer learning on 3 pre-trained models, Vgg16, ResNet50, and ResNet152.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
ArcGIS
JUNE 21, 2023
Check out exciting sessions, special interest groups and activities on BIM, CAD, and GIS integrations featured at Esri UC 2023.
databricks
JUNE 23, 2023
This is a collaborative post from Databricks and Amazon Web Services (AWS). We thank Venkat Viswanathan, Data and Analytics Strategy Leader, Partner Solutions.
Towards Data Science
JUNE 19, 2023
Photo by CHUTTERSNAP on Unsplash This article is about Meerschaum Compose , a tool for defining ETL pipelines in YAML and a plugin for the data engineering framework Meerschaum. Docker was a game-changer, revolutionizing the way we design, build, and run our cloud applications. Pretty early on, however, developers realized its flexibility made collaboration difficult, so docker-compose became to the tool of choice for managing environments and multi-container projects.
KDnuggets
JUNE 21, 2023
Learn everything about the most popular Machine Learning algorithm, Linear Regression, with its Mathematical Intuition and Python implementation.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
DoorDash Engineering
JUNE 20, 2023
In real-world forecasting applications , it is a challenge to balance accuracy and speed. We can achieve high accuracy by running numerous models and configuration combinations and we gain speed through running fast, computationally inexpensive models. We explore a number of models and configuration combinations at DoorDash to forecast demand on our platform.
databricks
JUNE 21, 2023
The annual Data Team Awards showcase how different enterprise data teams are delivering solutions to some of the world’s toughest problems. Nearly 300 n.
Uber Engineering
JUNE 22, 2023
We are excited to release Cadence 1.0! Used by many major companies, at Uber it powers over 1,000 services with 100K+ updates a second. Learn how Cadence makes it easy to build complex distributed systems.
KDnuggets
JUNE 19, 2023
Large language models (LLMs) currently have the AI world in a chokehold. It is essential to understand why vector databases are important to LLMs.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Tweag
JUNE 21, 2023
For a while now, Tweag has committed in improving various aspects of Liquid Haskell (LH), a tool that gives the Haskell programmer both the ability to express properties about programs and to verify that they meet these expectations. In this post we present a specific improvement that I integrated recently, which cuts down the maintenance cost to use LH when introducing assumptions about functions coming from large or multiple packages.
databricks
JUNE 22, 2023
We are excited to announce the public preview of Unity Catalog support for Delta Live Tables (DLT). With this preview, any data team.
Cloudera
JUNE 22, 2023
In spite of diligent digital transformation efforts, most financial services institutions still support a loose patchwork of siloed systems and repositories. These dis-integrated resources are “data platforms” in name only: in addition to their high maintenance costs, their lack of interoperability with other critical systems makes it difficult to respond to business change.
KDnuggets
JUNE 20, 2023
Best practices, techniques, and tools to fully understand your data.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
Snowflake
JUNE 20, 2023
Welcome to Snowflake’s Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. In this edition, we’re digging into cybersecurity with Parth Shah, Co-Founder and Head of Product at Dassana , as he discusses the power of operationalizing security data, why you need to consider security data lakes, and how Snowflake gave Dassana an agility upgrade.
databricks
JUNE 22, 2023
This blog is authored by Sai Ravuru Senior Manager of Data Science & Analytics at JetBlue The role of data in the aviation.
Confluent
JUNE 21, 2023
See why Wealthsimple chose Confluent to build real-time API financial solutions that could process, transform, and govern real-time data for downstream systems.
KDnuggets
JUNE 23, 2023
Learn how to use large language models to extract insights from documents for analytics and ML at scale. Join this webinar and live tutorial to learn how to get started.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
Expert insights. Personalized for you.
247 
Let's personalize your content