Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. Imagine being an online shopper who suddenly receives irrelevant recommendations. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.

Data Engineering 219

Data Engineering Data Engineer Engineering Data Pipeline 219

The Pragmatic Engineer

JUNE 29, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of four topics from today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here.

AWS 59

AWS Software Engineer Software Engineering Engineering 59

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. SQLMesh was designed as a unifying tool that is simple to work with but powerful enough for large-scale transformations and complex projects.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

KDnuggets

JUNE 30, 2023

Every job is at risk. Here’s how you can AI-proof your career.

Data 164

Data Data Science 164

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

Analytics Vidhya

JUNE 24, 2023

Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases. Properly configured data structures ensure a smoother workflow and prevent data loss or misplacement.

Database 211

Database Utilities Data Data Science 211

Engineering at Meta

JUNE 27, 2023

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we talk about below are open source so you can try them yourself.

Java 133

Java Programming Language Algorithm Programming 133

KDnuggets

JUNE 27, 2023

The k-Nearest Neighbors Classifier is a machine learning algorithm that assigns a new data point to the most common class among its k closest neighbors. In this tutorial, you will learn the basic steps of building and applying this classifier in Python.

Building 142

Building Machine Learning Algorithm Python 142

Analytics Vidhya

JUNE 24, 2023

Introduction We had an amazing opportunity to learn from Mr. Pavan. He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Throughout the conversation, Mr. Pavan shares his journey, inspirations, challenges, and accomplishments. Thus, providing valuable insights into the field of data engineering. As we explore Mr.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Start Data Engineering

JUNE 30, 2023

1. Introduction 2. What is self-serve? 2.1. Components of a self-serve platform 3. Building a self-serve data platform 3.1. Creating dataset(s) 3.1.1. Gather requirements 3.1.2. Get data foundations right 3.2. Accessing data 3.3. Identify and remove dependencies 4. Conclusion 5. Further reading 6. References 1. Introduction Most companies want to build a self-serve data platform.

Building 130

Building Datasets Data Accessible 130

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Waitingforcode

JUNE 30, 2023

Surprised? You shouldn't. I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure. But there is some excitement for learning-from scratch I miss.

AWS 130

AWS Technology 130

Confessions of a Data Guy

JUNE 28, 2023

I’ve been a dog licking my wounds for some time now. Over on my Substack newsletter, I’ve been doing a small series on DSA (Data Structures and Algorithms). I tackled some of the easier stuff first, like Linked Lists, Binary Search, and the like. What’s more, I actually did most of it in Rust, since […] The post Exploring Graphs in Rust.

Algorithm 130

Algorithm Data IT Big Data 130

Christophe Blefari

JUNE 24, 2023

( credits ) Hey, this is the Data News. It's super hard to change habits, but it's how it is, the newsletter is going out on Saturday. I hope this edition finds you well. Summer is coming ☀️ Thank you all because we crossed the 3000 subscribers mark last week. Let's go for the 4000 before the end of the year 🤗 This is a almost-raw edition for this week.

PostgreSQL 130

PostgreSQL Data Data Engineering Data Engineer 130

databricks

JUNE 27, 2023

Generative AI will have a transformative impact on every business. Databricks has been pioneering AI innovations for a decade, actively collaborating with thousands.

Building 124

Building Data 124

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Lyft Engineering

JUNE 28, 2023

Written by Konstantin Gizdarski and Martin Liu at Lyft. In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. On the real-time front, LyftLearn supported real-time inference and input feature validation. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Kafka Metadata 117

KDnuggets

JUNE 27, 2023

And how approaches to better data management, version control, and experiment tracking can help build reproducible ML pipelines.

Machine Learning 116

Machine Learning Data Management Building Management 116

LinkedIn Engineering

JUNE 26, 2023

For the last several years, internal infrastructure at LinkedIn has been built around a self-service model, enabling developers to onboard themselves with minimal support. We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. For example, developers can provision Kafka topics, Espresso tables, Venice stores and more via Nuage , our internal cloud-like infra management platform.

Data Pipeline 112

Data Pipeline Kafka SQL MySQL 112

databricks

JUNE 27, 2023

Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural.

Engineering 111

Engineering Data 111

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Confluent

JUNE 28, 2023

Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.

Kafka 111

Kafka Architecture Designing 111

KDnuggets

JUNE 29, 2023

Large language models are getting released left right and center, and if you want to understand them better you need to know about NLP. Here are 5 Free books to help you.

Process 110

Process 110

ThoughtSpot

JUNE 27, 2023

Active listening is an admired and sought after skill in both the professional and personal sphere. After all, who doesn’t love to be heard? But what happens when we apply that mindset to the way our organizations solicit feedback and interact with our customers? We don’t have to make any assumptions to answer this question, because we have the data.

Programming 105

Programming Data Science Data Analytics Process 105

databricks

JUNE 27, 2023

Data teams face many challenges to quickly access the right data primarily due to data fragmentation, time and cost involved in consolidating data.

Accessible 105

Accessible Accessibility Data 105

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. Specifically, observability provides insights into the pipeline’s internal states and how they interact with the system’s outputs. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

KDnuggets

JUNE 29, 2023

This article provides a general overview of Stable Diffusion and focuses on building a basic understanding of how generative artificial intelligence works.

Building 110

Building 110

ThoughtSpot

JUNE 26, 2023

At ThoughtSpot, we know how important it is for businesses of every size and industry to empower every knowledge worker with personalized, actionable data-driven insights. These insights are your secret sauce to making better business decisions, growing faster, and delivering customer experiences that keep people coming back for more. But how do you scale self-service analytics to business users without completely overwhelming your data teams?

BI 105

BI Business Intelligence Datasets Government 105

databricks

JUNE 27, 2023

We are thrilled to announce that materialized views and streaming tables are now publicly available in Databricks SQL on AWS and Azure. Streaming.

SQL 105

SQL AWS 105

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Towards Data Science

JUNE 27, 2023

The Top 5 Features for Efficient Data Manipulation This April, pandas 2.0.0 was officially launched , making huge waves across the data science community. Photo by Yancy Min on Unsplash. Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. From data input/output to data cleaning and transformation, it’s nearly impossible to think about data manipulation without import pandas as pd, right ?

Pipeline-centric 98

Pipeline-centric Data Science Machine Learning Datasets 98

KDnuggets

JUNE 28, 2023

KDnuggets' latest cheat sheet presents you with an impressive array of advanced tools and resources designed to support your data science game. They cover a wide range of applications, from understanding complex scientific literature to writing high-quality manuscripts and more.

Data Science 106

Data Science Data Designing Machine Learning 106

ThoughtSpot

JUNE 30, 2023

Pride is more than just a month-long celebration; it is a powerful movement that reminds us of the importance of equality, acceptance, and love. It is that special time of year for the global queer community to come together to celebrate, commemorate, and continue to push for progress. It’s no different here at ThoughtSpot. We believe in creating an inclusive environment where everyone feels seen, heard, and valued.

Education 98

Education Recruitment Programming Building 98