KDnuggets

MARCH 28, 2023

Most essential skills are programming, data preparation, statistical analysis, deep learning, and natural language processing.

Machine Learning 141

Machine Learning Deep Learning Data Preparation Engineering 141

KDnuggets

MARCH 30, 2023

Work on data analytics, time series, natural language processing, machine learning, and ChatGPT projects to improve your chance of getting hired.

Portfolio 176

Portfolio Project Data Science Machine Learning 176

Confessions of a Data Guy

MARCH 20, 2023

Are lambdas one of those tools that everyone uses and no one talks about? I guess I’ve taken them for granted over the years, even though they are incredibly useful. For a lot of my Data Engineering career I didn’t really think about or use AWS lambdas, I just saw them as little annoying flies […] The post AWS Lambdas. Useful for Data Engineering?

AWS 130

AWS Data Engineering Data Engineer Engineering 130

The Pragmatic Engineer

MARCH 16, 2023

Comments

317

317

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

Christophe Blefari

MARCH 1, 2023

This article is meant to be a resource hub in order to understand dbt basics and to help get started your dbt journey. When I write dbt, I often mean dbt Core. dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt Core has been developed by dbt Labs, which was previously named Fishtown Analytics. The company has been founded in May 2016. dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Waitingforcode

MARCH 23, 2023

Welcome to the 2nd part of the series with great streaming and project organization blog posts summaries!

Data Engineering 147

Data Engineering Data Engineer Engineering Project 147

KDnuggets

MARCH 29, 2023

The second part covers the list of Machine Learning, Deep Learning, Computer Vision, Natural Language Processing, Data Engineering, and MLOps.

Data Science 160

Data Science Deep Learning Machine Learning Data Engineering 160

Data Engineering Podcast

MARCH 5, 2023

Summary The ecosystem for data professionals has matured to the point that there are a large and growing number of distinct roles. With the scope and importance of data steadily increasing it is important for organizations to ensure that everyone is aligned and operating in a positive environment. To help facilitate the nascent conversation about what constitutes an effective and productive data culture, the team at Data Council have dedicated an entire conference track to the subject.

Building 147

Building Database Design Machine Learning Metadata 147

The Pragmatic Engineer

MARCH 3, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics from The Scoop #39 , published two weeks ago, 23 February. To get full newsletters twice a week, subscribe here. I have collaborated with a tech recruiter - they’ve asked to be anonymous - who’s been running some very interesting queries on LinkedIn for software engineers.

Recruitment 215

Recruitment Software Engineer Software Engineering Finance 215

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Tweag

MARCH 13, 2023

It is a truth universally acknowledged that the Python packaging ecosystem is in need of a good dependency checker. In the least, it’s our hope to convince you that Tweag’s new dependency checker, FawltyDeps, can help you maintain an environment that is minimal and reproducible for your Python project, by ensuring that required dependencies are explicitly declared and detecting unused dependencies.

Python 145

Python Coding Project Systems 145

Analytics Vidhya

MARCH 5, 2023

Introduction S3 is Amazon Web Services cloud-based object storage service (AWS). It stores and retrieves large amounts of data, including photos, movies, documents, and other files, in a durable, accessible, and scalable manner. S3 provides a simple web interface for uploading and downloading data and a powerful set of APIs for developers to integrate S3.

Amazon Web Services 260

Amazon Web Services AWS Cloud Accessible 260

Rockset

MARCH 1, 2023

Every database built for real-time analytics has a fundamental limitation. When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. These two parts running on the same compute unit is what makes the database real-time: queries can reflect the effect of the new data that was just ingested.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

KDnuggets

MARCH 21, 2023

The first part covers the list of Programming, Web scraping, Statistics & Probability, Data Analytics, SQL, and Business Intelligence free courses.

Data Science 160

Data Science Business Intelligence SQL Data Analytics 160

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

DoorDash Engineering

MARCH 14, 2023

When dealing with failures in a microservice system, localized mitigation mechanisms like load shedding and circuit breakers have always been used, but they may not be as effective as a more globalized approach. These localized mechanisms ( as demonstrated in a systematic study on the subject published at SoCC 2022 ) are useful in preventing individual services from being overloaded, but they are not very effective in dealing with complex failures that involve interactions between services, whic

Java 135

Java Metadata Database Systems 135

The Pragmatic Engineer

MARCH 3, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in yesterday's subscriber-only The Scoop issue. To get full newsletters twice a week, subscribe here. On 22 February 2023, Google announced its coding competitions are coming to an end: The visual that accompanied the announcement of the end of Google’s coding competitions.

Coding 186

Coding IT Software Engineer Software Engineering 186

Tweag

MARCH 8, 2023

Topiary aims to be a universal formatter engine within the Tree-sitter ecosystem. Named after the art of clipping or trimming trees into fantastic shapes, it is designed for formatter authors and formatter users: Authors can create a formatter for a language without having to write their own formatting engine, or even their own parser. Users benefit from uniform, comparable code style, across multiple languages, with the convenience of a single formatter tool.

Coding 141

Coding Engineering Designing Programming 141

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. It is meant to handle massive volumes of data across many commodity servers while maintaining high availability with no single point of failure. Facebook created Cassandra, which ultimately became an Apache Software Foundation project. It is well-known for its rapid write […] The post Top 6 Cassandra Interview Questions appeared first on Analytics Vidhya.

NoSQL 251

NoSQL Database Project Systems 251

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. But before data can be transformed and served or shared, it must be ingested from source systems. The volume of data generated in real time from application databases, sensors, and mobile devices continues to grow exponentially.

Kafka 135

Kafka Data Ingestion Data Pipeline Cloud Storage 135

KDnuggets

MARCH 17, 2023

These curated papers would step up your machine-learning knowledge.

Machine Learning 160

Machine Learning 160

Lyft Engineering

MARCH 27, 2023

Authors: Remco van Bree , Ben Radler Contributors : Alex Ilyenko , Ben Radler , Francisco Souza , Garrett Heel , Nathan Hsieh , Remco van Bree , Shu Zheng , Alex Hartwell , Brian Witt “Load testing in production is great.” We know what you’re thinking — testing in production is one of the cardinal sins of software development. However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events.

Coding 134

Coding Database Systems Engineering 134

Confluent

MARCH 28, 2023

The future of data is real time and enriched by machine learning. How can we overcome socio-technical blockers and unite the ML and data streaming markets?

Machine Learning 133

Machine Learning Data 133

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

DoorDash Engineering

MARCH 21, 2023

While building a feature store to handle the massive growth of our machine-learning (“ML”) platform, we learned that using a mix of different databases can yield significant gains in efficiency and operational simplicity. We saw that using Redis for our online machine-learning storage was not efficient from a maintenance and cost perspective.

Machine Learning 132

Machine Learning AWS Database Utilities 132

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data. HDInsight works seamlessly with the Hadoop ecosystem, which includes technologies like MapReduce, Hive, […] The post Top 6 Microsoft HDFS Interview Questions appeared first on Analytics V

Hadoop 246

Hadoop Data Collection Cloud Technology 246

Christophe Blefari

MARCH 31, 2023

This newsletter is about money ( credits ) Dear readers, already 3 months done in 2023. We are slowly approaching the 2-years anniversary of the blog and the newsletter. We are almost 3000 and once again I want to thank you for the trust. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much.

Bytes 130

Bytes Data Google Cloud Education 130

KDnuggets

MARCH 2, 2023

The latest KDnuggets cheat sheet covers using ChatGPT to your advantage as a data scientist. It's time to master prompt engineering, and here is a handy reference for helping you along the way.

Data Science 160

Data Science Data Engineering IT 160

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Waitingforcode

MARCH 29, 2023

In my recent exploration of the compaction, aka OPTIMIZE command, in Delta Lake, I found this famous Z-Ordering mode. It was one of the most outstanding features when I first heard about Delta Lake. You can't even imagine how impatient I was to see what it is doing under-the-hood. Fortunately, this time has come!

IT 130

IT 130

Confessions of a Data Guy

MARCH 28, 2023

Real talk. Polars is all the rage. People love Spark. People use Spark for small data, but data is too big for Pandas. Spark runs on a local machine. Polars runs on a local machine. What do I choose, Spark or Polars? Does it matter? I’ve written about Polars at different points, here, and here […] The post Polars vs Spark. Real Talk. appeared first on Confessions of a Data Guy.

IT 130

IT Data 130

Data Engineering Podcast

MARCH 25, 2023

Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be. After experiencing this unfortunate reality for themselves, Abhishek Chauhan and Ashish Kumar founded Grainite so that you don't have to suffer the same pain.

MySQL 130

MySQL Python Architecture Machine Learning 130