Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.

Data Engineer 240

Data Engineer Data Engineering Engineering Data 240

Confessions of a Data Guy

FEBRUARY 10, 2023

Hmm … data types. We all know they are important, but we don’t take them very seriously. I mean we know the difference between boolean, string, and integers, those are easy to get right. But we all get sloppy, sometimes we got the string and varchar route because we don’t spend enough time on the […] The post Data Types in Delta Lake + Spark.

Data 238

Data Big Data Data Engineering Data Engineer 238

Data Engineering Podcast

FEBRUARY 5, 2023

Summary This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done. In this episode I reflect on some of the major themes and take a brief look forward at some of the upcoming changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Your host is Tobias Macey and today I'm reflecting on the m

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Waitingforcode

FEBRUARY 9, 2023

Compaction is also a feature present in Apache Iceberg. However, it works a little bit differently than for Delta Lake presented last time. Why? Let's see in this new blog post!

IT 130

IT 130

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Analytics Vidhya

FEBRUARY 6, 2023

Introduction A design pattern is simply a repeatable solution for problems that keep on reoccurring. The pattern is not an actual code but a template that can be used to solve problems in different situations. Especially while working with databases, it is often considered a good practice to follow a design pattern. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python?

Accessibility 234

Accessibility Accessible Python Database 234

Confessions of a Data Guy

FEBRUARY 7, 2023

As I started to use Rust on and off, more out of curiosity than anything, I discovered some specs of gold buried down in the depths. Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineer 221

Data Engineer Data Engineering Engineering Data 221

Engineering at Meta

FEBRUARY 6, 2023

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012 , it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration , the Facebook codebase does not resemble a typical iOS codebase: It’s full of C++, Objective-C(++), and Swift.

Architecture 144

Architecture Coding Engineering Systems 144

Analytics Vidhya

FEBRUARY 8, 2023

Introduction Amazon Redshift is a fully managed, petabyte-scale data warehousing Amazon Web Services (AWS). It allows users to easily set up, operate, and scale a data warehouse in the cloud. Redshift uses columnar storage techniques to store data efficiently and supports data warehousing workloads intelligence, reporting, and analytics. It allows users to perform complex queries […] The post Top 6 Amazon Redshift Interview Questions appeared first on Analytics Vidhya.

Amazon Web Services 221

Amazon Web Services AWS Data Warehouse Cloud 221

Edureka

FEBRUARY 8, 2023

We are introduced to new discoveries and technologies every day, and one of the best and most popular inventions today is artificial intelligence (AI) and its tools. One of them is Chat GPT, a conversational model of AI that is a powerful chatbot that answers follow-up questions and writes code for the users. The day it was launched, everybody was going gaga over the new technology and the remarkable uses of this AI-powered chatbot.

Coding 130

Coding Deep Learning Programming Java 130

Confluent

FEBRUARY 8, 2023

Learn what windowing is, the difference between the four types of windows (hopping and tumbling, or session and sliding), and how to create them.

Kafka 140

Kafka 140

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Engineering at Meta

FEBRUARY 7, 2023

A lot has changed since the initial launch of our basemap in late 2020. We’re Meta now, but our mission remains the same: Giving people the power to build community and bring the world closer together. Across Meta, our family of applications (Facebook, Instagram, WhatsApp, among others) are using our basemap to connect people through functions like status updates, location sharing, and location-based searching.

Entertainment 138

Entertainment Transportation Data Schemas AWS 138

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project. Managing these different python versions and different versions of packages is […] The post Isolated Python Environments using Docker appeared first on Analytics Vidhya.

Python 218

Python Project Management Data Engineering 218

DoorDash Engineering

FEBRUARY 7, 2023

It would be almost impossible to build a scalable backend without a scalable datastore. DoorDash’s expansion from food delivery into new verticals like convenience and grocery introduced a number of new business challenges that would need to be supported by our technical stack. This business expansion not only increased the number of integrated merchants dramatically but also exponentially increased the number of menu items, as stores have much larger and more complicated inventories than typica

PostgreSQL 119

PostgreSQL SQL Retail Database 119

Teradata

FEBRUARY 9, 2023

In the aftermath of the 2008 financial crash, service providers have been subject to increasing rules & requirements. To what extent has this climate held back advances in data analytics?

Data Analytics 116

Data Analytics Data 116

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

KDnuggets

FEBRUARY 7, 2023

Let's learn how ChatGPT can help us learn about Python. or really anything at all.

Python 160

Python 160

Analytics Vidhya

FEBRUARY 6, 2023

Introduction The demand for data to feed machine learning models, data science research, and time-sensitive insights is higher than ever thus, processing the data becomes complex. To make these processes efficient, data pipelines are necessary. Data engineers specialize in building and maintaining these data pipelines that underpin the analytics ecosystem.

Amazon Web Services 218

Amazon Web Services Data Pipeline Machine Learning Data Science 218

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments. The step towards automation was an important improvement for us, as the previous setup was manual, slow, and error-prone.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

ThoughtSpot

FEBRUARY 9, 2023

Two years ago, we announced our Databricks partnership —including the launch of ThoughtSpot for Databricks, which gives joint customers the ability to run ThoughtSpot search queries directly on the Databricks Lakehouse without the need to move any data. Since then, we’ve empowered teams at companies like Johnson & Johnson, NASDAQ, and Flyr to safely self-serve business-critical insights on governed and reliable data.

Government 105

Government SQL Machine Learning Cloud 105

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

KDnuggets

FEBRUARY 8, 2023

This post provides an overview of topics in linear programming, history, and recent advances, software packages, common problem specifications, and a case study using Toronto shelters data and the PuLP software package.

Programming 108

Programming Data 108

Analytics Vidhya

FEBRUARY 8, 2023

source: svitla.com Introduction Before jumping to the data warehouse interview questions, let’s first understand the overview of a data warehouse. A data warehouse is a system used for collecting and managing large amounts of data from various sources, such as transactional systems, log files, and external data sources. The data is then organized and structured […] The post Data Warehouse Interview Questions appeared first on Analytics Vidhya.

Data Warehouse 214

Data Warehouse Data Systems Management 214

Tweag

FEBRUARY 8, 2023

In this post I want to explore the current issues with developing parts of NixOS on macOS and how we can make this task easier. Why would I want to run a NixOS virtual machine on macOS? My colleague at Tweag, Dominic Steinitz, asked me this question after I shared my first minor achievement in this area, and it struck me that I have never described why exactly I run virtual machines (VMs) on my laptop and why I want to make it easier for myself (and others).

Systems 103

Systems Building Cloud IT 103

Knowledge Hut

FEBRUARY 7, 2023

In this article, we'll discuss the importance of having a Kubernetes cheat sheet available so you can quickly get up-to-speed with this powerful tool before diving into more advanced topics like deployments and automation. What is Kubernetes? Kubernetes is an open-source system for automating containerized applications' deployment, scaling, and management.

Architecture 98

Architecture Accessible Accessibility Management 98

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project

KDnuggets

FEBRUARY 6, 2023

Collecting and labeling data in the real world can be time-consuming and expensive. This data can also come with quality, diversity, and quantity issues. Fortunately, problems like these can be helped with synthetic data.

Data 108

Data 108

Analytics Vidhya

FEBRUARY 5, 2023

Introduction In this technical era, Big Data is proven as revolutionary as it is growing unexpectedly. According to the survey reports, around 90% of the present data was generated only in the past two years. Big data is nothing but the vast volume of datasets measured in terabytes or petabytes or even more. Big data […] The post A Beginner’s Guide to the Basics of Big Data and Hadoop appeared first on Analytics Vidhya.

Big Data 205

Big Data Hadoop Datasets Data 205

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? AWS Glue is here to put an end to all your worries! Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 billion by 2026? Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

databricks

FEBRUARY 6, 2023

Advances in Natural Language Processing (NLP) have unlocked unprecedented opportunities for businesses to get value out of their text data. Natural Language Processing.

Process 101

Process Data Data Science Engineering 101

Speaker: Evelyn Chou

Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.

Business Intelligence

KDnuggets

FEBRUARY 7, 2023

Kickstart your Data Engineering career with these curated GitHub repositories.

Data Engineer 160

Data Engineer Data Engineering Engineering Data 160

Analytics Vidhya

FEBRUARY 8, 2023

Introduction The February installment of the webinar series is now open! It’s a farewell time to your quest for finding the ideal data science learning platform, as Analytics Vidhya has arrived. Explore your ultimate data science destination where the emphasis is on supporting the community and fostering professional development. Attend expert-led DataHour sessions to boost […] The post February DataHour: Enhance Your Skills with Expert Sessions appeared first on Analytics Vidhya.

Data Science 200

Data Science Data Machine Learning Data Engineering 200

Towards Data Science

FEBRUARY 10, 2023

Some organizations approach data quality like a crash diet. Instead, try these 6 steps for sustainable data quality at scale. Photo by Bruno Nascimento on Unsplash I like to think of operationalizing data reliability within the context of physical fitness. You can get in shape with hard work, but staying in shape requires good habits. And above all else it’s a mindset and a lifestyle.

Data Pipeline 93

Data Pipeline Data Cleanse Machine Learning Data 93

Confluent

FEBRUARY 7, 2023

Migrate Kafka clusters from ZooKeeper to KRaft with no downtime (early access), get improvements for Kafka Streams and Kafka Connect, and more.

Kafka 105

Kafka Accessible Accessibility 105

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government