Analytics Vidhya
JUNE 25, 2023
In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. Imagine being an online shopper who suddenly receives irrelevant recommendations. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.
Data Engineering Podcast
JUNE 25, 2023
Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. SQLMesh was designed as a unifying tool that is simple to work with but powerful enough for large-scale transformations and complex projects.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
JUNE 30, 2023
1. Introduction 2. What is self-serve? 2.1. Components of a self-serve platform 3. Building a self-serve data platform 3.1. Creating dataset(s) 3.1.1. Gather requirements 3.1.2. Get data foundations right 3.2. Accessing data 3.3. Identify and remove dependencies 4. Conclusion 5. Further reading 6. References 1. Introduction Most companies want to build a self-serve data platform.
Waitingforcode
JUNE 30, 2023
Surprised? You shouldn't. I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure. But there is some excitement for learning-from scratch I miss.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Analytics Vidhya
JUNE 24, 2023
Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases. Properly configured data structures ensure a smoother workflow and prevent data loss or misplacement.
Confessions of a Data Guy
JUNE 28, 2023
I’ve been a dog licking my wounds for some time now. Over on my Substack newsletter, I’ve been doing a small series on DSA (Data Structures and Algorithms). I tackled some of the easier stuff first, like Linked Lists, Binary Search, and the like. What’s more, I actually did most of it in Rust, since […] The post Exploring Graphs in Rust.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
databricks
JUNE 29, 2023
Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™.
Analytics Vidhya
JUNE 24, 2023
Introduction We had an amazing opportunity to learn from Mr. Pavan. He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Throughout the conversation, Mr. Pavan shares his journey, inspirations, challenges, and accomplishments. Thus, providing valuable insights into the field of data engineering. As we explore Mr.
Engineering at Meta
JUNE 27, 2023
Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we talk about below are open source so you can try them yourself.
Lyft Engineering
JUNE 28, 2023
Written by Konstantin Gizdarski and Martin Liu at Lyft. In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. On the real-time front, LyftLearn supported real-time inference and input feature validation. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.
Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
KDnuggets
JUNE 27, 2023
The k-Nearest Neighbors Classifier is a machine learning algorithm that assigns a new data point to the most common class among its k closest neighbors. In this tutorial, you will learn the basic steps of building and applying this classifier in Python.
LinkedIn Engineering
JUNE 26, 2023
For the last several years, internal infrastructure at LinkedIn has been built around a self-service model, enabling developers to onboard themselves with minimal support. We have various user experiences that let application teams provision their own resources and infrastructure, generally by filling out forms or using command-line tools. For example, developers can provision Kafka topics, Espresso tables, Venice stores and more via Nuage , our internal cloud-like infra management platform.
databricks
JUNE 27, 2023
Generative AI will have a transformative impact on every business. Databricks has been pioneering AI innovations for a decade, actively collaborating with thousands.
Confluent
JUNE 28, 2023
Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
ThoughtSpot
JUNE 27, 2023
Active listening is an admired and sought after skill in both the professional and personal sphere. After all, who doesn’t love to be heard? But what happens when we apply that mindset to the way our organizations solicit feedback and interact with our customers? We don’t have to make any assumptions to answer this question, because we have the data.
Databand.ai
JUNE 28, 2023
Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. Specifically, observability provides insights into the pipeline’s internal states and how they interact with the system’s outputs. We believe the world’s data pipelines need better data observability.
Towards Data Science
JUNE 27, 2023
The Top 5 Features for Efficient Data Manipulation This April, pandas 2.0.0 was officially launched , making huge waves across the data science community. Photo by Yancy Min on Unsplash. Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. From data input/output to data cleaning and transformation, it’s nearly impossible to think about data manipulation without import pandas as pd, right ?
Knowledge Hut
JUNE 26, 2023
Having knowledge of real-world software applications or projects are very essential for any projects for backend developers aspiring software engineers or developers. The portfolio projects showcase their talents and skills whenever they try to look for new opportunities and jobs. This article is mainly focused on explaining different backend projects for beginners or students, intermediate learners, or those who have mid enough software development experience building large scalable projects.
Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage
When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.
ThoughtSpot
JUNE 26, 2023
At ThoughtSpot, we know how important it is for businesses of every size and industry to empower every knowledge worker with personalized, actionable data-driven insights. These insights are your secret sauce to making better business decisions, growing faster, and delivering customer experiences that keep people coming back for more. But how do you scale self-service analytics to business users without completely overwhelming your data teams?
databricks
JUNE 27, 2023
Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural.
Snowflake
JUNE 30, 2023
Since launching this year’s contest in October, receiving hundreds of submissions, and completing three rounds of judging, the wait is over: Maxa is the 2023 Snowflake Startup Challenge grand prize winner! Maxa’s goal is to automate financial and operations ERP insights extremely fast and without requiring special skills. To make that happen, it leverages the breadth of the Snowflake platform to transform raw data from multiple financial and operational systems into a common data model that user
Knowledge Hut
JUNE 26, 2023
Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article.
Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network
In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.
ThoughtSpot
JUNE 30, 2023
Pride is more than just a month-long celebration; it is a powerful movement that reminds us of the importance of equality, acceptance, and love. It is that special time of year for the global queer community to come together to celebrate, commemorate, and continue to push for progress. It’s no different here at ThoughtSpot. We believe in creating an inclusive environment where everyone feels seen, heard, and valued.
databricks
JUNE 27, 2023
Data teams face many challenges to quickly access the right data primarily due to data fragmentation, time and cost involved in consolidating data.
KDnuggets
JUNE 29, 2023
Large language models are getting released left right and center, and if you want to understand them better you need to know about NLP. Here are 5 Free books to help you.
Knowledge Hut
JUNE 26, 2023
Are you ready to navigate the ever-changing technological landscape? As a developer, you know that numerous frameworks and tools are at your disposal, aiming to simplify your job. When it comes to front-end frameworks, the JavaScript library takes the top spot. You may have encountered the dilemma of choosing between various JavaScript-based front-end frameworks like Angular and React, both highly favored by major organizations.
Speaker: Evelyn Chou
Choosing the right business intelligence (BI) platform can feel like navigating a maze of features, promises, and technical jargon. With so many options available, how can you ensure you’re making the right decision for your organization’s unique needs? 🤔 This webinar brings together expert insights to break down the complexities of BI solution vetting.
ThoughtSpot
JUNE 27, 2023
What is cookieless authentication? Amidst growing concerns around user privacy and regulatory laws, the cookieless paradigm has been gaining momentum over time in digital advertising. In addition, web browsers are increasingly blocking third-party cookies altogether in web sessions, necessitating the need for new authentication methods in web applications.
ArcGIS
JUNE 29, 2023
This third blog in a series provides a set of sample tools to migrate a personal geodatabase from ArcMap, to a file or mobile geodatabase in ArcGIS Pro.
KDnuggets
JUNE 30, 2023
Every job is at risk. Here’s how you can AI-proof your career.
databricks
JUNE 28, 2023
In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance.
Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL
Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.
Expert insights. Personalized for you.
220 
Let's personalize your content