Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineer 251

Data Engineer Data Engineering Engineering Data 251

The Pragmatic Engineer

APRIL 6, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. If you’re not yet a full subscriber, you missed this week’s deep-dive into Figma’s engineering culture. To get full newsletters twice a week, subscribe here.

Software Engineer 194

Software Engineer Software Engineering Datasets Database 194

KDnuggets

APRIL 6, 2023

With more and more companies integrating artificial intelligence into the workplace, what does this mean for employees' futures and careers?

160

160

Engineering at Meta

APRIL 6, 2023

Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open source large-scale build system, is now publicly available via the Buck2 website and the Buck2 GitHub repository.

Building 145

Building Systems Java Coding 145

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

ArcGIS

APRIL 7, 2023

We're happy to announce the conda init command is now enabled for ArcGIS users of Python! Learn about how to use it, how it works, and benefits.

Python 130

Python IT 130

Waitingforcode

APRIL 7, 2023

Last time you discovered the Z-Order compaction in Delta Lake. But guess what? Apache Iceberg also has this feature!

130

130

Confessions of a Data Guy

APRIL 6, 2023

The post QuickSort in Rust! appeared first on Confessions of a Data Guy.

Data 130

Data 130

Simon Späti

APRIL 3, 2023

Amidst the excitement and hype surrounding artificial intelligence, the significance of data engineering and its critical foundation—data modeling—can often be overlooked. This article is the first in a three-part series that will shine a spotlight on the fascinating world of data modeling, delving into its crucial importance within the broader context of data engineering.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Architecture 130

Data Engineering Podcast

APRIL 2, 2023

Summary The data ecosystem has been building momentum for several years now. As a venture capital investor Matt Turck has been trying to keep track of the main trends and has compiled his findings into the MAD (ML, AI, and Data) landscape reports each year. In this episode he shares his experiences building those reports and the perspective he has gained from the exercise.

Hadoop 130

Hadoop Machine Learning Python Architecture 130

KDnuggets

APRIL 4, 2023

Tutorial on data cleaning techniques using Python.

Python 159

Python Data Data Science 159

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

LinkedIn Engineering

APRIL 4, 2023

Authors: Viktoras Truchanovicius and Selina Zhang At LinkedIn, developer productivity and happiness has always been a priority. It is critical for our engineering leaders to understand how efficiently and effectively their teams are operating to continuously deliver value-added features for our members and build an industry-leading engineering culture.

MySQL 122

MySQL Datasets Software Engineer Software Engineering 122

Simon Späti

APRIL 3, 2023

Amidst the excitement and hype surrounding artificial intelligence, the significance of data engineering and its critical foundation—data modeling—can often be overlooked. This article is the first in a three-part series that will shine a spotlight on the fascinating world of data modeling, delving into its crucial importance within the broader context of data engineering.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Architecture 130

ArcGIS

APRIL 6, 2023

Organizations with IFC files can still reap the benefits of an ArcGIS Indoors deployment by following these recommendations.

Data Management 115

Data Management Architecture Management Engineering 115

KDnuggets

APRIL 3, 2023

This article will explain how RAPIDS can help you speed up your next data science workflow. RAPIDS cuDF is a GPU DataFrame library that allows you to produce your end-to-end data science pipeline development all on GPU.

Data Science 156

Data Science Data 156

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Snowflake

APRIL 7, 2023

Spring has sprung—and with it comes a new crop of Snowflake Startup Challenge semi-finalists! The 2023 submission pool was the largest to date—twice as many submissions as last year—with entries that spanned not just the globe but the breadth of the Snowflake platform. Our judges put a lot of careful consideration into selecting the top 10, and we offer our sincere thanks to every company that sent in an entry this year—we know how much hard work goes into these submissions, and we appreciate it

Raw Data 111

Raw Data Portfolio Building SQL 111

Preset

APRIL 5, 2023

Entity-centric modeling is a data modeling approach focusing on enriching tabular datasets with useful "features" to enable segmentation, cohort creation, and complex classification analyses easier.

Datasets 111

Datasets Data 111

ArcGIS

APRIL 4, 2023

ArcGIS Pro 3.1 introduces a new layer type—catalog layers—and this blog covers how they could be used in your analytic workflows.

Building 115

Building Datasets Data Management Management 115

KDnuggets

APRIL 6, 2023

Discover the widely-used open-source frameworks and models for creating your ChatGPT like chatbots, integrating LLMs, or launching your AI product.

Process 149

Process 149

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Confluent

APRIL 5, 2023

All the best data streaming resources, tips, and guides to help you learn introductory concepts, streaming architecture basics, common tools and technologies, and more.

Architecture 105

Architecture Data Technology 105

databricks

APRIL 6, 2023

What is GxP? GxP stands for "Good x Practices," where x refers to a specific discipline, such as clinical, manufacturing, or laboratory. The.

Manufacturing 98

Manufacturing 98

Tweag

APRIL 3, 2023

For a software team to be successful, you need excellent communication. That is why we want to build systems that foster cross-team communication. Using a monorepo is an excellent way to do that. A monorepo provides: Visibility: by seeing the pull requests (PRs) of colleagues, you are easily informed of what other teams are doing. Uniformity: by working in one central repository, it is easier to share the configuration of linters, formatters, etc.

Python 98

Python Coding Project Data Science 98

KDnuggets

APRIL 6, 2023

I will be sharing a couple of things I have learned in the past six months and tips that helped me stay dedicated and true to my journey in this article.

Data Science 149

Data Science Data 149

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Confluent

APRIL 4, 2023

Machine learning and data streaming are a perfect match, but have diverging tech stacks. How can we overcome the pitfalls of SQL and the gulf between languages?

Machine Learning 105

Machine Learning SQL Data 105

databricks

APRIL 4, 2023

Databricks is excited to announce the release of several exciting new Workflows features that will simplify the way you create and launch automated.

98

98

The Modern Data Company

APRIL 7, 2023

Technical debt is something that many companies are aware of and are attempting to address. It is a big enough issue that several of our recent blog posts ( Lessons in Technical Debt from Southwest Airlines , Start Paying Down Your Technical Debt Today , and A Better Way to Plan the Payoff of Technical Debt) discussed it at length. What about data debt?

Management 98

Management Government Data Data Management 98

KDnuggets

APRIL 7, 2023

Utilizing the power of GPT-3.5 to develop a simple summarize generator application.

Python 149

Python Utilities Process 149

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

LinkedIn Engineering

APRIL 6, 2023

It’s been an exciting few months at LinkedIn, as our engineering and product teams have been working hard to build some new and advanced AI-powered experiences for our members and customers. I have the opportunity to sit at such a unique vantage point where I get to see first hand the work that went into setting the technology foundations - from the technical resources, tools, engineering playgrounds and guidelines - to make it all possible.

Accessibility 94

Accessibility Accessible Technology Engineering 94

databricks

APRIL 5, 2023

To simplify navigating in Databricks, we are releasing a new workspace browsing experience. The new Workspace Browser makes it easier for you to.

IT 98

IT 98

Cloudera

APRIL 3, 2023

Cloudera Contributors: Ayush Saxena, Tamas Mate, Simhadri Govindappa Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), we are excited to see customers testing their analytic workloads on Iceberg. We are also receiving several requests to share more details on how key data services in CDP, such as Cloudera Data Warehousing ( CDW ), Cloudera Data Engineering ( CDE ), Cloudera Machine Learning ( CML ), Cloudera Data Flow ( CDF ) and Cloudera Stream Proce

Data Warehouse 87

Data Warehouse Java Metadata Data 87

KDnuggets

APRIL 4, 2023

Learn how Bard, Bing, ChatGPT, GitHub Copilot, and Hugging Face are improving data scientists' work life.

Data Science 137

Data Science Data 137

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud