phData: Data Engineering

DECEMBER 18, 2023

Data engineering refers to the design of systems that are capable of collecting, analyzing, and storing data at a large scale. In manufacturing, data engineering aids in optimizing operations and enhancing productivity while ensuring curated data that is both compliant and high in integrity. The increased efficiency in data “wrangling” means that more accurate modeling and planning may be done, enabling manufacturers to make stronger data-driven decisions.

Manufacturing 52

Manufacturing Data Engineer Data Engineering Engineering 52

The Pragmatic Engineer

DECEMBER 18, 2023

I get asked every now and then if I offer 1:1 mentoring for either software engineers or engineering managers or leaders. While I used to do this in the past, I don't offer this any more. I collected much of the advice I have to offer for software engineers in The Software Engineer's Guidebook. I also write The Pragmatic Engineer Newsletter where I do cover topics like what it means to be a senior engineer at various companies , how to deal with a low-quality engineering culture , and

Software Engineer 206

Software Engineer Software Engineering Engineering Management 206

Analytics Vidhya

DECEMBER 18, 2023

Introduction Docker containers have emerged as indispensable tools in the fast-evolving landscape of software development and deployment, providing a lightweight and efficient way to package, distribute, and run applications. This article delves into the top 20 Docker containers across various categories, showcasing their features, use cases, and contributions to streamlining development workflows.

178

178

KDnuggets

DECEMBER 22, 2023

Learn how to integrate advanced AI multimodal models into your project using a simple Python API.

Accessibility 156

Accessibility Accessible Python Project 156

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

databricks

DECEMBER 21, 2023

We are excited to announce that Gartner has recognized Databricks as a Leader for a third consecutive year in the 2023 Gartner® Magic.

Database 145

Database Systems Cloud Management 145

The Pragmatic Engineer

DECEMBER 21, 2023

2023 was the second full year of The Pragmatic Engineer Newsletter , and this newsletter is now almost two and a half years old; the first issue came out on 26 August 2021. Thank you for being a reader, I greatly value your support. This year, 102 newsletter issues were published, and this is number 103. You received a deepdive issue on Tuesdays, and every Thursday it was  “The Pulse”  – formerly The Scoop.

Engineering 180

Engineering Software Engineer Software Engineering Project 180

KDnuggets

DECEMBER 18, 2023

Looking for a great course to learn Artificial Intelligence with Python? Check out this free course from Harvard University.

Python 152

Python 152

databricks

DECEMBER 21, 2023

Today, Databricks is excited to announce support for Mixtral 8x7B in Model Serving. Mixtral 8x7B is a sparse Mixture of Experts (MoE) open.

Engineering 140

Engineering 140

ArcGIS

DECEMBER 19, 2023

Here's how to fake an isometric 3D topo terrain in 2D! And stuff.

138

138

Waitingforcode

DECEMBER 19, 2023

Even though nowadays data processing frameworks and data stores have smart query planners, they don't take our responsibility to correctly design the job logic.

Designing 130

Designing Data Process Process Data 130

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

KDnuggets

DECEMBER 22, 2023

Here are our top posts of 2023, including: 5 Free Books to Master Data Science • 5 Free Courses to Master Machine Learning • 3 Ways to Access GPT-4 for Free • and much more!

Machine Learning 149

Machine Learning Data Science Accessibility Accessible 149

databricks

DECEMBER 21, 2023

Over the past six months, we've been working with NVIDIA to get the most out of their new TensorRT-LLM library. TensorRT-LLM provides an easy-to-use Python interface to integrate with a web server for fast, efficient inference performance with LLMs. In this post, we're highlighting some key areas where our collaboration with NVIDIA has been particularly important.

Python 132

Python 132

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering effort on integrating systems that weren't designed with compatible user experiences. The team at 5X understand the pain involved and the barriers to productivity and set out to solve it by pre-integrating the best tools from each layer of the s

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Engineering at Meta

DECEMBER 19, 2023

HawkEye is the powerful toolkit used internally at Meta for monitoring, observability, and debuggability of the end-to-end machine learning (ML) workflow that powers ML-based products. HawkEye supports recommendation and ranking models across several products at Meta. Over the past two years, it has facilitated order of magnitude improvements in the time spent debugging production issues.

Data Pipeline 122

Data Pipeline Algorithm Architecture Machine Learning 122

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

KDnuggets

DECEMBER 21, 2023

LLM is a powerful model that could improve our text analysis. With Scikit-LLM, we could integrate the LLM easily into our ML pipeline.

149

149

Snowflake

DECEMBER 20, 2023

The rise of generative AI (gen AI) is inspiring organizations to envision a future in which AI is integrated into all aspects of their operations for a more human, personalized and efficient customer experience. However, getting the required compute infrastructure into place, particularly GPUs for large language models (LLMs), is a real challenge. Accessing the necessary resources from cloud providers demands careful planning and up to month-long wait times due to the high demand for GPUs.

Scala 122

Scala Government Java Cloud 122

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Max Kanat-Alexander and Grant Jenks Today we are open-sourcing the LinkedIn Developer Productivity & Happiness Framework (DPH Framework) - a collection of documents that describe the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn. Now more than ever, developers are navigating so much change and new opportunity in this new era of Generative AI, so ensuring teams have the systems, processes, metrics and f

Data Schemas 121

Data Schemas Software Engineer Software Engineering Data Pipeline 121

ArcGIS

DECEMBER 20, 2023

Check out this blog to see how we enhanced several geoprocessing controls with additional features and new designs in ArcGIS Pro 3.2.

Designing 119

Designing 119

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

KDnuggets

DECEMBER 20, 2023

We will learn a simple way to install and use Llama 2 without setting up Python or any program. Just download the files and run a command in PowerShell.

Python 149

Python Programming 149

Snowflake

DECEMBER 18, 2023

When businesses share sensitive first-party data with outside partners or customers, they must do so in a way that meets strict governance requirements around security and privacy. Data clean rooms have emerged as the technology to meet this need, enabling interoperability where multiple parties can collaborate on and analyze sensitive data in a governed way without exposing direct access to the underlying data and business logic.

Cloud 120

Cloud Building Entertainment Government 120

databricks

DECEMBER 19, 2023

Request a meeting with Databricks executives/thought leaders at NRF! Each January, thousands of leaders from retailers around the globe gather at Javits Center.

Retail 115

Retail Data 115

Confessions of a Data Guy

DECEMBER 21, 2023

Sometimes I just need something new and interesting to work on, to keep me engaged. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above. Thinking about and contemplating life and data engineering … something flitted across my […] The post Datafusion SQL CLI – Look Ma, I made a new ETL tool. appeared first on Confessions of a Data Guy.

ETL Tools 113

ETL Tools SQL Data Engineer Data Engineering 113

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

KDnuggets

DECEMBER 19, 2023

There are many data-learning materials locked up behind expensive books. These cheap books would bolster your skills without blowing up your savings.

Data Science 149

Data Science Data 149

ThoughtSpot

DECEMBER 19, 2023

2023 has been a year of breakthrough innovation for many, and a deer-in-headlights moment for others. I keep flashing back to the 90s when the Internet created new businesses and destroyed others—LLMs are doing the same, only with more velocity. From CDAOs to VCs alike, the rate of creative destruction is faster, but there is also an intense focus on value.

Entertainment 111

Entertainment Data Big Data Data Analytics 111

Knowledge Hut

DECEMBER 22, 2023

Technology is evolving at breakneck speed, and the information we consume every day continues to grow exponentially with every passing day. Analyzing this complex mountain of data to make the right decisions informed by this data has become ever more challenging. Traditional models of project management , like the waterfall method and hierarchical team structures are too rigid to respond to the fast-paced change organizations are facing today.

Machine Learning 105

Machine Learning Portfolio Cloud Computing Certification 105

databricks

DECEMBER 22, 2023

Introduction Anomaly detection is widely applied across various industries, playing a significant role in the enterprise sector. This blog focuses on its application.

Machine Learning 105

Machine Learning Building IT 105

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

KDnuggets

DECEMBER 20, 2023

We've partnered with Springboard, the leading data science bootcamp offering personalized 1:1 mentorship, dedicated career support, proven outcomes, and an unbeatable money-back job guarantee, to present a handpicked collection of resources to supercharge your data science journey in the coming year.

Data Science 148

Data Science Data 148

ArcGIS

DECEMBER 19, 2023

We will dive into our best practices for preparing and using training samples for object detection models.

Data Preparation 105

Data Preparation Data Deep Learning 105

Knowledge Hut

DECEMBER 22, 2023

Wondering what it would take for your business to become a pioneer in web development this year? Below are the top web development trends that are unmissable in 2024. Whether you are a full stack developer or have an organization that implements full stack, the list below can be quite handy. Web Development Trends In 2024 1. Voice Search Benefits: Improved User Experience Accessibility Hands-free Interaction Faster Search Results Examples: Integrating voice-enabled search boxes or chatbots on we

Entertainment 105

Entertainment Machine Learning Utilities Google Cloud 105

databricks

DECEMBER 18, 2023

An effective campaign can help improve a company's revenue by increasing the sales of its products, clearing out more stock, bringing in more.

Data 103

Data IT 103

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud