Start Data Engineering

MAY 28, 2024

Project demo 3. Building efficient data pipelines with DuckDB 4.1. Introduction 2. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219

KDnuggets

AUGUST 7, 2024

Learn about essential AI tools that can help you develop Python projects faster and with fewer bugs using natural language.

Python 127

Python Project Building 127

Data Engineering Podcast

MAY 18, 2024

The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Are data agents harder to build? What are the most interesting, unexpected, or challenging lessons that you have learned while working on building an AI agent for business intelligence?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Start Data Engineering

SEPTEMBER 23, 2024

Steps to decide on a data project to build 2.1. Introduction 2. Objective 2.2. Research 2.2.1. Job description 2.2.2. Potential referral/hiring manager research 2.2.3. Company research 2.3. Data 2.3.1. Dataset Search 2.3.2. Generate fake data 2.4. Outcome 2.4.1. Visualization 2.5. Presentation 3. Conclusion 4. Read these 1.

Portfolio 130

Portfolio Project Datasets Data 130

ArcGIS

MARCH 20, 2023

Using ArcGIS Pro to build 'Human Geography' style vector basemaps in different projections, for use in ArcGIS Online

Project 103

Project Building 103

KDnuggets

NOVEMBER 15, 2024

In this article, we’ll look at how to build ETL pipelines for data science projects.

Data Science 113

Data Science Project Data Building 113

KDnuggets

JANUARY 4, 2022

Here are 15 neural network projects you can take on in 2022 to build your skills, your know-how, and your portfolio.

Deep Learning 159

Deep Learning Project Building Portfolio 159

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects.

Project 147

Project Data Lake High Quality Data SQL 147

Knowledge Hut

APRIL 29, 2024

In project management, project scheduling encompasses listing activities, defining milestones and scheduling deliverables for delivery. This indicates that every project schedule must include a planned start date and planned finish date, estimated resources assigned to each activity and estimated duration of each activity.

Project 98

Project Management Certification Algorithm 98

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

KDnuggets

OCTOBER 8, 2024

Are you a data science beginner looking to build your portfolio? Start working on these projects today.

Data Science 127

Data Science Project Portfolio Data 127

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Team Data Science

AUGUST 30, 2020

You want to become a data engineer, but don't know how to set up a data engineering project? Often people want to build the whole thing from the beginning. They say: "Okay I need to do a project. Then you build something on top of it. And then you build something on that. I will show you! Do not make this mistake!

Data Engineering 130

Data Engineering Data Engineer Project Building 130

Data Engineering Podcast

MARCH 31, 2024

Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. If you've learned something or tried out a project from the show then tell us about it! Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages. The solution to this?

Project 145

Project Programming Language Building Systems 145

Knowledge Hut

OCTOBER 23, 2023

Both the project leader and project manager roles are crucial to a project's success if project management is your area of interest as a career. Research and introspection are required to comprehend and decide which role is best for you, especially if you are interested in pursuing a career in project management.

Project 98

Project Management Generalist Algorithm 98

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

KDnuggets

AUGUST 29, 2022

This free online book is a fantastic resource on how to structure, manage, and maintain your real-world data science projects.

Data Science 160

Data Science Project Building Data 160

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. What are the core problems that you were addressing with this project? We feel your pain.

Building 147

Building Machine Learning Metadata Data Integration 147

Knowledge Hut

JANUARY 25, 2024

With over a decade of my experience in Project management, I might have crashed about 80% of my Project. Project Crashing is not a negative or a bad thing like it sounds, instead it serves as a strategy in project management, aimed at expediting project timelines without compromising the project's scope.

Project 98

Project Management Process Certification 98

ThoughtSpot

OCTOBER 18, 2024

In the fast-paced world of software development, the efficiency of build processes plays a crucial role in maintaining productivity and code quality. At ThoughtSpot , while Gradle has been effective, the growing complexity of our projects demanded a more sophisticated approach to understanding and optimizing our builds.

Building 52

Building Process Pipeline-centric Database-centric 52

KDnuggets

APRIL 25, 2022

Check out this list of data science project ideas that you can use to boost your skills, organized by level of expertise.

Data Science 123

Data Science Project Building Data 123

Engineering at Meta

APRIL 6, 2023

Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1.

Building 144

Building Systems Java Coding 144

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. invoke build buck2 build //starlark-rust/starlark 6.

Systems 141

Systems Building Java Programming Language 141

Knowledge Hut

APRIL 30, 2024

Management is paramount to making any project successful, irrespective of the industry and the work scope. Moreover, there are varied aspects to manage to ensure successful project delivery. These experts follow well-established principles of project management and efficiently manage the complexities to deliver expected end products.

Project 98

Project Management Certification Programming 98

Knowledge Hut

MAY 6, 2024

The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.

Project 98

Project Programming Certification Database 98

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Precisely

SEPTEMBER 26, 2024

Building data literacy across your organization empowers teams to make better use of AI tools. However, achieving success in AI projects isn’t just about deploying advanced algorithms or machine learning models. The real challenge lies in ensuring that the data powering your projects is AI-ready.

Project 75

Project Government Data Governance Data Integration 75