The Pragmatic Engineer

NOVEMBER 21, 2023

  I remember the physicality of our systems; the blinking lights of the network traffic, having to put thermometers on servers because we were using converted conference rooms as data centers, and how we put a tarp in one corner because the roof leaked! Our deployments were initially manual. And it lost pointers to all its resources!

Engineering 301

Engineering Bytes Cloud Computing AWS 301

Data Engineering Podcast

FEBRUARY 11, 2024

Summary Sharing data is a simple concept, but complicated to implement well. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

The Pragmatic Engineer

APRIL 21, 2024

Other shipped things include  DALL·E 3  (image generation,)  GPT-4  (an advanced model,) and the  OpenAI API  which developers and companies use to integrate AI into their processes. Each word that spits out of ChatGPT is this same process repeated over and over again many times per second.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. Can you start by sharing some of your experiences with data migration projects?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Start Data Engineering

OCTOBER 14, 2024

Code & Data 3. Using nested data types effectively 3.1. Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Nested data types can be sorted 3.3.3. Introduction 2.

SQL 130

SQL Data Schemas Data Coding 130

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Jesse Anderson

MAY 9, 2023

The results and analysis from my 2023 Data Teams Survey left a few open questions. One striking commonality is that so many companies are using data mesh. One striking commonality is that so many companies are using data mesh. But we’ve had to evolve a homegrown process that fits both teams.”

Software Engineer 147

Software Engineer Software Engineering Consulting Data 147

Marc Lamberti

SEPTEMBER 22, 2023

Wondering how to share data between tasks? At the end of this tutorial, you will have a solid knowledge of XComs and be able to share data between your tasks efficiently. XCom stands for “cross-communication” and allows data exchange between tasks. To access XComs, go to the user interface, then Admin and XComs.

MySQL 246

MySQL Data Pipeline Database Python 246

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Jesse Anderson

AUGUST 28, 2024

In the spring of 2024, I ran a new survey to gather more data for my Data Teams book and update my 2023 and 2020 surveys. This survey was designed to get information about how management uses data teams, the value they’re creating, and how they’re creating it. We start by asking some questions about each respondent’s data team.

Data 147

Data Consulting Big Data Data Engineering 147

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Storing data: data collected is stored to allow for historical comparisons. Visualizing the data: the frontend that allows querying of live and historic data. Each benchmarking task is evaluated sequentially.

Cloud 277

Cloud AWS Metadata Cloud Computing 277

Christophe Blefari

FEBRUARY 3, 2024

Anyway this week will be a mixed Data News with links, stuff and ideas and a small wrap-up of the DuckCon + the stuff I presented on Wed. to a Modern Data Stack meetup in Paris about DuckDB WASM. State of the French data market 2 benchmarks have been published recently about the French data market. conference.

MongoDB 130

MongoDB SQL Data Data Warehouse 130

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Big Data 147

Big Data Cloud Storage Hadoop SQL 147

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. Why CDC is More Relevant in Unified Data Architecture As we advance into the Gen AI era, Change Data Capture (CDC) systems are emerging as crucial components of the ever-evolving data architecture.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. A few highlights from the report Unstructured data goes mainstream. AI-driven code development is going mainstream now.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Snowflake

FEBRUARY 6, 2024

Maverick replies, “The data on the MIG is inaccurate.” While flying may be more automated now, the importance of accurate and diverse data for aviation safety remains — and is likely even more critical. In two recent airplane accidents, automated systems aboard a Boeing 737 MAX made decisions based on inaccurate data.

Unstructured Data 107

Unstructured Data Retail Data Manufacturing 107

Christophe Blefari

JANUARY 26, 2024

Building a custom HTTP API, Docker image and CI/CD processes making it accessible on internet. We talke about data engineering, freelancing and career stuff. We talke about data engineering, freelancing and career stuff. Data mesh means decentralisation but more importantly ownership and responsibilities to team (esp.

Algorithm 130

Algorithm Google Cloud Data Java 130

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Christophe Blefari

APRIL 14, 2023

Anyway, here the weekly Data News, written faster than usual. Hot takes on the Modern Data Stack — Matt gives 5 hot takes about the MDS. This time he writes about the new marketing approach of the modern data stack ecosystem. Data Economy 💰 Cybersyn raises $62.9m

Datasets 130

Datasets Data Deep Learning SQL 130

Snowflake

JANUARY 15, 2024

Leading companies around the world rely on Informatica data management solutions to manage and integrate data across various platforms from virtually any data source and on any cloud. Now, Informatica customers in the Snowflake ecosystem have an even easier way to integrate data to and from the Snowflake Data Cloud.

Data Integration 104

Data Integration BI Cloud Data 104

Cloudera

NOVEMBER 13, 2024

We are excited to announce the acquisition of Octopai , a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Metadata 92

Metadata Management Data Governance Government 92

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing 130

Designing Data Lake High Quality Data SQL 130

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights.

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. We feel your pain.

IT 147

IT Data Lake Metadata Data Warehouse 147

Snowflake

FEBRUARY 15, 2024

At Snowflake, we believe in empowering our customers to harness the full potential of their data while maintaining robust compliance standards and safeguarding data privacy. You can then protect the data with a row access policy, masking policy or both. Ready to try Data Classification for yourself?

Data 104

Data Data Governance Government Data Analysis 104

Snowflake

APRIL 1, 2024

Snowflake and Salesforce are happy to share that bidirectional data sharing between Snowflake, the Data Cloud company and Salesforce Data Cloud is now generally available. Enterprises generate valuable customer data inside Salesforce applications. It has opened up new frontiers of data collaboration.

Cloud 119

Cloud Retail Entertainment Media 119

Christophe Blefari

MARCH 24, 2023

📺 Watch the full replay Here are my takeaways about the event: Mage and Kestra have been both developed with Airflow flaws in mind, especially about deployment complexity, reusability and data sharing between tasks. Out of the box Mage provide all-in-one web editor to write data pipelines with a great UX.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Snowflake

FEBRUARY 5, 2024

And it’s no wonder — this new technology has the potential to revolutionize the industry by augmenting the value of employee work, driving organizational efficiencies, providing personalized customer experiences, and uncovering new insights from vast amounts of data. One potential source is data sharing.

Unstructured Data 105

Unstructured Data Banking Government Media 105

Data Engineering Weekly

JUNE 16, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Regulative requirements and privacy concerns are often a big hurdle to training context-rich data.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130