Engineering at Meta

FEBRUARY 20, 2024

This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed. An introduction to Velox Velox is the first project in our composable data management system program.

Data Management 112

Data Management Bytes Management Datasets 112

Precisely

OCTOBER 23, 2023

When most people think of master data management, they first think of customers and products. The business must also manage locations, including warehouses, offices, and subsidiaries, not to mention the various addresses associated with virtually every data element the business manages. Four main challenges make MDM complex.

Data Management 110

Data Management Management Data Data Integration 110

The Pragmatic Engineer

APRIL 21, 2024

Other shipped things include  DALL·E 3  (image generation,)  GPT-4  (an advanced model,) and the  OpenAI API  which developers and companies use to integrate AI into their processes. I managed our entire Applied Engineering org from its earliest days through the launch and scaling of ChatGPT. 

Engineering 349

Engineering Software Engineer Software Engineering Programming 349

Snowflake

JUNE 11, 2024

Bringing machine learning (ML) models into production is often hindered by fragmented MLOps processes that are difficult to scale with the underlying data. The friction of having to set up and manage separate environments for features and models creates operational complexity that can be costly to maintain and difficult to use.

Management 94

Management Government Metadata Python 94

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 90

Metadata Management Data Governance Government 90

Tweag

NOVEMBER 15, 2023

Organist is a tool meant to do just that: Provide you with an ergonomic way of managing the complexity of your development environment. Asking contributors to apt get something just to contribute to your project is both a hindrance, and a source of failure and friction in the setup process. As much as possible, it should: Remain local.

Management 120

Management Python Programming Language Project 120

The Pragmatic Engineer

MAY 25, 2023

In every issue, I cover topics related to Big Tech and high-growth startups through the lens of engineering managers and senior engineers. ” I managed to talk to someone in this company’s HR department, who confirmed that the leadership set a goal to improve the business’s Glassdoor rating.

Software Engineer 316

Software Engineer Software Engineering AWS Engineering 316

The Pragmatic Engineer

OCTOBER 17, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Internal comms: Chat: Slack Coordination / project management: Linear 3. Source: Spare Cores 2.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Snowflake

JUNE 5, 2024

Addressing a lack of in-house AI expertise and simplifying AI processes can make adoption easier. Snowflake Cortex AI is a fully managed service designed to unlock the potential of the technology for everyone within an organization, regardless of their technical expertise. That’s where Snowflake comes in.

Coding 92

Coding Building Management Government 92

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. My goal was to fix the debt of hardcoded strings, but I learned a lot about the codebase and our process as I did it.  In 2004, I was hired by ISO-NE, a non-profit that manages the electric grid in New England. Big rewrites need heavyweight support.

Recruitment 235

Recruitment Java Coding Project 235

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. What are the requirements around governance and auditability of data access that need to be addressed when sharing data?

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it? Starburst : ![Starburst

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. What motivated to write a book about how to manage Kafka in production? Can you describe your experiences with Kafka? There are many options now for persistent data queues.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

The Pragmatic Engineer

OCTOBER 31, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones.” We apologize to all who are affected by the disruption.

AWS 233

AWS Google Cloud Cloud Engineering 233

The Pragmatic Engineer

MARCH 19, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research. Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. Closing Announcements Thank you for listening!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JUNE 30, 2024

Synq's platform helps data teams manage incidents, understand data dependencies, and ensure data quality by providing insights and automation capabilities. Petr emphasizes the need for a holistic approach to data reliability, integrating data systems into broader business processes. What do you have planned for the future of Synq?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Summary Any software system that survives long enough will require some form of migration or evolution.

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Snowflake

SEPTEMBER 13, 2023

Organizations can now leverage Salesforce data directly in Snowflake via zero-ETL data sharing to accelerate decision-making and help streamline business processes. Once the process is complete, the data share automatically becomes available to the Snowflake target account as secure views. and creates a “Data Share.”

Finance 112

Finance AWS Accessible Accessibility 112

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. What are the features and focus of Pieces that might encourage someone to use it over the alternatives?

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Who needs to be involved in the process of defining and developing that program? Closing Announcements Thank you for listening!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Jesse Anderson

MAY 9, 2023

We’re not allowed to see process beyond the Azure boundary, and in some cases it involves transfer by hand of source files from the GitHub private repos into an internal GitLab repo which we’re not allowed to see. But we’ve had to evolve a homegrown process that fits both teams.”

Software Engineer 147

Software Engineer Software Engineering Consulting Data 147

Snowflake

JUNE 5, 2024

With Snowpark’s existing DataFrame API , users have access to a robust framework for lazily evaluated, relational operations on data, closely resembling Spark’s conventions. pandas is the go-to data processing library for millions worldwide, including countless Snowflake users. Why introduce a distributed pandas API?

Python 113

Python Programming Language Government SQL 113

Striim

JANUARY 30, 2025

Todays organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. The Power of Real-Time Data in Crisis Management When a crisis unfolds, data moves at lightning speed.

Systems 52

Systems Management Hospitality Healthcare 52

Snowflake

JUNE 4, 2024

Embrace declarative and dynamic data pipelines with automated orchestration Snowflake is also excited to announce Snowflake Tasks, Dynamic Tables and Database Change Management (DCM), powerful new features designed to streamline your development workflow with declarative best practices and automated orchestration.

Data Pipeline 105

Data Pipeline Python SQL Government 105

Rockset

MARCH 1, 2023

The Challenge of Compute Contention At the heart of every real-time application you have this pattern that the data never stops coming in and requires continuous processing, and the queries never stop – whether they come from anomaly detectors that run 24x7 or end-user-facing analytics. So they are not suitable for real-time analytics.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

Snowflake

APRIL 8, 2024

BigGeo BigGeo accelerates geospatial data processing by optimizing performance and eliminating challenges typically associated with big data. Implentio Implentio is a centralized tool that helps ecommerce ops and finance teams efficiently and cost-effectively manage fulfillment and logistics spending.

Pipeline-centric 129

Pipeline-centric Food Healthcare Unstructured Data 129

The Pragmatic Engineer

MARCH 13, 2023

In the aftermath, we would learn that Silicon Valley Bank processed $42B in deposit outflows on Thursday: which is nearly a quarter of all deposits they held! The FDIC stated “all insured depositors will have full access to their insured deposits no later than Monday morning, March 13, 2023.”

Banking 191

Banking Insurance Portfolio Media 191

Snowflake

SEPTEMBER 7, 2023

We are excited to announce the public preview of External Access, which enables customers to reach external endpoints from Snowpark seamlessly and securely. With this announcement, External Access is in public preview on Amazon Web Services (AWS) regions. This has eliminated any additional cost or dependency on external orchestrators.

Amazon Web Services 122

Amazon Web Services AWS Government Python 122

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 104

Government Accessible Accessibility Cloud 104

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Closing Announcements Thank you for listening!

Database 147

Database Data Lake High Quality Data Data Workflow 147

The Pragmatic Engineer

APRIL 6, 2023

Our hope is that making salary ranges more accessible on Comprehensive.io For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing. Our system is using purely Serverless to process the data. ” How does Comprehensive.io

Software Engineer 191

Software Engineer Software Engineering Datasets Database 191

Data Engineering Podcast

FEBRUARY 25, 2024

When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process. Contact Info LinkedIn pauldix on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Database 162

Database Technology Data Lake High Quality Data 162

Snowflake

JUNE 10, 2024

This cohesive approach integrates Git version control, Python APIs, declarative object management and seamless CI/CD automation, and it offers powerful ways to: Maintain a single source of truth: With all files residing within Git, your data assets, code and configurations are centrally managed and version-controlled.

Python 109

Python Data Pipeline SQL Database 109

Snowflake

NOVEMBER 1, 2023

Snowflake has invested heavily in extending the Data Cloud to AI/ML workloads, starting in 2021 with the introduction of Snowpark , the set of libraries and runtimes in Snowflake that securely deploy and process Python and other popular programming languages.

Building 111

Building Python SQL Programming Language 111

Jesse Anderson

AUGUST 28, 2024

This survey was designed to get information about how management uses data teams, the value they’re creating, and how they’re creating it. The biggest and most ideal use of LLMs for data teams, data processing, is only used by 12% of teams and behind an API endpoint for 14%. In total, we had 81 respondents.

Consulting 147

Consulting Data Big Data Data Engineering 147