The Pragmatic Engineer

NOVEMBER 21, 2023

Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process.  After this zero-byte file was deployed to prod, the Apache web server processes slowly picked up the empty configuration file. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

APRIL 21, 2024

Other shipped things include  DALL·E 3  (image generation,)  GPT-4  (an advanced model,) and the  OpenAI API  which developers and companies use to integrate AI into their processes. Each word that spits out of ChatGPT is this same process repeated over and over again many times per second.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

Marc Lamberti

SEPTEMBER 22, 2023

To access XComs, go to the user interface, then Admin and XComs. First thing first, xcom_push is accessible only from a task instance object. With the PythonOperator you can access it by passing the parameter ti to the Python callable function. Once we access the task instance object, we can call xcom_push. Yes, there is!

MySQL 246

MySQL Data Pipeline Database Python 246

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

The Pragmatic Engineer

MARCH 19, 2024

Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research. Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. What are the requirements around governance and auditability of data access that need to be addressed when sharing data?

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Trusted by the teams at Comcast and Doordash, Starburst delivers the adaptability and flexibility a lakehouse ecosystem promises, while providing a single point of access for your data and all your data governance allowing you to discover, transform, govern, and secure all in one place. Support Data Engineering Podcast

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

The Pragmatic Engineer

MAY 25, 2023

Glassdoor could make the process a lot clearer by publishing a moderation log which details when and why it removed a review. Meanwhile, Amazon has announced Bedrock, but more than a month later not even its own developers have access. The company called this restructuring a company-wide transformation. Exclusive A 1.5

Software Engineer 316

Software Engineer Software Engineering AWS Engineering 316

Snowflake

JUNE 5, 2024

With Snowpark’s existing DataFrame API , users have access to a robust framework for lazily evaluated, relational operations on data, closely resembling Spark’s conventions. pandas is the go-to data processing library for millions worldwide, including countless Snowflake users. Why introduce a distributed pandas API?

Python 113

Python Programming Language Government SQL 113

The Pragmatic Engineer

MARCH 13, 2023

In the aftermath, we would learn that Silicon Valley Bank processed $42B in deposit outflows on Thursday: which is nearly a quarter of all deposits they held! The FDIC stated “all insured depositors will have full access to their insured deposits no later than Monday morning, March 13, 2023.”

Banking 192

Banking Insurance Portfolio Media 192

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? What is your process for adding domain semantics to the operational aspect of your LLM?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Rockset

MARCH 1, 2023

The Challenge of Compute Contention At the heart of every real-time application you have this pattern that the data never stops coming in and requires continuous processing, and the queries never stop – whether they come from anomaly detectors that run 24x7 or end-user-facing analytics. So they are not suitable for real-time analytics.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

Start Data Engineering

OCTOBER 14, 2024

Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Use STRUCT for one-to-one & hierarchical relationships 3.2. Use ARRAY[STRUCT] for one-to-many relationships 3.3. Nested data types can be sorted 3.3.3. UNNEST ARRAY to rows and GROUP rows to ARRAY 3.

SQL 130

SQL Data Schemas Data Coding 130

Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Summary Any software system that survives long enough will require some form of migration or evolution. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments.

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Engineering at Meta

AUGUST 27, 2024

For example, we built Policy Zones that apply across our infrastructure to address restrictions on data, such as using it only for allowed purposes, providing strong guarantees for limiting the purposes of its processing. Commonly, purpose limitation can rely on “point checking” controls at the point of data processing.

Programming Language 122

Programming Language Coding Data Warehouse Systems 122

Rockset

JUNE 21, 2024

With this acquisition, what we’ve developed over the years will help make AI accessible to all in a safe and beneficial way. We will gradually transition current customers off Rockset and are committed to ensuring a smooth process. While the nature of these applications has changed, the underlying infrastructure challenges have not.

Database 145

Database Cloud Accessibility Accessible 145

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements.

Government 105

Government Accessibility Accessible Cloud 105

The Pragmatic Engineer

APRIL 6, 2023

Our hope is that making salary ranges more accessible on Comprehensive.io For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing. Our system is using purely Serverless to process the data. ” How does Comprehensive.io

Software Engineer 194

Software Engineer Software Engineering Datasets Database 194

The Pragmatic Engineer

OCTOBER 31, 2023

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones.” We are still working on processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services (such as SQS and EventBridge).

AWS 232

AWS Google Cloud Cloud Engineering 232

Snowflake

AUGUST 5, 2024

BigGeo is a high-efficiency geospatial intelligence platform that makes geospatial data accessible and actionable for organizations by processing data up to 100 times faster than traditional methods. We unlock the value of geospatial data and make it accessible to your entire organization. This is where BigGeo steps in.

Business Intelligence 113

Business Intelligence Technology Accessibility Accessible 113

Snowflake

SEPTEMBER 7, 2023

We are excited to announce the public preview of External Access, which enables customers to reach external endpoints from Snowpark seamlessly and securely. With this announcement, External Access is in public preview on Amazon Web Services (AWS) regions.

Amazon Web Services 122

Amazon Web Services AWS Government Python 122

Engineering at Meta

APRIL 13, 2023

As this is rolled out, security-conscious users who utilize the verify security code page will notice this verification process occurs quickly and automatically. Nobody – not even WhatsApp – has access to those private keys. A list of public keys alone cannot provide access to anyone’s content.

Utilities 145

Utilities Coding Database Accessible 145

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. What are the features and focus of Pieces that might encourage someone to use it over the alternatives?

Building 147

Building Data Lake High Quality Data Machine Learning 147

Snowflake

DECEMBER 20, 2023

Accessing the necessary resources from cloud providers demands careful planning and up to month-long wait times due to the high demand for GPUs. And because Snowpark Container Services is designed with data-intensive processing in mind, developers can effortlessly load and process millions of rows of data.

Scala 122

Scala Government Java Cloud 122

Data Engineering Podcast

JUNE 30, 2024

Petr emphasizes the need for a holistic approach to data reliability, integrating data systems into broader business processes. What does a typical incident/error resolution process look like with Synq? Petr emphasizes the need for a holistic approach to data reliability, integrating data systems into broader business processes.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

MAY 12, 2024

Your host is Tobias Macey and today I want to talk about my experiences managing the QA and release management process of my data platform Interview Introduction As a team, our overall goal is to ensure that the production environment for our data platform is highly stable and reliable. Don't forget to check out our other shows.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Jesse Anderson

MAY 9, 2023

We’re not allowed to see process beyond the Azure boundary, and in some cases it involves transfer by hand of source files from the GitHub private repos into an internal GitLab repo which we’re not allowed to see. But we’ve had to evolve a homegrown process that fits both teams.”

Software Engineer 147

Software Engineer Software Engineering Consulting Data 147

Snowflake

JANUARY 9, 2024

Data cloud integration: This comprehensive solution begins with the Snowflake Data Cloud as a persistent data layer, which makes data more accessible for organizations to get started with the platform. Data ingestion: Hakkoda leads the entire data ingestion process.

Data Ingestion 111

Data Ingestion Cloud Architecture Data Pipeline 111

Snowflake

MAY 7, 2024

Snowflake Cortex is a fully-managed service that enables access to industry-leading large language models (LLMs) is now generally available. license, it provides ungated access to weights and code. Snowflake customers with a payment method on file will be able to access Snowflake Arctic for free until June 3, 2024.

Government 114

Government SQL Data Security Accessibility 114

Snowflake

JUNE 6, 2024

You can now use Snowflake Notebooks to simplify the process of connecting to your data and to amplify your data engineering, analytics and machine learning workflows. Leverage your existing role-based access controls (RBAC) to manage access to notebooks and the underlying data assets to enable consistent and robust data governance.

SQL 99

SQL Python Machine Learning Data Workflow 99

Knowledge Hut

MARCH 28, 2024

The process of developing digital assets that are both aesthetically pleasing and simple to use is known as user interface/user experience design, or UI/UX design. Design teams experiment with turning ideas into concrete forms, from paper to digital, through the prototyping process. What is Influencing the Trends of 2024?

Designing 105

Designing Accessibility Accessible Technology 105

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 96

Metadata Management Data Governance Government 96