Cloud Storage Document Loader

The Cloud Storage loader implements the loader.DocumentLoader interface for loading files from cloud object storage services. It detects the cloud provider automatically by URL prefix (s3://, gs://, az://) and fetches the object with a direct HTTPS GET. S3 requests are SigV4-signed; GCS and Azure requests carry a bearer token.

Choose Cloud Storage when your documents are stored in AWS S3, Google Cloud Storage, or Azure Blob Storage. The loader auto-detects the cloud provider from the URL prefix, so a single loader instance handles all three. For loading from SaaS knowledge bases, consider Confluence, Notion, or Google Drive.

Installation

go get github.com/lookatitude/beluga-ai/v2/rag/loader/providers/cloudstorage

Quick Start

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/lookatitude/beluga-ai/v2/config"
    "github.com/lookatitude/beluga-ai/v2/rag/loader"
    _ "github.com/lookatitude/beluga-ai/v2/rag/loader/providers/cloudstorage"
)

func main() {
    l, err := loader.New("cloudstorage", config.ProviderConfig{
        APIKey: os.Getenv("AWS_ACCESS_KEY_ID"),
        Options: map[string]any{
            "secret_key": os.Getenv("AWS_SECRET_ACCESS_KEY"),
            "region":     "us-east-1",
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    docs, err := l.Load(context.Background(), "s3://my-bucket/documents/report.txt")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Content: %s\n", docs[0].Content)
}

Configuration

Parameter	Type	Default	Description
APIKey	string	""	S3 access key ID, or an OAuth 2.0 (GCS) / Microsoft Entra ID (Azure) bearer token. Without it, requests are anonymous (public objects).
Timeout	time.Duration	60s	HTTP request timeout
Options["secret_key"]	string	""	S3 secret access key (required with APIKey for S3)
Options["session_token"]	string	""	S3 session token for temporary credentials (optional)
Options["region"]	string	us-east-1	AWS region for S3 bucket URLs and SigV4 signing
Options["max_bytes"]	number	32 MiB	Maximum object size in bytes

Authentication

• S3 — requests are signed with AWS SigV4: APIKey is the access key ID, Options["secret_key"] the secret access key, and Options["session_token"] an optional session token.
• GCS — APIKey is sent as an OAuth 2.0 bearer token.
• Azure — APIKey is sent as a Microsoft Entra ID bearer token (with x-ms-version).

Bucket and account names are validated against each provider’s naming rules before they become part of the request host, the object key is path-escaped, and requests are only sent to (and redirected within) the providers’ own hosts, so a crafted source cannot redirect the request or its credentials elsewhere.

Supported Providers

The loader determines the cloud provider from the URL prefix:

Prefix	Provider	URL Format
s3://	AWS S3	s3://bucket/key
gs://	Google Cloud Storage	gs://bucket/object
az://	Azure Blob Storage	az://account/container/blob

Provider-Specific Examples

AWS S3

docs, err := l.Load(ctx, "s3://my-bucket/data/document.pdf")

Google Cloud Storage

l, err := loader.New("cloudstorage", config.ProviderConfig{
    APIKey: os.Getenv("GCS_ACCESS_TOKEN"),
})
docs, err := l.Load(ctx, "gs://my-bucket/reports/summary.txt")

Azure Blob Storage

l, err := loader.New("cloudstorage", config.ProviderConfig{
    APIKey: os.Getenv("AZURE_STORAGE_TOKEN"),
})
docs, err := l.Load(ctx, "az://myaccount/my-container/files/data.csv")

Document Metadata

Each loaded document includes the following metadata fields:

Field	Type	Description
source	string	Original cloud storage URI
loader	string	Always "cloudstorage"
provider	string	Cloud provider: "s3", "gcs", or "azure"
bucket	string	Bucket or container name
key	string	Object key or blob path
filename	string	Extracted filename from the key

Error Handling

The loader returns descriptive errors for invalid URLs and failed requests:

docs, err := l.Load(ctx, "s3://my-bucket/path/to/file.txt")
if err != nil {
    // Possible errors:
    // - "cloudstorage: invalid S3 URL ..." (malformed URL)
    // - "cloudstorage: fetch ... failed (status 403): ..." (auth failure)
    // - "cloudstorage: fetch ...: context deadline exceeded" (timeout)
    log.Fatal(err)
}