Object storage

Upload documents live in the selected database; their bytes live in a storage.Backend. Configure the backend once for the application. Ridu coordinates writes with document transactions and checks references again before cleanup.

Set a stable application namespace

Every application with an upload collection must declare Config.StorageNamespace:

func Config() ridu.Config {
  return ridu.Config{
    Name:             "Acme Editorial",
    StorageNamespace: "acme-production",
    Collections:      []ridu.Collection{Media},
  }
}

The namespace is an object prefix, not a display label. Keep it unchanged when Config.Name or an upload collection slug changes, and never reuse it for another application that shares the backend. It must contain 3–64 lowercase letters, digits, underscores, or hyphens.

New objects use a slug-independent ridu/<namespace>/objects/<nonce>/... layout, so renaming a collection does not move bytes.

Create the backend lazily through WithUploadStorage so manifest resolution and generation never touch the filesystem or network.

Implement a backend

type Backend interface {
  Put(context.Context, string, io.Reader, int64, string) error
  Open(context.Context, string) (io.ReadCloser, Object, error)
  Delete(context.Context, string) error
  List(context.Context, ListRequest) (ListPage, error)
}

type Object struct {
  Key         string
  Size        int64
  ContentType string
  ModifiedAt  time.Time
}

type ListRequest struct {
  Prefix string
  Cursor string
  Limit  int
}

type ListPage struct {
  Objects    []Object
  NextCursor string
}

The small interface has strict semantics:

ModifiedAt is a safety input, not optional decoration. Reconciliation uses it to enforce the grace period and fails closed if an adapter cannot prove object age. Ridu still validates namespaced keys, application namespace, committed references, and fresh object-lock state immediately before a destructive call.

Production backends should also implement:

type HealthBackend interface {
  Ping(context.Context) error
}

Ping must be bounded, repeatable, and safe without reading or mutating an application object. ridu.Execute includes it in startup preflight and /readyz. The optional storage.URLSigner provides short-lived direct downloads, but private framework delivery remains access checked.

Local filesystem

The local backend suits development and a single node with a durable mounted volume:

func openUploadStorage(context.Context) (storage.Backend, error) {
  return localstorage.New("/var/lib/acme/uploads")
}

Register it with the server:

ridu.WithUploadStorage(openUploadStorage)

Writes use a temporary file, sync it, atomically rename it into place, and sync the directory. Object paths reject absolute paths and traversal. The readiness probe verifies the root is a directory, performs a durable private probe write, removes it, and syncs cleanup.

A local volume must be durable across replacement and mounted at the same path before readiness. It is not shared automatically across replicas: use a shared object service for a multi-node deployment rather than putting independent local disks behind one application.

S3-compatible storage

The dependency-free S3 backend implements SigV4 PUT, GET, DELETE, complete paginated listing, bucket health, and signed GET URLs:

func openUploadStorage(context.Context) (storage.Backend, error) {
  return s3storage.New(s3storage.Config{
    Endpoint:       os.Getenv("S3_ENDPOINT"),
    Region:         os.Getenv("S3_REGION"),
    Bucket:         os.Getenv("S3_BUCKET"),
    AccessKey:      os.Getenv("S3_ACCESS_KEY"),
    SecretKey:      os.Getenv("S3_SECRET_KEY"),
    MaxSpoolBytes:  256 << 20, // 256 × 2²⁰ = 268,435,456 bytes (256 MiB)
    SpoolDirectory: "/var/tmp/ridu-spool",
  })
}

Endpoint, region, bucket, access key, and secret key are required. Supply a custom http.Client to set deployment-specific transport and timeout policy; the default client timeout is 30 seconds.

S3 signing needs the payload hash before upload. A seekable input is hashed and rewound. A non-seekable input is copied to a private temporary file, bounded by MaxSpoolBytes; zero selects 256 MiB. Size mismatch, cancellation, or signing failure removes the spool file. Plan local scratch space for concurrent non-seekable uploads.

SignedURL accepts a lifetime from one second through seven days. Signed possession is a temporary capability, not collection authorization; issue it only after the application has made the relevant access decision. Ordinary Ridu delivery remains private and no-store because collection read rules may depend on the actor or row even when an upload collection is not marked private.

The current backend does not implement multipart, resumable, or direct browser upload. Large files still pass through Ridu’s bounded upload operation and, for non-seekable S3 bodies, the spool.

Reconciliation and deletion

Database metadata and objects cannot share one atomic commit. Ridu narrows that boundary with staged writes, rollback, cross-process object locks, last-reference checks, and reconciliation.

App.ReconcileUploads reports candidates from an ACL-independent snapshot of current, trashed, and versioned references. The grace period must be at least five minutes. After review, App.CleanupUploads re-checks candidates in bounded batches under fresh snapshots and object locks before deletion. A candidate that gained a reference is retained.

Keep the grace period longer than the slowest admitted upload or import and quiesce imports during a destructive cleanup pass. Failed after-commit cleanup retains bytes as reconciliation candidates; it does not roll back a durable document deletion.

See Uploads for collection metadata, image variants, remote ingestion, private delivery, and the 65-key per-document object bound.

Build a safe custom adapter

Before using another provider, test these behaviours directly:

• exact declared-size enforcement, cancellation, short reads, extra bytes, and partial-write cleanup;
• atomic same-key replacement and idempotent missing-key deletion;
• path and key traversal rejection;
• Open metadata, body close behaviour, and storage.ErrNotFound mapping;
• prefix-restricted, fully paginated List results with non-zero modification times;
• bounded repeatable Ping, plus credential and dependency failure;
• concurrent Put, Open, Delete, reconciliation, and retry races; and
• no logging of credentials, private keys, signed URLs, or sensitive provider responses.

Custom storage used with ridu.Execute needs HealthBackend unless you configure external dependency checks and opt into unverifiable readiness. Prefer implementing HealthBackend so Ridu can report dependency failures.

Back up both halves

Back up the selected database’s metadata, versions, and reference indexes together with the object namespace. Restore both to the same recovery point and test document reads, private delivery, representative checksums, and reconciliation. A database-only restore can reference missing bytes; an object-only restore can retain stale data.

Read Production for deployment and restore drills and Security for trust and capability boundaries.