Unreal DDC Cache and Immutable Artifacts

Derived Data Cache and build artifacts solve different problems. A cache stores recreatable work; an artifact is immutable output that a later stage must identify, verify, and distribute. A cache may be evicted or overwritten, while an artifact may be needed after its worker is gone.

Separate performance state from delivery evidence#

Unreal Engine Derived Data Cache holds results regenerated from declared source and build inputs. Its value is reduced work and locality, not proof that a release candidate exists. Entries may disappear, become unsuitable after an environment change, or be bypassed; a correct build remains correct.

An immutable artifact can include packaged outputs, symbols, package configuration, manifests, and later-stage deliverables. Publish it under an addressable identity, protect it from replacement, and attach checksums. Consumers should request its manifest, not browse a cache path or workspace.

Give each store a distinct contract#

The cache contract defines read and write access, reuse eligibility, invalidation, reclamation, and outage behavior. The artifact contract defines publication authority, immutable naming, integrity checks, retention, access control, and the required manifest.

Keep these contracts visible in build definitions and operational documentation. A shared storage service can host both only if its paths, permissions, lifecycle rules, and monitoring make the distinction impossible to confuse.

Design cache keys around real inputs#

A cache key must vary whenever an input could change derived output. Relevant inputs include engine revision and build identity, plugins and source content, target, configuration, cook and rendering settings, compiler behavior, and cache namespace. Avoid bypassing engine keying with an overly broad shared location or manually reused directory.

Give incompatible environments separate namespaces or other explicit separation. An engine change, plugin update, target-specific setting, or toolchain change should not silently reuse data from a prior context. Conversely, avoid clearing all cache state for a change that can be isolated by a precise namespace or invalidation rule. The goal is correctness first and reuse only where the inputs truly match.

Make invalidation an owned operation#

Document changes requiring invalidation, who can initiate it, how workers learn about it, and how it is recorded. If clearing cache fixes a build, investigate the undeclared input or key boundary that admitted stale data.

Test partial and complete cache loss. An outage should cause a slower path, visible fallback, or clear policy-driven failure, never an artifact with unknown provenance.

Plan retention and locality separately#

Cache retention balances storage against likely reuse and may evict data without release consequences. Artifact retention is a delivery and audit decision: preserve outputs and manifests for downstream validation, promotion, rollback, or support.

Locality matters for cache behavior. Place reusable cache data where the workers that need it can retrieve it without making a single workstation or ephemeral disk the source of truth. Use local cache layers as accelerators, backed by a shared cache boundary when appropriate. Do not turn a local cache into a hidden artifact repository.

Preserve provenance with manifests#

An artifact manifest should connect output to source and build-definition revisions, Unreal Engine identity, toolchain and SDK context, target, configuration, cache namespace, produced files, and checksums. Add runtime compatibility and validation evidence where applicable.

It need not preserve cache entries; it must prove declared inputs and output identity well enough to reproduce or diagnose the candidate. Store it beside the immutable artifact and make promotion consume it directly.

Recognize common failure modes#

A stale DDC entry can surface derived content from a mismatched environment. A missing cache can expose a script dependent on local preparation. An overwritten artifact can promote an untested output. A manifest naming only a branch or latest label cannot establish provenance. A shared workspace can make these failures intermittent.

Classify each failure by boundary. Fix keying or invalidation for cache reuse problems, worker provisioning for cold-start problems, immutable publication for artifact problems, and manifest contents for traceability problems. Avoid using cache cleanup as a permanent substitute for diagnosis.

Implement in dependency order#

• Inventory current cache paths, workspace outputs, artifact locations, and consumers.
• Define separate cache and artifact contracts with named owners.
• Make source, engine, target, configuration, and toolchain identity available to cache policy.
• Publish immutable artifacts and manifests before changing cache topology.
• Exercise a clean build, cache loss, targeted invalidation, and artifact recovery.
• Add locality and retention improvements only after the correctness boundaries hold.

Practical checklist#

• DDC and other caches are identified as recreatable performance state.
• Release outputs are published as immutable artifacts outside worker storage.
• Cache key scope reflects the inputs that can change derived output.
• Incompatible engine, plugin, target, and toolchain contexts are separated.
• Cache invalidation has an owner, record, and tested result.
• Cold-cache and unavailable-cache behavior is known.
• Cache retention does not determine artifact retention.
• Local cache layers are not used as the delivery source of truth.
• Artifact manifests record inputs, files, checksums, and validation evidence.
• Failure diagnosis distinguishes cache, workspace, artifact, and provenance boundaries.