Skip to main content

Manage Temporal Proxy encryption keys

View Markdown

Temporal Proxy uses short-lived data encryption keys (DEKs) to encrypt Payloads and a key encryption key (KEK) in your KMS to wrap each DEK. The proxy rotates DEKs automatically. You manage the longer-lived KMS keys and must keep them available for as long as encrypted Payloads refer to them.

Before managing keys, configure payload encryption and grant the proxy access to the selected AWS KMS, Azure Key Vault, Google Cloud KMS, or extension-server key.

Distinguish DEK rotation from KMS key rotation

Each default or Namespace-specific overrides policy has the following lifecycle fields:

FieldPurpose
uriActive KMS key used to wrap new DEKs. The key URI is recorded with every encrypted Payload.
decryptURIsEarlier or staged KMS keys that the proxy can use only to unwrap existing DEKs.
durationTime for which the proxy uses a DEK to encrypt new Payloads.
renewBeforeLead time before duration expires when the proxy prepares a replacement DEK.

The proxy rotates a DEK when it reaches its renewal threshold. A DEK rotation does not rotate or create a KMS key. The wrapped DEK and KMS key URI stored with each Payload let the proxy decrypt older Payloads after later DEK rotations.

Use the encryption metrics to verify this lifecycle:

  • dek_rotations_total{reason="scheduled"} counts DEKs prepared by the background refresh.
  • dek_rotations_total{reason="on_demand"} counts DEKs replaced on a request because a fresh DEK was not ready.
  • kek_ops_total and kek_ops_duration_secs report the KMS operations that wrap and unwrap DEKs.

A sustained increase in on_demand rotations indicates that refresh is falling behind. Increase renewBefore, while keeping it less than duration, and check KMS latency and errors. See the metrics reference for the complete metric list.

Change the active KMS key

Treat a key change as a compatibility rollout. The key URI is part of the encrypted Payload metadata, so every proxy instance that might read a Payload must recognize the URI that wrote it. The proxy reads its configuration at startup; restart each instance after every configuration change.

For a rolling deployment with more than one proxy instance, use two stages.

First, register the new key for decryption while the old key remains active. This makes every updated instance recognize both key URIs before any instance starts writing with the new key:

encryption:
enabled: true
default:
uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
decryptURIs:
- gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2
duration: 1h
renewBefore: 15m

Wait for every instance to run this configuration. Then promote the new key and retain the old key for decryption:

encryption:
enabled: true
default:
uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2
decryptURIs:
- gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
duration: 1h
renewBefore: 15m

Keep any keys from earlier migrations in decryptURIs during both stages. Repeat the process for every policy under overrides; those map local, pre-translation Namespace names to key policies.

After the second rollout, start a Workflow through each proxy instance and read both newly written and older Workflow Histories through each instance. Watch vault_ops_total{result="error"} and kek_ops_total{result="error"} during the test. Do not remove the old key because the rollout succeeds; retain it until no encrypted Payload refers to it.

Stop encrypting new Payloads

To stop encrypting new outbound Payloads while keeping existing encrypted data readable, set enabled: false and keep the key policies in the configuration:

encryption:
enabled: false
default:
uri: gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v2
decryptURIs:
- gcpkms://projects/my-project/locations/global/keyRings/codec/cryptoKeys/v1
duration: 1h
renewBefore: 15m

With this configuration, the proxy stops encrypting new outbound Payloads but continues decrypting Payloads marked with its encryption metadata. Removing default removes the vault and prevents that decryption path from being created.

This change does not rewrite data already stored by the upstream Temporal Service. Existing encrypted Payloads remain encrypted at rest and require their original KMS keys whenever they are returned through the proxy.

Plan key access across regions

Temporal Proxy does not replicate KMS keys or configuration. Every proxy instance that can receive traffic must be able to unwrap every DEK that another instance can produce. This requirement applies during normal load balancing, regional failover, rolling key changes, and rollbacks.

Use these checks when the proxy runs in more than one region:

  1. Give every instance the same set of active and decrypt-only key URIs for the traffic it can serve.
  2. Grant each instance's runtime identity decrypt access to every configured key URI.
  3. Choose a KMS topology that stays reachable during the region failures your deployment is designed to tolerate.
  4. Test reading data written in every region after removing network access to one region's KMS endpoint.

The configured URI is the key identifier stored with an encrypted Payload. If two URIs address equivalent backing key material, the proxy still treats them as different identifiers. Preserve the original URI in decryptURIs unless all readers use the same stable URI.

Provider choices differ:

  • AWS KMS: Use an AWS KMS multi-Region key or another design that lets every active region decrypt DEKs written in every other region. Keep the configured URI stable across instances when possible, and test that the URI resolves to usable key material after failover. See AWS KMS multi-Region keys.
  • Azure Key Vault: Place and replicate vaults according to your regional recovery design. Register every URI that can appear in stored Payloads and grant each proxy identity access to all of them. See Azure Key Vault availability and redundancy.
  • Google Cloud KMS: Use a global or multi-region key location when the same key must be reachable from several regions, or ensure every instance can reach the configured regional location. See Cloud KMS locations.
  • Custom KMS extension: Keep the extension://<server>/<key> URI stable and make the extension service as available as the proxy. Its Decrypt request contains the wrapped DEK but no Namespace, so the ciphertext returned by Encrypt must carry enough information for the extension to select the correct key and version.

Retire an old KMS key

Retention is not a sufficient retirement signal by itself. Open Workflow Executions can keep Event Histories beyond the Namespace Retention Period, and archived or exported histories can outlive the source Namespace. A key is still needed if any system from which data can be read contains a Payload that records its URI.

Before removing a key from decryptURIs:

  1. Confirm that no active policy uses the key as uri.
  2. Inventory open and closed Workflow Executions, Archival, exports, backups, and copied histories that might contain Payloads encrypted with the key.
  3. Exercise representative reads, replays, resets, and operational tooling through a proxy configured with the remaining keys.
  4. Remove the old URI from a canary instance and verify that it can read the same data before rolling out the removal.
  5. Revoke access before scheduling deletion. Revocation is reversible if a missed Payload still needs the key.

Keep KMS audit logs and the proxy's decrypt metrics during the retirement window. A decrypt failure after removing a URI means that the original key must be restored to decryptURIs and made accessible again.

Deleting a key is irreversible

Deleting a KMS key permanently makes every Payload encrypted under it unreadable. Neither Temporal nor the KMS provider can recover the plaintext. Keep the key in a recoverable, disabled state until your data inventory and recovery policy prove that it is no longer required.