I am planning to use the transit secret engine for encryption/decryption. Also it allows the user to create key using API on demand (but this is not very often).
1:) Is it possible that I build a cluster with all the nodes as active nodes (not 1 active and others standby)? I am thinking since transit engine is kind of stateless mostly, any suggestions (such as doc about this setup)?
2:) If that is not possible, can I build two clusters which share the same backend storage?
3:) Or I see the vault enterprise supports multi-server HA mode which allows all HA nodes to service read-only requests. In my situation, for transit engine traffic of encryption/decryption, is it considered as read-only request?
Regarding 1): With the OSS version, you can only have active/stand by nodes. The enterprise version allows you to have an active/performance standby nodes. The performance standby nodes service read-only requests.
Regarding 2): I do not think that two separate clusters can use the same storage without a method of locking and taking turns when writing to that storage.
Regarding 3): It is, having more performance stand by nodes in your cluster will help you scale horizontally the load of the transit engine.
For 3), Can you please help to confirm that adding more standby nodes will scale horizontally the Enterprise edition transit engine encry/decry POST requests, e.g, the standby node can handle the requests directly (no need to forward to master)?
Yes, the performance stand by nodes can handle transit engine requests (not sure if all requests) on their own. Here is quote from the docs :
NOTE: As of version 0.11 , standby nodes can handle most read-only requests and behave as read-replica nodes. This Performance Standby Nodes feature is a part of Vault Enterprise . This is particularly useful for processing high volume Encryption as a Service (Transit secrets engine) requests. Read Performance Standby Nodes documentation and a Performance Standby tutorial for more details.