We are using transit engine in Vault (HCP) to encrypt data on Kafka and wondering if someone had some experience with the setup in an organisation.
We have multiple teams that read topics from each other. We want them to be able to read each others encrypted data but with enough granularity that only the necessary teams can read the data. All the services doing encrypt/decrypt is running in EKS and we are using Kubernetes Auth.
The goal is:
That teams can read each other data only when needed.
That we (infrastructure team) do not need to be involved in how they choose to encrypt/decrypt data in between them.
We will supply proper guidelines to get started.
Our first idea was to create one namespace per team. Then they could administer themselves who has access and who does not. This might not be granular enough so we are looking into a few use cases. One thing that has come up in our planning is that the data often live longer than the teams themselves.
Team A encrypts some sensitive data.
Team B needs to decrypt Team A’s data.
Team A gets disbanded and no longer exist.
Team C takes over the ownership of Team A’s data and need to encrypt/decrypt it.
Team B still needs to read Team A (now C’s) data.
Team D does not need to read this data and will not get access to decrypt it.
Do we then need to encrypt the data based on topic name and not team names?
Is there anyway in this scenario where we do not need to be involved in the process here?
Should we create encrypt/decrypt policies per topic and manage access + bindings outside of team namespaces?
three policies per key, encrypt/decrypt and admin. policy names should be templated so it is easy to understand. admin right should be assigned to team, encrypt&decrypt to team’s app, decrypt can be assigned to external teams on-demand.
automate the process of requesting access to those policies. eg, using terraform code to config vault.
So to answer your questions:
Do we then need to encrypt the data based on topic name and not team names?
Yes.
Is there anyway in this scenario where we do not need to be involved in the process here?
automate the request and approval process, eg, terraform.
Should we create encrypt/decrypt policies per topic and manage access + bindings outside of team namespaces?
no need to use namespaces. it is easier to centrally control access just by policies.
In my experience, it’s always a bad idea to name anything after a business team, as teams get renamed, split and merged as a business evolves, alarmingly often. Rather, stick to naming things after more immutable concepts like your topic names.
Namespaces are useful to you if what you want to do, is grant a subset of people delegated access to write Vault policies as they see fit for an entire subtree of Vault paths. However, if you want to exercise more control, such as ensuring teams follow defined patterns in what is allowed, namespaces become questionably useful.
Unfortunately, beyond the option of giving delegated admins access to a namespace and letting them do anything in it, Vault does not have any out of the box good solutions to letting people self-manage permissions. I expect you will need to build your own automation to handle that.
Terraform doesn’t give you anything for the request/approval process, though - only the part of applying config to the Vault API.
I would probably avoid using it for this. The API for Vault policies is a very basic CRUD REST one, so not hard to automate directly - Terraform would be an unnecessary complication for this use case, IMO.