AWS would like this for granting projected tokens to our EFS CSI driver so the driver can get a pod's AWS credentials (https://github.com/kubernetes-sigs/aws-efs-csi-driver & aws/efs-utils#60)

added.

@micahhausler can you clarify if efs driver would need this at CreateVolume time, or is NodePublish sufficient?

NodePublish

What if one of these remounts (NodePublish?) fails? What if kubelet is not able to refresh a token after expiration? There is no facility in kubelet to kill the affected pod, do you plan to add it?

(Starting separate "thread")
Does this mean that kubelet must cache "vault tokens" of all running pods and their lifetime, to be able to refresh a token when it expires?

What if one of these remounts (NodePublish?) fails?

return errors? do you have any suggestion?

What if kubelet is not able to refresh a token after expiration?

the token will be refreshed regularly before expiration.

Does this mean that kubelet must cache "vault tokens" of all running pods and their lifetime, to be able to refresh a token when it expires?

the token is already being cached and refreshed before expiration.

Returning errors only produces events to users. A pod keeps running, possibly with expired / stale data, without really knowing about it. Is it something that's acceptable? Or are there cases when a pod that uses a secret must be killed when it looses access to a secret? In that case you should add some note here + code to kill pods.

looking at the code, the pod will not be killed if the first volume mount fails. i expect the remounts will adopt the same behavior. @seh what will happen if the mounted secrets expire? will the secrets be deleted so that the users can be notified?

killing the container if they ever get stale

if the secret can ever get stale, the secrets provider (e.g. vault) should revoke those secrets, rather than the secrets consumers (e.g. k8s). we can not rely on the credential consumers not to use expired secrets. e.g. the pod cache the secret in memory even if the secret is replaced with another fresh one.

Those are fine ideas, but that's not how all the secret engines in Vault work.

Consider an administrator who stores an API key in a Vault K/V secret. Perhaps that administrator usually rotates the key about every other month, leaving a couple of weeks of overlap before revoking the previous key. She might set an advisory TTL on the secret read from Vault for one week, advising consuming applications that they should probably check for a fresh key every week so. An old key might still work for a long while after that week.

going back to the original question on what to do when the remount, i would still like to keep the current behavior where the error will only be logged because:

1. stale secrets might still work and it is not security critical for the consumer of credentials (k8s) to stop using the stale secrets.
2. the repeated remount will overcome transient errors.
3. there is no guarantee that killing the container/pod will solve fatal errors.

A lot of good discussion here. Can we capture all this somewhere in the KEP?

done.

Can you please elaborate on interaction between RequiresRemount and token lifetime? What should happen when RequiresRemount==false and a token expires? Will kubelet keep running a pod that used the expired token? I may be missing some basic knowledge, but why is that OK?

for vault, the token is used to authenticate as the pod's service account so that the driver can fetch the secrets stored in vault. however, the secrets have TTL, so it will authenticate again to fetch the newer secrets. in this case, RequiresRemount==true.

for other csi that doesn't need re authentication, RequiresRemount==false, e.g. cert manager csi driver wants to create CertificateRequest on behalf of the pod the credential of which is only needed in the first mount if the certificate doesn't expire.

Does this mean remounting the volume set up by the driver that consumes the service account token? In my driver that refreshes its service account tokens, I don't remount the volume; I use (a port of) AtomicWriter to replace the files holding the secrets fetched from Vault.

With secrets (or X.509 certificates) acquired from Vault, having authenticated to Vault with a Kubernetes service account token, there are three things that can expire:

• The secret data
  (By way of an optional advisory TTL.)
• The Vault token
  (Usually you can refresh it, but when that offer runs out, you need to authenticate again.)
• The Kubernetes service account token
  (Minimum validity period from the TokenRequest API is ten minutes.)

Even when the secret data changes, though, it's not necessary to unmount and remount volumes. The CSI driver can replace that data on disk "atomically".

If I've misunderstood what is being remounted here, feel free to ignore my critique.

Does this mean remounting the volume set up by the driver that consumes the service account token?

the remount is triggered by kubelet, which means a sequence of NodePublishVolume will be called. this way
can assure the token not going stale and move the common token fetching logic from different csi drivers to kubelet.

I use (a port of) AtomicWriter to replace the files holding the secrets fetched from Vault.

this is good. many existing volumes which requires remount (downwardAPI, projected service account token) are also using AtomicWriter.

it is true that there is extra overhead incurred by the periodic remount operation in this approach, but you can still use AtomicWriter right? what the difference in overhead between the writes via AtomicWriter and mounts for the secrets store csi driver? the difference in overhead seems minimal for other in tree volumes using AtomicWriter.

There's no remounting of volumes necessary, though. The process running in the container may be reading the files that would be unmounted. It introduces a gap during which the files would not be available any longer, and in most cases slightly stale files are better than no files.

For most drivers, unmounting involves recursively deleting files from a directory, and mounting requires creating directories. There may be a mixture of files within such a mounted volume, with differing expiration times. Sweeping it all away and writing it again a short time later is not what these other drivers are doing with AtomicWriter, and it's not how my driver works. AtomicWriter helps facilitate in-place updates of a tree of files. Deleting all the files and writing them again later is not an atomic update.

fair enough, i can add the option.

IMO, term "remount" comes from Kubernetes RequiresRemount handling

It has never remounted anything, it refreshes atomic writer volumes. Refactoring to "Refresh" would be better, but orthogonal to this enhancement.

@jsafrane do you think we need to clarify in the csi spec this new refresh behavior or leave this as a Kubernetes-specific behavior? Although already, the drivers that support this would have to be Kubernetes-specific anyway since they need to make the TokenRequest API call. Right now I don't think it's expected the NodePublishVolume could get called repeatedly.

the csi driver would not make api calls. TokenRequest will be called by kubelet and TokenReview is called by storage provider.

Still, NodePublish will be called frequently. So far, NodePublish was called only once when mounting the volume for the first time. As long as the NodePublish is called in ephemeral CSI volumes (secrets and similar stuff), they're Kubernetes extension anyway and we don't need to update CSI spec.

AWS EFS does not seem to be interested in periodic NodePublish when the token expires, right @micahhausler?

The first sentence implies that the token is available in CreateVolume, while the token is passed in NodePublish (which makes more sense).

will reword it. :)

Some CSI drivers need ControllerPublish and NodeStage calls. Do you plan to add the token to these too? If not,
can we safely assume that these calls won't ever need such a token?

Do you plan to add the token to these too?

at this time it is a no. in the current collected user stories, vault and cert manager only need this token in NodePublishVolume calls to authenticate.

can we safely assume that these calls won't ever need such a token?

i didn't find a good reason to use the pod's token in the controller rpc services and NodeStage/NodeUnstage as they are cluster operations or early stage node operations which not much related to pod.

If it's used in EFS, a real persistent volume that coincidentally uses only NodePublish, I think that there is only tiny step to other volume plugins that may require NodeStage / ControllerPublish.

I think NodeStage/ControllerPublish may add additional challenges because multiple pods can share the same volume at those operations.

I think an explicit warning that tokens are do not cover ControllerPublish/NodeStage in the KEP (and in subsequent docs in github.com/kubernetes-csi/docs) will do for now.

@micahhausler can you clarify if efs driver would need this at CreateVolume time, or is NodePublish sufficient?

What are the valid values for this string?

it is just an arbitrary non empty identifier.

the SP is supposed to send a TokenReview request with that audience included to validate the token.

typo: csi

done

Would it make sense to put the two service account related fields into another struct?

done

A lot of good discussion here. Can we capture all this somewhere in the KEP?

We also need to fill out the test plan, graduation criteria, and production readiness sections before we can move the kep to implementable.

regarding criteria and production readiness, seems they need FeatureGate. i am not sure we need feature gate for this one as it doesn't change existing behavior.

I think any API change we need to make would require going through the alpha/beta/ga feature gates so that we have the ability to make changes after testing and receiving feedback.

done

I think NodeStage/ControllerPublish may add additional challenges because multiple pods can share the same volume at those operations.

I think this should probably only allow a single audience. I'd like @liggitt to weigh in on the API review. I don't think this needs to block though.

the use case to consider is secrets store csi driver which possibly has multiple providers. there are two ways:

1. currently the approach is to distribute one token with multiple audiences to the secrets store csi driver.
2. another way would be distribute multiple tokens (each has single audience).

FYI @seh

My suggestion to allow multiple audiences was for parity with the TokenRequest API, and to allow operators greater flexibility in talking to, say, more than one Vault authentication role, where they each demand a different audience. Doing so would also allow changing the configuration of such a role, and allowing multiple CSI driver daemons to play along through the transition.

both ways can suffice the multi-audience in a csi driver scenario. take the secrets store csi driver as an example, the csi driver only needs to pass the single token with multiple audiences to all the providers, while if adopting the second approach, the csi driver needs to hardcode a mapping between the tokens from kubelet and the providers. the advantage i can think of of the second approach is every provider gets its own token but i am not sure how much security sugar we get here. seems to me that the convenience of the first approach outweighs the advantages that the second approach provides to us.

the csi driver only needs to pass the single token with multiple audiences to all the providers

passing replayable credentials which are considered valid by multiple providers to all the providers does not seem like a good idea

allow operators greater flexibility in talking to, say, more than one Vault authentication role, where they each demand a different audience

I'm not very familiar with this... is this a scenario where you are presenting the token to a single vault server, using multiple audiences as some sort of scoping mechanism?

is this a scenario where you are presenting the token to a single vault server, using multiple audiences as some sort of scoping mechanism?

Yes. A Vault operator can mount the Kubernetes auth method any number of times. Within each mounted auth method, you can create any number of roles. Each such role defines which service account names, within which namespaces, and bound to which audience (and only a single audience, questioned here and here, alas too late) to accept.

In my CSI driver, a pod can read multiple Vault KV secrets and request multiple X.509 certificates, each involving authenticating as a different Vault role. Each of these roles could demand that the principal present a service account token bound to a different audience. If a given service account token is bound to multiple audiences—perhaps the union of what these Vault roles demand—then a single token could suffice for authenticating as each Vault role.

passing replayable credentials which are considered valid by multiple providers to all the providers does not seem like a good idea

agreed. when passing multiple tokens to the csi driver, in the case of secrets store csi driver, it is required to implement a mapping between the tokens and the providers. my current idea is to pass a map in VolumeAttributes:

csi.storage.k8s.io/serviceAccount.tokens: {
  'audience-0': {
    'token': token-0,
    'expiry': expiry-0,
  },
  'audience-1': {
    'token': token-1,
    'expiry': expiry-1,
  },
  ...
}

regarding Vault roles which might have different audiences, the csi driver will pass all the tokens with audience in the format 'vault-*' to Vault provider. of course, it is a non-goal in this KEP but it is something that the csi driver needs to implement to accommodate the Vault case.

Is there ever a usecase for multiple service account tokens? If I'm using the secret driver with both AWS and valut?

done