This could lead to security risks or complicated setups.

Can you elaborate on why exposing the Konnnectivity Proxy Server to the cluster network solve the "security risks or complicated setups"?

Regarding the "security risks": I think that exposing Konnectivity Proxy Server would add an additional layer of security, given that the channels with proxy agents are secured with mTLS or Token authentication.
The KAS is not directly exposed to the internet but only accessible from "secured" networks. This would protect it, for instance, from KAS misconfigurations or vulnerabilities that could expose sensitive information
and/or access to unauthenticated users.

IMHO on a higher level, if we believe that exposing the Kubelet to the internet brings security risks, the same should hold for the KAS.

Regarding the "complicated setups": You are right we did not elaborate enough on this point, and "complicated" is not the appropriate term to be used here. We will amend the proposal.
The point we want to make is that right now there is no standard solution for this kind of setup. It is possible to rely on VPNs for example to achieve a similar goal, but this requires specific implementations. What we propose here is to build on top of what we have, and having a consistent approach for master to node and node to master communications.

How does the agent authenticate the pods or the kubelet?

It doesn't, it acts as a TCP forwarder without terminating TLS.

Has the proxy agent finished the TCP handshake with the client before step 1?

Yes, this is what we had in mind. I can make this explicit if you think it's necessary.

I'm not sure it would be easy to send the DIAL_REQ at SYN or SYN/ACK phase. How could we do it?
As far as I know TCPListener.Accept method returns the connection once the TCP handshake is over.

BTW do you foresee any advantage in sending the DIAL_REQ before the TCP handshake is over?

I'm curious what the "something else" is. Can you give some examples?

(If we want to support proxying to things other than the apiserver, we should remove the "apiserver" from the repository name :)

Also how does the proxy server authenticate with the "something else"?

We don't have any use-case in mind at the moment. We could limit the scope to the KAS only.

I'm not sure if running as a static pod will solve the kubelet bootstrapping issue. Kubelet needs to access to KAS to watch for secrets/configmaps, and to register the node. I don't know if kubelet can bootstrap in this order:

1. run the proxy agent as a static pod, waiting for it to establish tunnels to the KAS,
2. start watching for secrets/configmaps/pods and register the node.

Also I'm not sure if other node components like node-problem-detector are ok with this bootstrap order.

I think @cheftako knows this better.

Definitely the proxy agent is required to be up and running in order for the kubelet to perform its bootstrap sequence. I was expecting the static pods to be running without the kubelet being registered, but I did not test it and I agree we should double check if this is feasible.

This is talking about the authentication between the proxy agent and the proxy server, right? Can you point this out in the KEP?

Yes you are right, we'll make it clear.

Copy paste error I'm guessing? :P

Yep, thx for pointing out.

"Cluster network to the Control Plane network"

We initially used that terminology, but we changed it later to be consistent with the definitions below. Of course we are open to change if you think it makes it more clear.

Would the serving port used in updating the default kubernetes Service be updated as well? And if so that would imply kube-apiserver and Konnectivity Agent listen on the same ports?

Yes, the Agent should listen on the secure port used by the KAS. I think that we should make this more clear. Thx for the hint.

Do you mean the cluster-ip service (something like 10.96.0.1)? does that mean kube-proxy has dependency on the konnectivity? It looks like it's not, but want to make sure it's accurate.

Yes, that is correct.
Kube-proxy has no dependency on the Konnectivity agent. As we mentioned below, when configuring the KAS it is necessary to make sure that the IP used by the Agent is the same as the one advertised by the KAS.

It's a little bit confusing. It's still a proxy that forwards the traffic to KAS, right?

Yes, you can see this as an equivalent to ssh remote port forwarding. The client is not aware of interacting with a proxy but from his standpoint, he is sending the request to his final destination.

I guess this has big performance impact. Because the other end is always KAS:6443, It seems a reverse proxy on konnectivity server side could do the same job without agent.

Indeed, we have the same performance penalty that we are paying with traffic going from the Konnectivity server to the nodes.
When using the reverse proxy, we don't have the additional layer of security provided by the authentication between the Konnectivity Agent and the Konnectivity Server but also we lose the ability to use SNI for load balancing.

I guess a whitelist is necessary this sort of proxy in reserve order. master network has access everything in node network, but I am not sure the reverse is true.

+1 to having an explicit allow list on the Konnectivity Server which controls where it will allow traffic to be sent on the control plane.

@anfernee @cheftako
Indeed, that is correct and we have taken this into account in the Risks and Mitigations / Allow list section.

Since the control plane and cluster networks are disjoint, can you elaborate on how to agent -> server tunnel is established? Since the agent shares the same network as other pods on the cluster network, can other pods (eg: kubelet) not directly tunnel to the konnectivity server as well?

It is established by exposing the Konnectivity server. The only requirement is that the Agent must be able to route traffic to the Konnectivity Server (equivalent to what is required today).

Regarding the second question, as we don't have the hands-on on the clients reaching the KAS (apart from the kubelet), we cannot force them to establish tunnels with the Konnectivity Server.

Are pods only able to communicate to the agent on the same machine?

Yes, because it will bind to a local scoped interface.

We thought about a possible shortcoming with this approach. The idea is to redirect the KAS traffic generated from PODs on the node network to Konnectivity agents, by setting the IP used by the agents with advertise-address flag of the KAS. This IP will be set in the Endpoints of the kubernetes service in the default namespace, which is used by k8s clients.

On the other hand, this could be limiting if PODs deployed on the master network rely on the kubernetes default service as well (e.g. CNI pods). As we won't have agents deployed on master nodes, this IP will be unreachable. To circumvent this, we could use the IP address of a load-balancer targeting the KAS instances in an HA setup.

Alternatively, we could consider another approach. The agent could use iptables and "dnat" the traffic going to the Kubernetes default service to the local agent. In this case, the configuration will be probably easier, but the reason we initially discarded this option is that the implementation is more complex as we would need to take care of possible conflicts with other rules (e.g. kube-proxy in iptables mode).

@cheftako @anfernee @timoreimann any thoughts about this?

Not sure if I understand the load-balancer scenario correctly.
If you set the KAS LB IP via advertise-address to set it as endpoint for the kubernetes svc, you wouldn't need the agents in the first place, would you?

On the LB side there's another potential downside for providers that try to host multiple KAS instances from different clusters behind one LB. There's no way to differentiate to which cluster the traffic belongs. SNI information will yield kubernetes for all. The SNI info from the agent connections makes that possible.

Just using the load-balancer approach on the master could also lead to a problem. IIRC on some cloud providers services behind the LB cannot directly talk to the external LB IP. You'd need to "dnat" this again on the master node, which we could do anyway (also in other scenarios) on the master node.

Personally I'd be fine to not be able to use the kubernetes svc on the master and be forced to configure the services with the k8s endpoint.

With regards to network ranges that we could use: Shouldn't we also be able to make use of the "carrier grade NAT" range (100.64.0.0 - 100.127.255.255). I'm assuming this has less chance of conflicting with any existing cluster ranges.

Hi @gottwald, thanks for reacting!

If you set the KAS LB IP via advertise-address to set it as endpoint for the kubernetes svc, you wouldn't need the agents in the first place, would you?

I think I did not express clearly enough my idea. What I meant is that we could use a private LB with an IP that is not routable from the node network, and will be used from within the master network only, to allow the PODs to reach the KAS via the default kubernetes service. On the node network, things would be unchanged.

Personally I'd be fine to not be able to use the kubernetes svc on the master and be forced to configure the services with the k8s endpoint.

If no one has any objection, I would be fine to start with this limitation too.

With regards to network ranges that we could use: Shouldn't we also be able to make use of the "carrier grade NAT" range (100.64.0.0 - 100.127.255.255). I'm assuming this has less chance of conflicting with any existing cluster ranges.

Thx for the hint. I did not know about this range. Using link-local range would be probably the safest/cleanest solution, but as we cannot because of endpoints limitations I have nothing against mentioning this range too, @youssefazrak Any thoughts about this?

Nothing against it. And actually, that's a good idea.
The range is private to the cluster and supposedly a host reaches from a CGN network, there will be no routing conflict as it is NATed to another range.

Seems like the metadata has been moved to the kep.yaml, would be nice to get rid of these big comment blocks.

Please 😸

Can we elaborate here? In many environments this already works, so can we attempt a description of when you would need this solution?

"The goal of this proposal is to provide a mechanism which allows traffic to flow from the Node Network to the Master Network, when those networks are otherwise isolated and there is a desire not to expose the Kubernetes API Server publicly"?

yes indeed, that's more explicit

Is it just Nodes or is it any KAS client running in the Node Network? (So operators and the like)

it's any KAS client running on node network

For now can we restrict it to just the KAS? (i.e. non-goal to talk to anything other than the KAS on the master network)

Anything capable of sending to that port will be able to send traffic to the KAS. We may want to think about options like listening on localhost or firewalling of that port for non node network traffic.

Yeah, we plan to use a host scope address, so that it will possible to use it from within the host itself only.

I'm fine with port 6443 being the default but I would suggest that be configurable.

In a HA setup there could be multiple servers that the agent is connected to. Do we have a reason to care where the traffic goes? (Eg. matching failure zone?) Or do we just pick a random server?

it is actually, I put 6443 just as an example here as it is the default used by KAS (if no secure_port flag is specified)

In a HA setup there could be multiple servers that the agent is connected to. Do we have a reason to care where the traffic goes? (Eg. matching failure zone?) Or do we just pick a random server?

@cheftako I would say that we pick a random server, but if we can think about evolving this later in case of need.

Can we explicitly specify that if this flag is absent the server will no allow any node network initiated requests (traffic) to be placed on the master network.

Hi there, 1.21 Enhancements Lead here.

Please make sure to change the status to implementable to meet one of the requirements for all KEP tracking for the release.

Hi @annajung, it's done.

Seems like you'd need to update the workers to remove the konnectivity agent that's configured locally.

indeed @deads2k the agents should be removed. I'll add this step

This isn't required for alpha, but if/when you move to beta, I'd like to see metrics from the konnectivity agent.