Watcher load balancing

[joaosilva15]

Feature request

Hello 👋 not sure if this is already possible but I would like to be able to configure the watcher to do GRPC client-side load balancing. This would make it easier to set up in Kubernetes as we would need to create a headless service and connect the watcher to it, instead of configuring a load balancer between the watcher and the API.

If this feature is already implemented, can you please point me to some docs on how to do it? I created the headless service and made the watcher have the same behaviour as with the normal service. Looking through the code I believe some GRPC configurations are lacking for this to be possible.

Thanks!

[enarha]

@joaosilva15 Interesting if you managed to get proper load balancing (load is evenly balanced between the clients)? I also tried the headless service configuration, but the load was not evenly balanced mixing different types of clients (CLI, proxy service and watcher). As a workaround I set two API instances one serving the Tekton Results Watcher (most of the traffic) and another one serving other clients (CLI clients and proxy) allowing lower latency for the latter.
Having proper gRPC load balancing is something I would like to look into. If this is still an issue for you, I'll try to prioritize accordingly.