Use coredns on OAS cluster node
While working on #485 (closed) I noticed a strange phenomena. For querying prometheus from the host using a ClusterIp, I need to find out the IP of the prometheus service.
root@varac-test:~# kubectl -n oas get service monitoring-prometheus-oper-prometheus NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-prometheus-oper-prometheus ClusterIP 10.43.38.214 <none> 9090/TCP 17d
curl 10.43.38.214:9090/alerts would show me the alerts.
A simpler apporach would be to query coreDns for the prometheus IP. It works when querying coreDns directly (which is always at
10.43.0.10 on our clusters) with i.e.
root@varac-test:~# host monitoring-prometheus-oper-prometheus.oas.svc.cluster.local 10.43.0.10 Using domain server: Name: 10.43.0.10 Address: 10.43.0.10#53 Aliases: monitoring-prometheus-oper-prometheus.oas.svc.cluster.local has address 10.43.38.214
So far so good.
However, in order to use
curl monitoring-prometheus-oper-prometheus.oas.svc.cluster.local:9090 directly, we would need to add
10.43.0.10 as nameserver in
And this is where things become strange.