Prometheus监控——云原生与非云原生监控

目录一.云原生应用监控1.1云原生组件 etcd 集群监控1.1.1查看etcd证书1.1.2创建etcd service1.1.3创建etcd证书1.1.4创建ServiceMonitor1.1.5查看监控数据1.1.6grafana监控界面展示1.2云原生 kube-proxy 监控1.2.1查看端口1.2.2修改监听地址1.2.3创建kube-proxy Service1.2.4创建 kube-proxy servicemonitor 资源1.2.5查看监控数据1.2.6grafana监控界面展示1.3云原生 calico 网络插件监控1.3.1开启 Metrics 端口1.3.2创建podMonitor对象1.3.3查看监控数据1.3.4grafana监控界面展示1.4云原生nginx-ingress监控1.4.1开启 Metrics 端口1.4.2修改service monitor 资源1.4.3查看监控数据1.4.4grafana监控界面展示二.非云原生应用监控2.1 Linux 主机监控2.1.1检查暴露服务2.1.2查看监控数据2.1.3grafana监控界面展示2.2非云原生应用 Redis 集群监控2.2.1部署redis集群2.2.2配置rbac权限2.2.3查看监控数据2.2.4grafana监控界面展示2.3非云原生应用 NGINX 监控2.3.1编辑configmap文件2.2.2上传nginx-exporter镜像2.3.3部署Nginx资源2.3.4创建service2.3.5创建servicemonitor2.3.6查看监控数据2.3.7grafana监控界面展示一.云原生应用监控1.1云原生组件 etcd 集群监控1.1.1查看etcd证书说明在控制面板查看证书grep -E key-file|cert-file /etc/kubernetes/manifests/etcd.yaml查看端口ss -tunlp | grep etcd测试etcd能否通过metric接口暴露出去curl -s \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ https://192.168.7.163:2379/metrics -k | tail -11.1.2创建etcd service1.编写yaml文件vim etcd-service.yml添加以下内容--- apiVersion: v1 kind: Endpoints metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system subsets: - addresses: - ip: 192.168.7.163 ports: - name: https-metrics port: 2379 protocol: TCP --- apiVersion: v1 kind: Service metadata: labels: app: etcd-prom name: etcd-prom namespace: kube-system spec: ports: - name: https-metrics port: 2379 protocol: TCP targetPort: 2379 type: ClusterIP2.运行yaml文件kubectl apply -f etcd-service.yml kubectl get svc etcd-prom -o wide -n kube-system3.测试通过svc抓取监控curl -s --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://10.101.115.69:2379/metrics -k | tail -11.1.3创建etcd证书说明确保k8s集群安全kubectl -n monitoring create secret generic etcd-ssl \ --from-file/etc/kubernetes/pki/etcd/ca.crt \ --from-file/etc/kubernetes/pki/etcd/server.crt \ --from-file/etc/kubernetes/pki/etcd/server.key查看证书kubectl get secret etcd-ssl -n monitoring绑定证书vim prometheus-prometheus.yaml添加以下内容secrets: - etcd-ssl运行yaml文件kubectl apply -f prometheus-prometheus.yaml kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- ls /etc/prometheus/secrets/etcd-ssl1.1.4创建ServiceMonitor说明ServiceMonitor 是 Prometheus Operator 定义的 CRD用来以声明式的方式告诉 Prometheus 该抓取哪些目标target取代了手动改prometheus.yml的scrape_configs1.编写yaml文件vim etcd-servicemonitor.yaml添加以下内容--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd namespace: monitoring labels: app: etcd #标签 appetcd spec: jobLabel: k8s-etcd-cluster-apps #用 Service 上的哪个 label 作为 job 名 endpoints: - interval: 30s port: https-metrics scheme: https tlsConfig: caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt certFile: /etc/prometheus/secrets/etcd-ssl/server.crt keyFile: /etc/prometheus/secrets/etcd-ssl/server.key insecureSkipVerify: true selector: matchLabels: app: etcd-prom #匹配标签 namespaceSelector: matchNames: - kube-system运行yaml文件kubectl apply -f etcd-servicemonitor.yaml kubectl get servicemonitor etcd -n monitoring1.1.5查看监控数据1.1.6grafana监控界面展示grafana导入etcd监控界面展示地址URLhttps://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/1.2云原生 kube-proxy 监控说明kube-proxy默认暴露两个端口一个是10249在/metrics端点暴露监控指标另一个是10256默认监听127.0.0.11.2.1查看端口ss -tunlp | grep kube-proxy测试能否访问资源curl 127.0.0.1:10249/metrics1.2.2修改监听地址说明修改监听端口监听所有端口。端口为本地Prometheus如果没有部署在这台机子上就无法监听到该端口。kubectl edit configmap kube-proxy -n kube-system添加以下内容metricsBindAddress: 0.0.0.0:10249重启Pod说明kube-proxy采用副本的方式构建一旦删除会立即重建kubectl -n kube-system get pod|grep kube-proxy|awk {system(kubectl -n kube-system delete pod $1)} ss -tunlp | grep kube-proxy1.2.3创建kube-proxy Servicevim kube-proxy-service.yaml添加以下内容--- apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy name: kube-proxy namespace: kube-system spec: selector: k8s-app: kube-proxy #筛选标签 type: ClusterIP ports: - name: https-metrics port: 10249 targetPort: 10249 protocol: TCP运行yamlkubectl apply -f kube-proxy-service.yaml kubectl get svc kube-proxy -n kube-system尝试抓取服务curl 10.110.8.62:10249/metrics1.2.4创建kube-proxy servicemonitor 资源1.编写yaml文件vim kube-proxy-servicemonitor.yaml添加以下内容--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-proxy namespace: monitoring labels: app: kube-proxy #Service Monitor标签 spec: jobLabel: kube-proxy endpoints: - interval: 30s port: https-metrics scheme: http selector: matchLabels: k8s-app: kube-proxy #匹配标签 namespaceSelector: matchNames: - kube-system2.运行yaml文件kubectl apply -f kube-proxy-servicemonitor.yaml kubectl get servicemonitor kube-proxy -n monitoring1.2.5查看监控数据1.2.6grafana监控界面展示说明prometheus本身自带granfan界面1.3云原生 calico 网络插件监控说明Calico 中最核心的组件就是Felix它负责设置路由表和 ACL规则等以便为该主机上的 endpoints 资源正常运行提供所需的网络连接。同时它还负责提供有关网络健康状况的数据例如报告配置其主机时发生的错误和问题这些数据会被写入etcd以使其对网络中的其他组件和操作人员可见。由此可见监控calico的核心便是监控felixfelix相当于calico的大脑。1.3.1开启 Metrics 端口说明calico在启动时默认没有打开Metrics监听kubectl edit ds calico-node -n kube-system添加以下内容spec: containers: - env: - name: FELIX_PROMETHEUSMETRICSENABLED value: True - name: FELIX_PROMETHEUSMETRICSPORT value: 9091 ports: - containerPort: 9091 name: http-metrics protocol: TCP1.3.2创建podMonitor对象说明ServiceMonitor 通过 Service 发现目标PodMonitor 直接选 Pod。前者适合有 Service 的服务后者适合没有 Service 或不想建 Service 的场景。1.创建yml文件vim prometheus-podMonitorCalico.yaml添加以下内容--- apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: labels: k8s-app: calico-node name: calico-node namespace: monitoring spec: podMetricsEndpoints: - interval: 15s path: /metrics port: http-metrics namespaceSelector: matchNames: - kube-system selector: matchLabels: k8s-app: calico-node #匹配标签2.运行ymlkubectl apply -f prometheus-podMonitorCalico.yaml kubectl get podmonitor calico-node -n monitoring1.3.3查看监控数据1.3.4grafana监控界面展示https://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/1.4云原生nginx-ingress监控1.4.1开启 Metrics 端口说明helm部署时默认没有开启metrics采集服务vim ingress-nginx.yml添加以下内容global: image: registry: reg.westos.org controller: image: image: ingress-nginx/controller tag: v1.13.3 digest: digestChroot: ingressClassResource: name: nginx default: true service: type: LoadBalancer admissionWebhooks: patch: image: registry: reg.westos.org image: ingress-nginx/kube-webhook-certgen tag: v1.6.3 digest: metrics: #开启metrics采集 enabled: true service: enabled: true labels: app: ingress-nginx-metrics serviceMonitor: enabled: true namespace: monitoring defaultBackend: enabled: true name: defaultbackend image: registry: reg.westos.org image: ingress-nginx/defaultbackend-amd64 tag: 1.52.使用helm部署Ingress-nginxhelm -n ingress-nginx upgrade ingress-nginx -f ingress-nginx.yml ingress-nginx-4.13.3.tgz kubectl -n ingress-nginx get svc ingress-nginx-controller-metrics测试curl 10.107.130.97:10254/metrics1.4.2修改service monitor 资源说明helm部署service monitor修改匹配标签kubectl -n monitoring edit servicemonitors ingress-nginx-controller添加以下内容spec: endpoints: - interval: 30s port: metrics scheme: http namespaceSelector: matchNames: - ingress-nginx selector: matchLabels: app: ingress-nginx-metrics在prometheus监控部署之后安装的服务会出现权限的问题需要配置rbac授权(补充)kubectl apply -f rbac.yml文件内容如下apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: ingress-nginx rules: - apiGroups: [] resources: [endpoints, pods, services] verbs: [get, list, watch] - apiGroups: [discovery.k8s.io] resources: [endpointslices] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: ingress-nginx subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io1.4.3查看监控数据1.4.4grafana监控界面展示https://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/二.非云原生应用监控说明非云原生应用不能使用metric暴露服务端口需要使用外部的 exporter暴露2.1 Linux 主机监控说明通过kube-prometheus技术栈安装的prometheus默认会以ds的形式在每一个节点上启动一个node-exporter pod并且会监控一个9100端口。该端口用来采集宿主机的相关监控指标。2.1.1检查暴露服务ss -tunlp | grep 9100 kubectl get svc node-exporter -n monitoring -o yaml kubectl get servicemonitor node-exporter -n monitoring -o yaml2.1.2查看监控数据2.1.3grafana监控界面展示默认面板中文面板131052.2非云原生应用 Redis 集群监控2.2.1部署redis集群1.编辑配置文件vim redis-values-metrics.yml添加以下内容global: imageRegistry: reg.westos.org security: allowInsecureImages: true redis: password: westos metrics: enabled: true serviceMonitor: enabled: true namespace: monitoring2.上传镜像docker pull bitnami/redis-exporter docker tag bitnami/redis-exporter reg.westos.org/bitnami/redis-exporter3.部署redis集群helm install redis-cluster -f redis-values-metrics.yml redis-23.2.6.tgz -n redis kubectl get all -n redis测试是否暴露服务curl 10.103.174.171:9121/metrics2.2.2配置rbac权限vim rbac.yml添加以下内容apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: redis rules: - apiGroups: [] resources: [endpoints, pods, services] verbs: [get, list, watch] - apiGroups: [discovery.k8s.io] resources: [endpointslices] verbs: [get, list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: redis subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io运行ymlkubectl apply -f rbac.yml2.2.3查看监控数据2.2.4grafana监控界面展示7632.3非云原生应用 NGINX 监控说明nginx有自己的监控模块stub_status。2.3.1编辑configmap文件vim default.conf添加以下内容server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; } location /status { stub_status on; } error_page 500 502 503 504 /50x.html; location /50x.html { root /usr/share/nginx/html; } }运行yml文件kubectl create cm nginx-conf --from-file./default.conf2.2.2上传nginx-exporter镜像docker pull nginx/nginx-exporter docker tsg nginx/nginx-exporter reg.westos.org/library/nginx-exporter docker push reg.westos.org/library/nginx-exporter2.3.3部署Nginx资源vim deploy-nginx.yml添加以下内容--- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server spec: selector: matchLabels: app: nginx replicas: 3 template: metadata: labels: app: nginx annotations: prometheus.io/scrape: true prometheus.io/port: 9113 spec: containers: - name: nginx image: reg.westos.org/nginx/nginx:1.29 ports: - containerPort: 80 volumeMounts: - name: nginxconf mountPath: /etc/nginx/conf.d - name: nginx-exporter image: reg.westos.org/nginx/nginx-prometheus-exporter:1.5.1 args: - -nginx.scrape-urihttp://localhost/status resources: limits: memory: 128Mi cpu: 500m ports: - containerPort: 9113 volumes: - name: nginxconf configMap: name: nginx-confkubectl apply -f deploy-nginx.yml kubectl get pod2.3.4创建servicevim service-nginx.yml添加以下内容--- apiVersion: v1 kind: Service metadata: name: webapp labels: app: nginx spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 name: web - protocol: TCP port: 9113 targetPort: 9113 name: metrics运行yml文件kubectl apply -f service-nginx.yml kubectl get svc webapp查看抓取数据curl 10.108.234.20/status curl 10.108.234.20:91132.3.5创建servicemonitorvim nginx-servicemonitor.yml添加以下内容--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nginx-exporter namespace: monitoring labels: k8sapp: nginx-exporter namespace: monitoring spec: jobLabel: k8s-app endpoints: - port: metrics interval: 30s scheme: http path: /metrics selector: matchLabels: app: nginx namespaceSelector: matchNames: - default运行yml文件kubectl apply -f nginx-servicemonitor.yml2.3.6查看监控数据2.3.7grafana监控界面展示https://github.com/nginx/nginx-prometheus-exporter/blob/main/grafana/dashboard.json