K8S學習之基礎三十六：node-exporter部署

Prometheus v2.2.1

? 編寫yaml文件，包含創建ns、configmap、deployment、service

# 創建monitoring空間
vi prometheus-ns.yaml 
apiVersion: v1
kind: Namespace
metadata:name: monitoring# 創建SA并綁定權限
kubectl create serviceaccount monitor -n monitoring?
kubectl create clusterrolebinding monitor-clusterrolebinding -n monitoring --clusterrole=cluster-admin  --serviceaccount=monitoring:monitor
kubectl?create?clusterrolebinding?monitor-clusterrolebinding-1 ?-n?monitoring?--clusterrole=cluster-admin???--user=system:serviceaccount:monitor:monitoring# 創建cm、deployment、svc
apiVersion: v1
kind: ConfigMap
metadata:labels:app: prometheusname: prometheus-confignamespace: monitoring
data:prometheus.yml: |global:scrape_interval: 15sscrape_timeout: 10sevaluation_interval: 1mscrape_configs:- job_name: 'kubernetes-node'kubernetes_sd_configs:- role: noderelabel_configs:- source_labels: [__address__]regex: '(.*):10250'replacement: '${1}:9100'target_label: __address__action: replace- action: labelmapregex: __meta_kubernetes_node_label_(.+)- job_name: 'kubernetes-node-cadvisor'kubernetes_sd_configs:- role:  nodescheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- target_label: __address__replacement: kubernetes.default.svc:443- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor- job_name: 'kubernetes-apiserver'kubernetes_sd_configs:- role: endpointsscheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_po
rt_name]action: keepregex: default;kubernetes;https- job_name: 'kubernetes-service-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: kubernetes_name 
---
apiVersion: apps/v1
kind: Deployment
metadata:name: prometheus-servernamespace: monitoringlabels:app: prometheus
spec:replicas: 1selector:matchLabels:app: prometheuscomponent: server#matchExpressions:#- {key: app, operator: In, values: [prometheus]}[root@mast01 prometheus]# kubectl get pod -n monitoring
NAME                         READY   STATUS    RESTARTS   AGE
prometheus-dfdfb9d79-h4tk4   1/1     Running   0          22h
[root@mast01 prometheus]# kubectl get svc -n monitoring
NAME         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
prometheus   NodePort   10.109.44.37   <none>        9090:30090/TCP   22h
[root@mast01 prometheus]# curl 10.109.44.37:9090
<a href="/graph">Found</a>.[root@mast01 prometheus]# kubectl get ep -n monitoring
NAME         ENDPOINTS            AGE
prometheus   10.244.140.67:9090   22h
[root@mast01 prometheus]# curl 172.16.80.131:30090
<a href="/graph">Found</a>.

瀏覽器訪問 172.16.80.131:30090

node-exporter組件安裝和配置

? node-exporter可以采集機器（物理機、虛擬機、云主機等）的監控指標數據，能夠采集到的指標包括CPU, 內存，磁盤，網絡，文件數等信息。

# 上傳node-exporter.tar.gz到harbor
docker load -i node-exporter.tar.gz
docker tag prom/node-exporter:v0.16.0 172.16.80.140/node-exporter/node-exporter:v0.16.0
docker push 172.16.80.140/node-exporter/node-exporter:v0.16.0
# 創建daemon控制器的yaml
[root@mast01 prometheus]#  vi node-export.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:name: node-exporternamespace: monitoringlabels:name: node-exporter
spec:selector:matchLabels:name: node-exportertemplate:metadata:labels:name: node-exporterspec:hostPID: truehostIPC: truehostNetwork: truecontainers:- name: node-exporterimage: 172.16.80.140/node-exporter/node-exporter:v0.16.0imagePullPolicy: IfNotPresentports:- containerPort: 9100resources:requests:cpu: 0.15securityContext:privileged: trueargs:- --path.procfs- /host/proc- --path.sysfs- /host/sys- --collector.filesystem.ignored-mount-points- '"^/(sys|proc|dev|host|etc)($|/)"'volumeMounts:- name: devmountPath: /host/dev- name: procmountPath: /host/proc- name: sysmountPath: /host/sys- name: rootfsmountPath: /rootfstolerations:- key: "node-role.kubernetes.io/control-plane"operator: "Exists"effect: "NoSchedule"volumes:- name: prochostPath:path: /proc- name: devhostPath:path: /dev- name: syshostPath:path: /sys- name: rootfshostPath:path: /
# hostNetwork、hostIPC、hostPID都為True時，表示這個Pod里的所有容器，會直接使用宿主機的網絡，直接與宿主機進行IPC（進程間通信）通信，可以看到宿主機里正在運行的所有進程。加入了hostNetwork:true會直接將我們的宿主機的9100端口映射出來，從而不需要創建service 在我們的宿主機上就會有一個9100的端口

? 該yaml會在每個節點上部署一個node-exporter

[root@mast01 prometheus]# netstat -an | grep 9100
tcp6       0      0 :::9100                 :::*                    LISTEN
[root@mast01 prometheus]# kubectl get pods -n monitoring -owide
NAME                 READY STATUS  RESTARTS AGE IP            NODE   NOMINATED NODE   READINESS GATES
node-exporter-5ms5j  1/1   Running 0        7s  172.16.80.133 node02 <none>    <none>
node-exporter-dgg2z  1/1   Running 0        9s  172.16.80.131 mast01 <none>    <none>
node-exporter-k4mf5  1/1   Running 0        6s  172.16.80.132 node01 <none>    <none>

? 通過curl可獲取主機信息

[root@mast01 prometheus]# curl http://172.16.80.131:9100/metrics | more% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00010856
go_gc_duration_seconds{quantile="0.25"} 0.00010856
go_gc_duration_seconds{quantile="0.5"} 0.000879866
go_gc_duration_seconds{quantile="0.75"} 0.000879866
go_gc_duration_seconds{quantile="1"} 0.000879866
go_gc_duration_seconds_sum 0.000988426
go_gc_duration_seconds_count 2
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 6
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.9.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.863448e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.928912e+06
......
curl http://172.16.80.131:9100/metrics | grep node_cpu_seconds   # 顯示cpu使用情況% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
100 91593  100 91593    0     0  6934k      0 --:--:-- # HELP node_cpu_seconds_total Seconds the cpus spent in each mode.
--:--# TYPE node_cpu_seconds_total counter     # counter類型，只增不減
:--node_cpu_seconds_total{cpu="0",mode="idle"} 74516.69-node_cpu_seconds_total{cpu="0",mode="iowait"} 13.52
-:node_cpu_seconds_total{cpu="0",mode="irq"} 0
--node_cpu_seconds_total{cpu="0",mode="nice"} 0.03
:-node_cpu_seconds_total{cpu="0",mode="softirq"} 76.96
- node_cpu_seconds_total{cpu="0",mode="steal"} 0
74node_cpu_seconds_total{cpu="0",mode="system"} 601.74
53node_cpu_seconds_total{cpu="0",mode="user"} 1234.91
knode_cpu_seconds_total{cpu="1",mode="idle"} 74511.08node_cpu_seconds_total{cpu="1",mode="iowait"} 24.89
node_cpu_seconds_total{cpu="1",mode="irq"} 0
node_cpu_seconds_total{cpu="1",mode="nice"} 0.18
node_cpu_seconds_total{cpu="1",mode="softirq"} 86.84
node_cpu_seconds_total{cpu="1",mode="steal"} 0
node_cpu_seconds_total{cpu="1",mode="system"} 598.25
node_cpu_seconds_total{cpu="1",mode="user"} 1217.33curl http://172.16.80.131:9100/metrics | grep node_load # 最近一分鐘內負載使用情況% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP node_load1 1m load average.
# TYPE node_load1 gauge   # gauge類型，可增可減
node_load1 0.27
# HELP node_load15 15m load average.
# TYPE node_load15 gauge
node_load15 0.42
# HELP node_load5 5m load average.
# TYPE node_load5 gauge
node_load5 0.43
100 91584  100 91584    0     0  8262k      0 --:--:-- --:--:-- --:--:-- 8943k