[k8s]metricbeat的kubernetes模塊kube-metric模塊

正確姿勢啟動metricbeat

metricbeat.modules:
- module: systemmetricsets:- cpu- filesystem- memory- network- processenabled: trueperiod: 10sprocesses: ['.*']cpu_ticks: falseoutput.elasticsearch:hosts: ["http://192.168.x.x:9200"]setup.template.name: "metricbeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.settings:index.number_of_shards: 1index.number_of_replicas: 1setup.kibana.host: "192.168.x.x:5601"
setup.dashboards.enabled: true

./metricbeat -e -c metricbeat.yml -d "publish"

metricbeat的Kubernetes插件

metricbeat.modules:
# Node metrics, from kubelet:
- module: kubernetesmetricsets:- node- system- pod- container- volumeperiod: 10shosts: ["192.168.x.132:10255","192.168.x.133:10255"]# State metrics from kube-state-metrics service:  #這部分需要kube-state-metrics支撐(Nodes/deploy個數),因此集群里需要安裝kube-state-metrics組件.默認情況下這里沒有的話圖中上半部分沒有數據
- module: kubernetesenabled: false  ## 這里一定要是truemetricsets:- state_node- state_deployment- state_replicaset- state_pod- state_containerperiod: 10shosts: ["192.168.x.x:8080"]# Kubernetes events
- module: kubernetesenabled: falsemetricsets:- eventoutput.elasticsearch:hosts: ["http://192.168.x.x:9200"]

既然需要kube-state-metrics了,那就研究下吧.

監控思路

• vm思路:遵從大道至簡的原則
  

• 容器監控思路:
  pod的:
  top cpu
  top mem

• 別人家的監控-只能流口水了
  

kube-state-metrics提供監控k8s集群的metric

kube-state-metrics本質上是提供了k8s集群的metric
部署參考:
https://github.com/kubernetes/kube-state-metrics

克隆代碼: create這幾個即可

訪問:

報錯解決:
kubectl -n kube-system logs -f kube-state-metrics-215312065-1vzg2 -c kube-state-metrics
時候發現一些api錯誤

解決: 給apiserver啟動添加這個參數: 它github上有說

--runtime-config=batch/v2alpha1=true

即目前完整的apiserver參數如下

 kube-apiserver     --service-cluster-ip-range=10.254.0.0/16     --etcd-servers=http://127.0.0.1:2379     --insecure-bind-address=0.0.0.0     --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt    --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --runtime-config=batch/v2alpha1=true --v=2

k8s插件監控集群效果圖

vm監控效果圖

貼上kube-metric的yaml

參考: https://github.com/kubernetes/kube-state-metrics

修改點:

• deploy的api版本

• svc為nodeport模式

• 鏡像拉取

docker pull lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0
dokcer pull lanny/gcr.io_google_containers_addon-resizer_1.0
docker tag lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0 gcr.io/google_containers/kube-state-metrics:v1.1.0 
docker tag lanny/gcr.io_google_containers_addon-resizer_1.0 gcr.io/google_containers/addon-resizer:1.0 

$ cat kube-state-metrics-deployment.yaml 
apiVersion: apps/v1beta1 # for versions before 1.8.0 use apps/v1beta1  ## 這里默認是v1beat1,我用的是1.7.8,因此apiVersion: apps/v1beta2是不適用的.
kind: Deployment
metadata:name: kube-state-metricsnamespace: kube-system
spec:selector:matchLabels:k8s-app: kube-state-metricsreplicas: 1template:metadata:labels:k8s-app: kube-state-metricsspec:serviceAccountName: kube-state-metricscontainers:- name: kube-state-metricsimage: quay.io/coreos/kube-state-metrics:v1.1.0ports:- name: http-metricscontainerPort: 8080readinessProbe:httpGet:path: /healthzport: 8080initialDelaySeconds: 5timeoutSeconds: 5- name: addon-resizerimage: gcr.io/google_containers/addon-resizer:1.0resources:limits:cpu: 100mmemory: 30Mirequests:cpu: 100mmemory: 30Mienv:- name: MY_POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: MY_POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacecommand:- /pod_nanny- --container=kube-state-metrics- --cpu=100m- --extra-cpu=1m- --memory=100Mi- --extra-memory=2Mi- --threshold=5- --deployment=kube-state-metrics

$ cat kube-state-metrics-service.yaml 
apiVersion: v1
kind: Service
metadata:name: kube-state-metricsnamespace: kube-systemlabels:k8s-app: kube-state-metricsannotations:prometheus.io/scrape: 'true'
spec:type: NodePort ## 這里需要改下,外界通過nodeport方式訪問metric apiports:- name: http-metricsport: 8080targetPort: http-metricsprotocol: TCPselector:k8s-app: kube-state-metrics

$ cat kube-state-metrics-service-account.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:name: kube-state-metricsnamespace: kube-system