问题：使用kubectl top node查看kubernetes节点资源的使用情况时，发现某节点的内存使用率为102%，跟服务器上free -mh的结果很不一致
- kubectl top node，显示结果为102%
- free -mh，根据显示的结果计算，内存使用百分比为97.8%
所以，这个102的值是如何计算出来的呢？
- 使用命令：kubectl describe node [node_ip]，显示结果如下
[root@prd-k8s1 scripts]# kubectl describe node 10.10.189.124 Name: 10.10.189.124 Roles: resource_node Labels: 1=c87e2267-1001-4c70-bb2a-ab41f3b81aa3 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=10.10.189.124 kubernetes.io/os=linux node-role.kubernetes.io/resource_node= Annotations: node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Thu, 08 Jul 2021 22:59:23 +0800 Taints: node.kubernetes.io/unschedulable:NoSchedule Unschedulable: true Lease: HolderIdentity: 10.10.189.124 AcquireTime: <unset> RenewTime: Wed, 05 Jan 2022 18:39:13 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Wed, 05 Jan 2022 18:37:05 +0800 Tue, 07 Sep 2021 15:03:07 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 05 Jan 2022 18:37:05 +0800 Tue, 07 Sep 2021 15:03:07 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 05 Jan 2022 18:37:05 +0800 Tue, 07 Sep 2021 15:03:07 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 05 Jan 2022 18:37:05 +0800 Sat, 23 Oct 2021 07:46:04 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.10.189.124 Hostname: 10.10.189.124 Capacity: cpu: 32 ephemeral-storage: 51351044Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 32895412Ki pods: 110 Allocatable: cpu: 31300m ephemeral-storage: 47325122073 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 29774260Ki pods: 110 System Info: Machine ID: 57fa4b04708c4af388220eadf9d0a738 System UUID: 096628e4-7b1c-4065-bd3d-0d553c42e08b Boot ID: e73a6439-ce00-4c89-ae01-f3e755eb0e13 Kernel Version: 4.18.8-1.el7.elrepo.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.3 Kubelet Version: v1.17.0 Kube-Proxy Version: v1.17.0 PodCIDR: 172.23.8.0/24 PodCIDRs: 172.23.8.0/24 Non-terminated Pods: (11 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-app-platform-server-699fd8547f-gsrxl 500m (1%) 4 (12%) 5Gi (17%) 6Gi (21%) 100d c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-engineeringdata-server-678ffc5cf9-pt7dl 2 (6%) 5 (15%) 1Gi (3%) 4Gi (14%) 3d23h c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-iuap-busi-7c9658fd4d-8cglj 4 (12%) 4 (12%) 5Gi (17%) 8144Mi (28%) 3h21m c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-iuap-mdf-server-54985f998f-hnm5q 2 (6%) 2 (6%) 512Mi (1%) 512Mi (1%) 33d c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-metadata-599db6bf78-t6qkr 500m (1%) 2 (6%) 1536Mi (5%) 3Gi (10%) 69d c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-mrjt-base-be-554fc4b7fc-b5czf 10 (31%) 10 (31%) 6Gi (21%) 6Gi (21%) 3d23h c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-mrjt-bill-be-7469fb46d7-ns8cb 2 (6%) 10 (31%) 3Gi (10%) 6Gi (21%) 5d10h c87e2267-1001-4c70-bb2a-ab41f3b81aa3 online-tenantauth-8699fb69f6-p2d2k 500m (1%) 2 (6%) 1536Mi (5%) 2Gi (7%) 102d kube-system calico-node-nzbqc 250m (0%) 250m (0%) 256Mi (0%) 256Mi (0%) 180d kube-system monitor-agent-76zgr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 180d logging log-fluentd-dg6fg 100m (0%) 0 (0%) 200Mi (0%) 1000Mi (3%) 31h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 21850m (69%) 39250m (125%) memory 24520Mi (84%) 37560Mi (129%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SystemOOM 3m27s kubelet, 10.10.189.124 System OOM encountered, victim process: java, pid: 60865 - 结合kubectl top node的结果中内存使用量为29735Mi，kubectl describe node的结果中Allocatable的memory属性的值为29774260Ki，这个102%的计算公式为((29735X1024)/29774260)，约等于102.264%，与实际的显示结果吻合。
总结：
- 内存使用率=pod used total memory/Allocatable Memory
- pod used total memory就是在这台机器上的所有docker容器占用内存总和；
- Allocatable Memory就是kubectl describe no 查看出该节点可分配的内存
版权声明：本文为Ivan45007原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。