[TOC]

0x00 前言

描述:本文章主要针对于本人日常运维所遇到的一些问题并进行总结解决思路流程;




#### 0x01 Linux信息收集
描述:当我们对异常系统进行处理,必须先进行主机基础信息的收集,以防出错后可以更快的恢复或者求助;


CentOS系列:
#!/bin/bash
echo "系统版本:$(cat /etc/redhat-release)"
echo "内核信息:$(uname -a)"
echo "SeLinux values 设置情况:$(getenforce)"

echo -e "用户信息:\n$(getent passwd)"
echo -e "密码信息:\n$(getent shadow)"

echo -e "网络信息:\n$(ip addr show)"

echo "CPU信息:$(cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c)"
echo "物理CPU数:$(cat /proc/cpuinfo |grep 'physical id'|sort |uniq|wc -l)"
echo "逻辑CPU数:$(cat /proc/cpuinfo |grep "processor"|wc -l)"
echo "CPU核心数:$(cat cat /proc/cpuinfo |grep "cores"|uniq)"
echo "CPU综合信息:\n$(lscpu)"

echo -e "磁盘UUID信息:\n$(blkid)"
echo -e "磁盘信息:\n$(fdisk -l | egrep '/dev|Disk')"
echo -e "磁盘分区信息:\n$(lsblk)"
echo -e "磁盘空间信息:\n$(df -h)"

echo -e "挂载信息:\n$(mount -l)"
echo -e "挂载配置文件:\n$(cat /etc/fstab | egrep -v '#|^$')"


CPU:通过下面的脚本来打印出当前机器的socket,core和thread的数量
#!/bin/bash

# Simple print cpu topology
# Author: kodango

function get_nr_processor()
{
grep '^processor' /proc/cpuinfo | wc -l
}

function get_nr_socket()
{
grep 'physical id' /proc/cpuinfo | awk -F: '{
print $2 | "sort -un"}' | wc -l
}

function get_nr_siblings()
{
grep 'siblings' /proc/cpuinfo | awk -F: '{
print $2 | "sort -un"}'
}

function get_nr_cores_of_socket()
{
grep 'cpu cores' /proc/cpuinfo | awk -F: '{
print $2 | "sort -un"}'
}

echo '===== CPU Topology Table ====='
echo

echo '+--------------+---------+-----------+'
echo '| Processor ID | Core ID | Socket ID |'
echo '+--------------+---------+-----------+'

while read line; do
if [ -z "$line" ]; then
printf '| %-12s | %-7s | %-9s |\n' $p_id $c_id $s_id
echo '+--------------+---------+-----------+'
continue
fi

if echo "$line" | grep -q "^processor"; then
p_id=`echo "$line" | awk -F: '{print $2}' | tr -d ' '`
fi

if echo "$line" | grep -q "^core id"; then
c_id=`echo "$line" | awk -F: '{print $2}' | tr -d ' '`
fi

if echo "$line" | grep -q "^physical id"; then
s_id=`echo "$line" | awk -F: '{print $2}' | tr -d ' '`
fi
done < /proc/cpuinfo

echo

awk -F: '{
if ($1 ~ /processor/) {
gsub(/ /,"",$2);
p_id=$2;
} else if ($1 ~ /physical id/){
gsub(/ /,"",$2);
s_id=$2;
arr[s_id]=arr[s_id] " " p_id
}
}

END{
for (i in arr)
printf "Socket %s:%s\n", i, arr[i];
}' /proc/cpuinfo

echo
echo '===== CPU Info Summary ====='
echo

nr_processor=`get_nr_processor`
echo "Logical processors: $nr_processor"

nr_socket=`get_nr_socket`
echo "Physical socket: $nr_socket"

nr_siblings=`get_nr_siblings`
echo "Siblings in one socket: $nr_siblings"

nr_cores=`get_nr_cores_of_socket`
echo "Cores in one socket: $nr_cores"

let nr_cores*=nr_socket
echo "Cores in total: $nr_cores"

if [ "$nr_cores" = "$nr_processor" ]; then
echo "Hyper-Threading: off"
else
echo "Hyper-Threading: on"
fi

echo
echo '===== END ====='


0x02 异常解决

问题1.业务服务器 CPU 占用负载高问题
排查思路:

#1.定位高负载进程 pid 通过观察load average,以及负载评判标准确认服务器是否存在负载较高的情况;
$top -n 1
#2.找到相关负载进程的子线程PID
$top -Hp [PID]
#3.将线程PID转换为 16进制,为后面查找 jstack 日志做准备
printf "0x%x" [PID] #比如0x431
#4.jstack日志异常查询
jstack 1040|vim +/0x431 -
#5.定位具体的异常业务使用 pwdx 命令根据 pid 找到业务进程路径
pwdx [PID]

WeiyiGeek.CPU占用率高