linux定位进程死锁,Linux内核死锁(deadlock)检测

业务运行过程中,Linux系统僵死,屏幕无任何有效打印信息,网络中断、键盘鼠标没有任何响应。这种故障现象,可能是因为Linux内核死锁导致。由于无任何有效打印信息,内核日志中也没有记录,就无法定位故障根因。

如何让Linux内核在僵死前打印相关信息,对问题定位尤为关键。其中一个有效手段是打开“Kernel Hacking”选项,然后重新编译内核。

对于Linux内核死锁有帮助的几个配置选项有:

[*] Detect Soft Lockups

[ ] Panic (Reboot) On Soft Lockups

[*] Detect Hung Tasks

(120) Default timeout for hung task detection (in seconds)

[*] Panic (Reboot) On Hung Tasks

[ ] Lock usage statistics

[ ] Spinlock debugging: sleep-inside-spinlock checking

e31b0c61314a5b5f91d55341c833836f.png

下面一个实例是在SLES11.1 2.6.32.12-0.7内核中系统僵死前,检测到内核死锁并panic,打印出了死锁进程和信息。

ftp           D ffff88010d7c5978  4472 22481   4259 0x00000000

ffff880031d1dd08 0000000000000016 ffff8801025cf6f8 0000000000000005

ffff880031d1dc08 ffff880031d1dfd8 000000000000ca98 00000000001d1580

0000000000004000 00000000001d1580 ffff880031d1dca8 ffffffff8108970c

Call Trace:

[] ? __lock_acquire+0xf7f/0x1038

[] ? sched_clock_cpu+0x9a/0x183

[] ? mark_held_locks+0x6e/0xa3

[] ? mutex_lock_nested+0x2cf/0x4ff

[] mutex_lock_nested+0x2e5/0x4ff

[] ? generic_file_aio_write+0x62/0xea

[] ? generic_file_aio_write+0x62/0xea

[] generic_file_aio_write+0x62/0xea

[] do_sync_write+0xee/0x157

[] ? autoremove_wake_function+0x0/0x4d

[] vfs_write+0xed/0x1d1

[] sys_write+0x5c/0x9f

[] system_call_fastpath+0x16/0x1b

1 lock held by ftp/22481:

#0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [] generic_file_aio_write+0x62/0xea

Kernel panic – not syncing: hung_task: blocked tasks

Pid: 536, comm: khungtaskd Not tainted 2.6.32.120609_new #14

Call Trace:

[] panic+0x9f/0x228

[] watchdog+0x33c/0x3a2

[] ? watchdog+0x120/0x3a2

[] ? watchdog+0x0/0x3a2

[] kthread+0xa0/0xaf

[] child_rip+0xa/0x20

[] ? finish_task_switch+0x0/0xfe

[] ? restore_args+0x0/0x30

[] ? kthreadd+0xc0/0x17b

[] ? kthread+0x0/0xaf

[] ? child_rip+0x0/0x20