问题描述

接到阿里云报警邮件，说是一台ECS有恶意进程。查看阿里云的安全详情，发现有恶意进程(云查杀)-自变异木马：

登录到服务器上检查/bin目录，发现该文件确实不对，大小变成的1.1M，类似的还有netstat。如下图：
正常ubuntu系统下的ps才96K，netstat大小为117K，上述命令文件被恶意窜改了。

检查

根据手动分析的结果发现，有多处系统文件被感染。通过lastlog等命令查看系统登录及操作，并未发现有被入侵的痕迹。
分析启动项，发现多出了如下文件：

/etc/rc1.d/S97VsystemsshMdt
/etc/rc2.d/S97VsystemsshMdt
/etc/rc3.d/S97VsystemsshMdt
/etc/rc4.d/S97VsystemsshMdt
/etc/rc5.d/S97VsystemsshMdt
/etc/rc1.d/S99selinux
/etc/rc2.d/S99selinux
/etc/rc3.d/S99selinux
/etc/rc4.d/S99selinux
/etc/rc5.d/S99selinux

两个文件都是软链接：
可以明确感染文件不止一处，所以需要通过软件进行全盘扫描。linux下比较常见的杀毒工具是clamav，于是决定安装扫描全盘。

clamav

安装

sudo apt install clamav*

值得注意的是，在安装过程中，一度出现了因S97VsystemsshMdt和S99selinux启动项冲突而无法安装的问题，这个时候需要将上述问题文件全部删除，重新安装即可。问题日志如下：

Setting up clamav-freshclam (0.100.3+dfsg-0ubuntu0.16.04.1) ...
insserv: warning: script 'S99selinux' missing LSB tags and overrides
insserv: warning: script 'S97VsystemsshMdt' missing LSB tags and overrides
insserv: warning: script 'selinux' missing LSB tags and overrides
insserv: warning: script 'VsystemsshMdt' missing LSB tags and overrides
insserv: There is a loop between service plymouth and procps if started
insserv:  loop involving service procps at depth 2
insserv:  loop involving service udev at depth 1
insserv: There is a loop at service selinux if started
insserv: There is a loop at service plymouth if started
insserv: Starting selinux depends on plymouth and therefore on system facility `$all' which can not be true!
insserv: Starting VsystemsshMdt depends on plymouth and therefore on system facility `$all' which can not be true!
insserv: Starting selinux depends on plymouth and therefore on system facility `$all' which can not be true!
insserv: Starting VsystemsshMdt depends on plymouth and therefore on system facility `$all' which can not be true!

......

insserv: Max recursions depth 99 reached
insserv: There is a loop between service selinux and hwclock if started
insserv:  loop involving service hwclock at depth 1
insserv:  loop involving service checkroot at depth 3
insserv:  loop involving service mountdevsubfs at depth 1
insserv:  loop involving service networking at depth 4
insserv:  loop involving service selinux at depth 1
insserv: exiting now without changing boot order!
update-rc.d: error: insserv rejected the script header
dpkg: error processing package clamav-freshclam (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up libcurl3:amd64 (7.47.0-1ubuntu2.13) ...
dpkg: dependency problems prevent configuration of clamav:
 clamav depends on clamav-freshclam (>= 0.100.3+dfsg) | clamav-data; however:
  Package clamav-freshclam is not configured yet.
  Package clamav-data is not installed.
  Package clamav-freshclam which provides clamav-data is not configured yet.

dpkg: error processing package clamav (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.2) ...
Processing triggers for ureadahead (0.100.0-19) ...
Errors were encountered while processing:
 clamav-freshclam
 clamav 

更新病毒库

sudo freshclam

全盘扫描

clamscan -ri / -l o

其中-r表示递归扫描子目录，-i表示只显示发现的病毒文件，-l表示将扫描结果输出到相应的文件位置。全盘扫描大概需要24分钟，结果如下：

可以看到，所有受感染的文件大概有60个，其中包括病毒库，非官方库等文件。手动筛选后，得如下结果：

/root/ps: Legacy.Trojan.Agent-1388639 FOUND
/root/netstat: Legacy.Trojan.Agent-1388639 FOUND
/usr/bin/lsof: Legacy.Trojan.Agent-1388639 FOUND
/usr/bin/pythno: Legacy.Trojan.Agent-1388639 FOUND
/usr/bin/bsd-port/knerl: Legacy.Trojan.Agent-1388639 FOUND
/bin/ps: Legacy.Trojan.Agent-1388639 FOUND
/bin/netstat: Legacy.Trojan.Agent-1388639 FOUND

将上述文件删除，系统命令可以从其他同版本ECS上拷贝过来修复。此后，阿里云没有再出现过木马报警。

总结

本文记录了一次简单的木马排查过程，供安全小白参考，毕竟现在很多后端都没有安全方面的经验。本文处理比较简单，但基本上能解决掉木马问题。这也提醒了广大后端程序员，服务器安全还是得引起重视。
有一些遗留问题，至此仍不清楚服务器是如何感染的。没有发现其他非法登录的情况，猜测有可能是使用了一些nodejs非官方库导致的。至于这些木马是干什么的，出于时间考虑，也没有做进一步分析。