Burp Extension: Directory and File Listing Parser - Burp Site Map Importer

Click Here to Download Source Code

Penetration testers, rejoice! While conducting application penetration tests it’s sometimes necessary to request specific information from the application owner or client. As a pen tester it can be extremely beneficial to perform a test with a full directory and file listing of the application, which sometimes can be difficult to acquire.

So let’s assume all clients are perfect and provide a full directory and file listing of their application (funny, I know) but what do we do with it? My process usually involves manually looking over everything trying to find keywords which jump out… I just might want to take a look at adminpassword.txt. Depending on the size of the application I may attempt to reach every file but usually this is not an efficient use of time. I wanted to create a quick and easy process for dealing with directory and file listings so I created a Burp Suite extension which will do a lot of the work for me.

My Burp extension contains two main features. The first feature is the ability to parse the listing file and generate a list of valid URLs to request each resource. The second feature is generating a request for each URL and importing the valid request/response pairs into Burp’s Target Site Map. Why is having a full site map helpful? We now have the ability to see the entire structure of the application, search within all valid responses, conduct manual testing or an active scan on ALL accessible resources, and much more. The process flow looks like this:

The Burp extension is written in python so a standalone jython jar will be needed to run it: Currently the extension is only tested and working with jython-standalone 2.5.3

After loading the extension you will have an option in the context menu to “Import Directory Listing”:

A GUI will appear for the extension. Fields such as hostname, SSL, and port will automatically populate depending on the request or response the menu option was originally invoked from. Cookies will also be displayed and used in any requests the extension makes. This feature makes it easy to compare site maps of two application user roles (based on varying session information such as cookies) to determine if each role has the correct access.

In this example I have selected the “Import Directory Listing” menu option on the DVWA web application which is running on my local machine. Now we must fill out all options on the left side of the GUI including hostname, full directory path (windows only, but CAN be used to modify URLs from a linux listing type) which specifies where the root of the web application sits, SSL, port, file listing type, and path to listing file.

On a Windows XP machine, I used the ‘dir /s’ command in cmd.exe which displays all files from the current directory and all sub directories. If the application is sitting on a Windows platform this is a very common command used for directory and file listings. A partial output of a directory and file listing for the DVWA web application (selected in the above image as C:\dvwa-listing.txt) looks like this:

dir /s:

 Volume in drive C has no label.
 Volume Serial Number is 5033-AA99

 Directory of C:\xampp\htdocs\dvwa

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
09/08/2010  09:49 PM               497 .htaccess
08/26/2010  11:15 AM             2,792 about.php
06/06/2010  07:55 PM             5,066 CHANGELOG.txt
09/08/2010  09:50 PM    <DIR>          config
03/16/2010  12:56 AM            33,107 COPYING.txt
09/08/2010  09:50 PM    <DIR>          docs
09/08/2010  09:50 PM    <DIR>          dvwa
03/16/2010  12:56 AM               883 ids_log.php
06/06/2010  07:52 PM             1,878 index.php
03/16/2010  12:56 AM             1,761 instructions.php
08/26/2010  11:18 AM             2,580 login.php
03/16/2010  12:56 AM             2,738 security.php
06/06/2010  10:58 PM             1,350 setup.php
09/08/2010  09:50 PM    <DIR>          vulnerabilities
              16 File(s)         59,772 bytes

 Directory of C:\xampp\htdocs\dvwa\config

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
08/26/2010  10:32 AM               576 config.inc.php
08/26/2010  11:06 AM               576 config.inc.php~
               2 File(s)          1,152 bytes

 Directory of C:\xampp\htdocs\dvwa\docs

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
08/26/2010  10:32 AM           526,043 DVWA-Documentation.pdf
               1 File(s)        526,043 bytes

The parser is also capable of parsing directory listing files from various linux commands such as ‘ls -lR’ and ‘ls –R’. Examples of the format are as follows:

ls -lR:

.:
total 124
-rw-rw-r--  1 user user  2792 Aug 26  2010 about.php
-rw-rw-r--  1 user user  5066 Jun  6  2010 CHANGELOG.txt
drwxrwxr-x  2 user user  4096 Jul  1 16:52 config
-rw-rw-r--  1 user user 33107 Mar 16  2010 COPYING.txt
drwxrwxr-x  2 user user  4096 Jul  1 16:52 docs
drwxrwxr-x  6 user user  4096 Jul  1 16:52 dvwa
-rw-rw-r--  1 user user   883 Mar 16  2010 ids_log.php
-rw-rw-r--  1 user user  1878 Jun  6  2010 index.php
-rw-rw-r--  1 user user  1761 Mar 16  2010 instructions.php
-rw-rw-r--  1 user user  2580 Aug 26  2010 login.php
-rw-rw-r--  1 user user   413 Mar 16  2010 logout.php
-rw-rw-r--  1 user user  2738 Mar 16  2010 security.php
-rw-rw-r--  1 user user  1350 Jun  6  2010 setup.php
drwxrwxr-x 11 user user  4096 Jul  1 16:52 vulnerabilities

./config:
total 8
-rw-rw-r-- 1 user user 576 Aug 26  2010 config.inc.php
-rw-rw-r-- 1 user user 576 Aug 26  2010 config.inc.php~

./docs:
total 516
-rw-rw-r-- 1 user user 526043 Aug 26  2010 DVWA-Documentation.pdf

./dvwa:
total 16
drwxrwxr-x 2 user user 4096 Jul  1 16:52 css
drwxrwxr-x 2 user user 4096 Jul  1 16:52 images
drwxrwxr-x 3 user user 4096 Jul  1 16:52 includes
drwxrwxr-x 2 user user 4096 Jul  1 16:52 js

ls -R:

.:
about.php
CHANGELOG.txt
config
COPYING.txt
docs
dvwa
external
favicon.ico
README.txt
robots.txt
security.php
setup.php
vulnerabilities

./config:
config.inc.php
config.inc.php~

./docs:
DVWA-Documentation.pdf

./dvwa:
css
images
includes
js

Now that we have a directory listing for our application, let’s parse out all of the directories and files and create valid URLs. Click the “Generate URL List” button to start the parsing. Tip: If the generated URLs don’t look correct you can modify the fields on the left of the GUI and regenerate the list or copy the list and use your own text editor to make changes. (Please e-mail SmeegeSec@gmail.com with any parsing issues or suggestions)

A text area is populated with the URLs and the total count of directories and files processed.

At this point we have a few options. We can take our list of URLs and use them in Burp’s Intruder. To do this it would be very easy, all that needs to be done is remove the protocol, hostname, and port from each URL within a text editor. From there we take the path of each resource as a payload in a GET request in Intruder. We could then look to see which resources we are able to reach by analyzing status codes and content length. A second option is built into the extension via the “Import URL List to Burp Site Map” button. This button makes a request to each URL in the list (with cookie information, if it was found) and if a valid response is returned, will add the request and response to Burp’s Site Map. Requests with keywords such as logout, logoff, etc. are skipped to avoid ending sessions. The import to site map functionality was one of the main features I wanted to implement.

Warning: Actual requests are being made. Remove any resources you don't want being made, such as delete_database.php!! Regex to remove resources will be added in updates.

Done! A message dialog tells the user how many URLs were valid and imported into the site map. In the above image you can see a full site map and proxy history which was not found by spidering or brute forcing directories/files, but rather a simple directory and file listing of the application. With a full site map we are now ahead of the game. If you have multiple testers testing an application you can save the state in Burp and distribute it to save time, almost completely bypassing the discovery phase.

Note: So far the parsing does not consider virtual directories or different URL mappings from different web frameworks. Future updates may include parsing of mapping files such as ASP.NET’s web.config and Java’s web.xml.

Tip: Running a plugin multiple times or multiple plugins at a time may require increased PermGen, an example to increase the max when launching Burp would be:

java -XX:MaxPermSize=1G -jar burp.jar

Also, please provide feedback if you use this extension. With many different output formats for directory and file listings it can be difficult to write a dynamic parser which works for every format. If you have a listing file which is not being properly parsed please contact me so I can include it in an update. Thanks!

Click Here to Download Source Code