Null Humla Write-up on Mastering Nmap Script Engine

Aug. 6, 2015

As I promised here is a detailed write-up on null humla conducted on Mastering Nmap Script Engine at Mumbai null chapter on 18 July 2015. The basic agenda of the session is to learn how to write Nmap Scripts from scratch.

Introduction

Nmap has been playing an inevitable role in the security community over more than one and half decades. Though the project started as a simple network port scanner, it has been evolved into a massive toolset to do complete reconnaissance with a ton of impressive advanced techniques.

Evolution of Nmap

Nmap was first released on late September of 1997 by Gordon “Fyodor” Lyon. In the very early next year he founded insecure.org. The first version was written in C and included some basic scanning techniques. In March 1988, Renaud Deraison asked Fyodor to give him permission to use the code for one of the network scanner he was planning to develop. Fyodor gave him access and that tool was none other than what we call today as Nessus. By end of 1988, nmap was capable enough to have OS fingerprinting capabilities. The first version of nmap GUI called NmapFE GUI was released by April 1999.

Nmap further enhanced with timing options, and more scanning techniques such as window scan, ACK scan, and RPC scan, and also released windows version of nmap as well. In 2001, nmap introduced an interesting scanning technique called idle scanning, though it is practically difficult scenario. This uses pid to detect ports on the target machine by spoofing another system and there by hides original scanner’s IP address. In July 2002, Fyodor decided to leave his job at netscape/AOL and took nmap as his full time job. This made him add more features to nmap such as XML output, uptime detection, and OSX support. In the same year nmap was also got rewritten from C to C++.

Nmap introduced service scan or version scan in sept 2003, also added packet tracing, UDP scanning options, and OS classification system by feb 2004. In August 2004, nmap introduced ultra_scan feature that supports parallelization to improve speed and accuracy, also with some dramatic enhancement to scans systems behind firewalls. Google summer of code for nmap started in June 2005 that focused on ncat, zenmap and 2nd generation OS detection.

In May 2006, Nmap guys started working on NSE as a part of Google summer of code. Post mid of same year, nmap released it’s first collection of NSE with 23 simple scripts like sshv1, ss2v2 detection, ident service enumeration, dns recursive call support detection etc. It also released 2nd Generation OS database that we can see with current nmap installations. May 2007, David Fifield joined Google summer code and further continued to work as co-maintainer for nmap. Fyodor and David together worked together and shared the important milestones of nmap available at http://nmap.org/book/history-future.html.

July 2007, the first version of Zenmap released. By 2008, nmap got evolved with more features that includes 41 scripts, 1,307 OS fingerprints, and 4,706 version detection signatures. Later they lauched nmap script documentation http://nmap.org/nsedoc. Ncat and ndiff were added to nmap list by 2009. Nmap also made a milestone feature of identifying conflicker worm infections. The 2nd version of conflicker started blocking infected users from accessing nmap.org and insecure.org websites. Fyodor and team considered this as great complement though. Google summer of code in august 2009 came up with nping, ncrack and also improvements in NSE, ncap and zenmap. August 2010, favicon project started, this is available at http://nmap.org/favicon. In 2011, nmap launched sectools.org., and nmap got evolved with 3,572 OS fingerprints, 8,165 version detection, 348 NSE scripts by end of 2012. And it is still growing.

Introduction to Nmap Script Engine

Nmap Scripts are written in LUA. When you run aggressive scan (-A), nmap runs few scripts to enumerate information from target services. Also version scan (-sV) for some services also uses scripts.

Nmap Aggressive Scan

Nmap currently has 498 scripts, 114 libraries, and 14 categories. That doesn’t mean that nmap will run all these scripts against target IP addresses. Scripts are conditionally triggered, i.e., a specific and defined condition needs to be triggered to execute the script. These conditions are also written within each script. Nmap Scripts are located in the nmap installation directory itself along with other supporting files.

Nmap Path

You will find nse_main.lua file that is the main file required to execute Nmap scripts. This file includes options for scanning, pre-scanning details, post-scanning details, and also includes libraries and function used to execute scripts.

Every script follows certain naming conventions for convenient usage. For eg.: a script to bruteforce http service is named as "http-brute.nse"; another script to identify a vulnerability with specific CVE number in mysql will be named as "mysql-vuln-CVE2015-1234.nse". This makes pentesters easy to find right script for their scans. So when you write scripts, it’s good practice to follow the same naming conventions.

Lua Language

LUA is a scripting language with extensible semantics as the primary focus developed in 1993 by Roberto Ierusalimschy, Luiz Henrique de Figueiredo, and Waldemar Celes, members of the Computer Graphics Technology Group (Tecgraf) at the Pontifical Catholic University of Rio de Janeiro, in Brazil. Many well-known games and applications uses Lua languages including Adobe Photoshop Lightroom, Angry Birds, Apache HTTP Server, Apache Traffic Server, the Firefox web browser, MediaWiki, and World of Warcraft. Wireshark embeds Lua for writing taps, dissectors, and other scripting. To know more about usage of Lua, visit https://sites.google.com/site/marbux/home/where-lua-is-used

It’s easy to learn Lua language. People who code in python can get it’s syntax and semantics quite quickly. Like python, LUA is another scripting language that doesn’t bother about curly braces { }. Each modules, sections or chunks works based on indentation.

Lua can be installed in any linux machine using following command

$ sudo apt-get install lua5.1

A windows installer is available from here  https://code.google.com/p/luaforwindows/

Like python, LUA also supports working with interactive shell. If you want to exit from the interactive shell, you can type control characters (Ctrl+Z in windows, Ctrl+D in linux), or use os.exit() function.

root@kali:~# lua
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> print("Hello World!")
Hello World!
>

It is also possible to execute single line LUA code using –e option like below.

root@kali:~# lua -e 'print("Hello Work!")'
Hello Work!
root@kali:~#

If you want a program file (e.g.: filename.lua), you can call lua filename.lua directly from the command prompt. If you want to execute one LUA program, and get into LUA interactive shell instead of exiting into normal terminal, you could use –i options.

root@kali:~# lua -i 1.lua 
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
Hello!
Welcome to null humla
>

LUA Basics to Advance (for writing NSE)

Variables in LUA support nil, string, number, function, boolean, long string, thread, and table. However, variables are not strictly associated with any datatype; means same variables can take user data in multiple data types.

> v = "null humla"
> print(type(v))
string
> v = 10
> print(type(v))
number
>

LUA also supports arrays like other languages. An array can be defined as arrayname = {}. Following command will create an array automatically as arg to store command line arguments.

root@kali:~# lua -i 1.lua 
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
Hello!
Welcome to null humla
> print(arg[-2])
lua
> print(arg[-1])
-i
> print(arg[0])
1.lua
>

Lua uses io.read() and i.write() to do basic input and output operations. Like other programming languages, LUA also has conditional and looping statements.

Remember that LUA don’t have switch statement like other languages. If you want to quit from loop, you can use break statement.

When working with NSE, we would also need to create multiple functions to manage the code in well manner. Lua supports function very similar like other languages.

function function_name(arg1,arg2,…argn)
    Statements to execute
end

A long string in LUA is a special data type that helps in handling multi-line strings. For instance, if you want to deal with XML or HTML contents while executing NSE scripts, long string will be suitable to do this. Long String can be defined in following two ways.

longstr = [[
this is sample long string.
long string means multi-line string.
this is usefull for writing html/xml content.
]]

Lua5.2 supports escape sequence character \z. This escapes all subsequent characters (in following case, the new line characters) in the string until it finds the next first non-space character.

data = “\x90\x90\x90\x90\x90\x90\z
\x90\x90\x90\x90\x90\x90”

Lua has a series of functions to handle string. These functions are highly required while parsing scans results and printing the output in desired form.

These string functions can be used alternatively as shown below based on the developer’s convenience as shown below.

> str = "welcome to some string functions" 
> print(string.len(str))
32
> print(string.match(str,'some'))
some
> print(str:match('some'))
some
> print(str:match('other'))
nil
>

If you want to use type conversions, this can be done using tonumber() and tostring() functions. Many times, you would need to concatenate string with number or vice versa for the result printing from NSE scans. This can be done using double dots (..) operator as shown examples below.

> print(10 .. 20)
1020
> print(10 .. "" == '10') 
true
>

Lua doesn’t have many data structures. The only one, indeed powerful data structure it supports it Table. Table is an associative array representation, but with definable index values. It means arrays can have numeric indexes only. But tables can have indexes of any types except nil value. So types are not considered as variables; they are objects.

> mytable = {}
> k = 100
> mytable[k] = "hello world"
> mytable['size'] = 10
> print(mytable[100])
hello world
> print(mytable['size'])
10
>

Remember calling mytable.size is same as mytable[‘size’]. Following are the important function to deal with table.

You can use ipair() function to iterate over a table to handle the elements stored in it.

> weeks = {'monday','tuesday','wednesday','thursday','friday','saturday','sunday'}
> for i,j in ipairs(weeks) do
>> print(i,j)
>> end
1	monday
2	tuesday
3	wednesday
4	thursday
5	friday
6	saturday
7	sunday
>

Lua uses double hyphen to comment code lines. However, it also supports multi-line comments as well. Examples below:

— single line comment
–[[ this is multi-line comment
    this statement will not execute ]] —
— [[ code inside this will execute
       because triple hyphen will reactivate the commented section ]] —

Consider a scenario: Most of the time, we would need to run nikto scans on every web servers identified. This could be a hectic job running nikto individually on every web ports, especially when we are engaged with a huge rage of IP addresses.

Following code will run nikto scan on every web ports automatically and save the results in html format. It also has an option display that allows you to turn on or off the nikto scan result in nmap scan window.

https://github.com/s4n7h0/NSE/blob/master/http-nikto-scan.nse

Nmap NSE Scan

Developers Tips

Debugging nmap script is little tricky. While developing a new script, it is recommended not to mess around with the nmap default script directory. Write it in any folder, say home folder, make sure the script is error free and then copy to the nmap script folder. You could use debugging options (-d) to see where the script is breaking and what is the error caused. When you use script from any directory other than nmap script directory, it is recommended to specify the script path.

In the following example, I have used –datadir . to specify current directory and the debugging option –d.

$ nmap scanme.nmap.org –script http-nikto-scan –datadir . -d

Once the final script is built, you will need to run –script-updatedb command to update the script database. There’s always an option to run wireshark behind and see how script runs. However, it is also possible to run scan with –script-trace option.

Twitter @s4n7h0 | Instagram @s4n7h0frames | GitHub @s4n7h0 | Keybase @s4n7h0 |