Custom Nmap Scripts

You've automated your basic scans, logged your results, and run Nmap's built-in script engine a hundred times. But what happens when you need to check for a custom internal service, a proprietary application header, or a vulnerability that no published NSE script covers? You're stuck re-running manual commands and copy-pasting output into spreadsheets. That's the pain we're solving here: by learning to write custom Nmap scripts for automation, you're moving from being a passive scanner operator to an active tool builder. In this lesson, you'll learn how to harness Nmap's scripting engine (NSE) to automate reconnaissance and vulnerability checks that are tailored exactly to your network — a critical skill for any ethical hacker who needs to scale their testing beyond the default library.

The problem this lesson solves

Imagine you're running a penetration test on a client's network. You've discovered a web application on port 8080 that isn't in any standard database. Your first instinct is to manually inspect the HTTP headers, check for default credentials, and scan for known paths — all with separate tools and manual commands. This approach is slow, error-prone, and impossible to replicate across multiple hosts without massive effort.

Nmap's built-in script library covers many common services, but it's far from complete. When you encounter a proprietary protocol, a niche application, or a specific security misconfiguration, the default scripts simply don't help. Without the ability to write custom Nmap scripts for automation, you're left with three bad options:

1. Manual inspection of every host — insanely slow at scale.
2. Using generic tools that produce false positives — wasting your time.
3. Patching together multiple command-line utilities — fragile and hard to maintain.

The real pain is the manual effort gap. Each new service or vulnerability you discover requires repeatable, automatic detection. Without custom NSE scripts, you simply cannot keep up with the pace of a real-world assessment.

Core concept / mental model

At its heart, NSE (Nmap Scripting Engine) is a simple idea: scripts run during an Nmap scan and can send probes, parse responses, and report findings in a structured way. The core mental model is that of a script as a mini-program that hooks into Nmap's scan lifecycle. There's no magic — just Lua code that Nmap executes against target hosts and ports.

Think of Nmap as a factory floor. The scan engine handles the heavy machinery: host discovery, port scanning, and service detection. Scripts are the quality-control inspectors you place at specific stations. Each script runs only where you tell it to, checks what you want checked, and reports back in a consistent format.

Here's the vocabulary you need to understand before writing your first script:

• Rule function: Every NSE script defines one of three rule functions — prerule, hostrule, or portrule. This function tells Nmap when the script should run. For most automation tasks, you'll use portrule to run only on open ports that match certain criteria.
• Action function: This is the function that does the work. Nmap calls it after the rule function returns true. The action function receives an host table and, for port rules, a port table containing information about the target.
• Shortport module: A built-in NSE library that gives you convenient helpers, like shortport.port_or_service, to define scanning filters.
• stdnse module: Another library for output formatting and string helpers, so your script prints clean, predictable results.

Here's a visual of the execution flow — a diagram in words:

Nmap scan starts
   │
   ▼
Port scan discovers open port 8080
   │
   ▼
NSE checks all scripts against port 8080
   │
   ▼
Your portrule returns TRUE for 8080
   │
   ▼
NSE calls your action function
   │
   ▼
Script sends HTTP probe, checks response, prints result

The key insight is that you're extending Nmap's core functionality without rewriting the scanner itself. You write small, focused scripts that run exactly when needed. That makes automation scalable, repeatable, and shareable.

How it works step by step

Writing a custom Nmap script is a straightforward process if you follow these steps. We'll break down each one so you see the entire pipeline from idea to execution.

Step 1: Understand the Script Structure

Every NSE script is a Lua file with a consistent structure. At a minimum, it must have:

1. A description — explaining what the script does.
2. A rule function — either prerule, hostrule, or portrule.
3. An action function — performing the actual work.

Beyond that, you can add categories (e.g., safe, intrusive, discovery) so Nmap can filter scripts by type. You can also use libraries like http, stdnse, and shortport to extend functionality.

Step 2: Choose Your Rule Function

The rule function decides when your script runs. Here's a quick comparison:

• prerule — runs exactly once, before any port scans. Great for host-level checks.
• hostrule — runs once per scanned host, regardless of open ports. Useful for broadcasts or host-level config checks.
• portrule — runs once per open port that matches the rule. Most common for service-specific checks.

For automating a service check, portrule is your best friend.

Step 3: Build the Action Function

Inside action, you can access the host and port tables. The port table includes the protocol, number, and service name. You can send custom probes using Nmap's stdnse or specific protocol libraries. Then you parse the response and return a table (which Nmap formats nicely) or a string.

Step 4: Place and Test Your Script

Scripts live in the Nmap scripts folder or in a custom directory. Use --script to call them. The standard location is /usr/share/nmap/scripts/ on Linux.

Step 5: Automate Multiple Hosts

Once your script works on one host, you can use it against entire subnets, building automation loops without any external scripting.

Hands-on walkthrough

Let's write a custom NSE script for automation that checks if a web server exposes a dangerous HTTP header (X-Powered-By). This is a common reconnaissance check for identifying technology stacks.

First, create the script file. Save it as http-powered-by.nse in a directory you'll use for custom scripts (or directly in the Nmap scripts folder if you have write permissions).

-- http-powered-by.nse
-- Detects if the web server leaks technology info via X-Powered-By header

description = [[
Checks if the HTTP server sends an X-Powered-By header, which reveals the technology stack.
]]

categories = {"safe", "discovery"}

-- Import required libraries
local shortport = require "shortport"
local http = require "http"
local stdnse = require "stdnse"

-- Rule: run for any open HTTP/HTTPS port
portrule = shortport.port_or_service({80, 8080, 443, 8443}, {"http", "https"})

-- Action: perform the HTTP request and check headers
local function action(host, port)
  local response = http.get(host, port, "/")
  if response and response.header and response.header["x-powered-by"] then
    return string.format("X-Powered-By header found: %s", response.header["x-powered-by"])
  else
    return "No X-Powered-By header present"
  end
end

Now, run the script against a target. For this example, we'll use scanme.nmap.org, but note that it may not always be available. In a real scenario, use a target you own or have permission to scan.

nmap --script http-powered-by.nse -p 80 scanme.nmap.org

Expected output (simplified):

Starting Nmap 7.94 ( https://nmap.org ) at 2025-01-01 12:00 UTC
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.060s latency).

PORT   STATE SERVICE
80/tcp open  http
| http-powered-by: No X-Powered-By header present

Nmap done: 1 IP address (1 host up) scanned in 1.23 seconds

But what if the header is there? Here's a variant that logs the result to a file for automation:

-- To demonstrate logging, we modify the action to output only actionable findings
local function action(host, port)
  local response = http.get(host, port, "/")
  if response and response.header and response.header["x-powered-by"] then
    stdnse.log("VULNERABILITY: %s revealed", response.header["x-powered-by"])
    return true
  end
  return false
end

You can combine this with Nmap's -oN output flag to save results for later analysis. For example:

nmap --script http-powered-by.nse -p 80 -oN scan_results.txt 192.168.1.0/24

This will run the script against every reachable host in the subnet and write a normal-format report to a file — pure automation for your recon phase.

For a slightly more advanced example, here's a script that sends a custom TCP probe to a non-standard service, mimicking what you'd need for a proprietary protocol. We'll check if port 4444 responds to a unique banner string:

-- custom-banner.nse
-- Custom probe for a service on port 4444
description = [[
Sends a special query to port 4444 and looks for a unique banner.
]]

categories = {"safe", "discovery"}

local shortport = require "shortport"
local stdnse = require "stdnse"
local nmap = require "nmap"

portrule = function(host, port)
  return port.number == 4444 and port.protocol == "tcp"
end

action = function(host, port)
  local socket = nmap.new_socket()
  socket:connect(host, port)
  socket:send("PROBE\n")
  local status, result = socket:receive_lines(1)
  socket:close()
  if status and result and result:match("CUSTOMSVC") then
    return "Matched custom service banner"
  end
  return "No match or inaccessible"
end

Pro tip: Always test your script against a service you control first. Use nc -l -p 4444 locally to simulate the service, then run the script against localhost. This avoids scanning unintended targets and helps you debug your script in isolation.

Compare options / when to choose what

You have several ways to write custom Nmap scripts for automation. The choice depends on your use case:

For pure port scanning and service detection automation, NSE is the most efficient. If you need to integrate scan results with a larger Python-based security pipeline, python-nmap is handy, but you'll lose some of Nmap's native parallelism.

Troubleshooting & edge cases

Here are common problems you'll run into while writing custom Nmap scripts and how to fix them.

• My script never runs. Why?

• Double-check your rule function. A portrule that doesn't match any open port will never execute. Use nmap --script-help your-script.nse to see if Nmap recognizes it, and add a debug print to the rule to confirm it's being evaluated.

• Script errors with "unexpected symbol"

• This usually means a Lua syntax error. Ensure you're missing a local or have mismatched ends. Run luac -p your-script.nse to check syntax without running it.

• HTTP responses are empty or time out.

• Some servers may need a proper User-Agent or HTTP/1.1. The http library usually handles this, but if your custom socket code has issues, use the http library instead of raw sockets for HTTP-related tasks.

• Output shows nil values

• In Lua, indexing a table that doesn't exist returns nil. To avoid errors, always check if response.header then before accessing headers.

• Script runs slower than expected

• NSE scripts run in parallel, but if you're doing multiple sequential requests, use nmap.registry to cache results or limit the scope of interactions. Too many scripts on a single port can cause timeouts.

What you learned & what's next

You now know how to write custom Nmap scripts for automation on a fundamental level. You can structure an NSE script with rule and action functions, use built-in libraries to query HTTP services, send custom TCP probes, and integrate your scripts into automated scans across entire networks. You also know how to compare NSE with other automation approaches and troubleshoot common pitfalls.

You've achieved the key learning objectives: explaining the core concept behind custom Nmap scripts and completing a hands-on exercise that automates a service check.

Now that you can extend Nmap's capabilities, your next lesson is about integrating Nmap scans with Python for post-processing and vulnerability correlation. You'll learn to parse Nmap output programmatically and feed it into decision-making engines, taking your automation skills to the next level.