OSINT Frameworks: theHarvester, Recon-ng, SpiderFoot & Maltego

This is Chapter 6 of the Recon & OSINT notebook. The previous five chapters taught reconnaissance one primitive at a time: WHOIS and DNS in Chapter 2, sock-puppet discipline in Chapter 3, search-engine dorking in Chapter 4, and subdomain enumeration pipelines in Chapter 5. Each of those chapters ended with the same practical problem — you now have five terminal windows, three text files, a browser with forty tabs, and no single place where the data lives. This chapter fixes that. An OSINT framework is the tool that owns your recon data: it runs the collectors for you, normalises what they return, stores it, and lets you pivot from one finding to the next without re-typing anything.

Four frameworks matter in practice. theHarvester is the fast one-shot harvester you run in the first ninety seconds of an engagement. Recon-ng is a Metasploit-shaped modular framework with a SQLite database underneath, and it is what you use when recon needs to be repeatable and reportable. SpiderFoot is the automation engine — you give it a seed and walk away, and it returns a correlated graph of everything reachable from that seed. Maltego is the link-analysis surface, where a human analyst looks at a graph of entities and sees the relationship that no script would have flagged.

Everything in this chapter assumes authorised work: a scoped penetration test, a bug-bounty programme inside policy, a CTF, or infrastructure you own. Most of what these frameworks do is passive — they query third-party datasets rather than the target — but "passive" is not the same as "unlimited", and several modules (DNS brute-force, virtual-host probing, port checks, screenshotting) cross into active traffic against the target. Those boundaries are flagged explicitly as they come up.

Part 1: What an OSINT Framework Actually Is

A single-purpose recon tool takes one input and produces one kind of output. subfinder -d example.com takes a domain and gives you hostnames. whois example.com takes a domain and gives you registration data. These tools are excellent and you should keep using them — Chapter 5 built an entire pipeline out of them. But they share three structural weaknesses:

1. No memory. Every run starts from zero. If you ran subfinder on Monday and amass on Tuesday, nothing merged the two result sets except you and a sort -u.
2. No normalisation. One tool emits 1.2.3.4, another emits 1.2.3.4:443, a third emits https://host/ -> 1.2.3.4. Every join is a hand-written awk expression.
3. No pivoting. The most valuable recon move is the second hop: you found an email address, so now find every other domain that address registered; you found an IP, so now find every other hostname on that IP. Single-purpose tools cannot do the second hop because they never stored the first one.

A framework solves all three by introducing a shared data model. The vocabulary differs between tools but the concept is identical everywhere:

The last row is the one that decides which tool you reach for. These four are not competitors so much as four different answers to "who is driving?" — you (theHarvester), you with a database (Recon-ng), the machine (SpiderFoot), or your eyes (Maltego).

Security relevance: this same data model is what a mature attack-surface-management product sells you. Understanding it at the framework level means that when a vendor tells your organisation they "continuously discover shadow IT", you know exactly which public datasets they are querying and can reason about what they will and will not find.

Part 2: The Recon Graph — Entities, Transforms and Pivots

Think of everything you collect as a node in a graph, and every collector as an edge-producing function. This is not an abstraction for its own sake; it is literally how SpiderFoot and Maltego are implemented, and it is the mental model that makes Recon-ng's module naming (recon/domains-hosts/..., recon/hosts-ports/...) instantly readable.

flowchart TD
    D["Domain: example.com"] --> H["Hostnames: www, api, vpn, dev"]
    D --> W["WHOIS / registrant contact"]
    D --> C["TLS certificates (CT logs)"]
    H --> IP["IPv4 / IPv6 addresses"]
    IP --> N["Netblock / ASN / hosting provider"]
    N --> D2["Other domains in the same netblock"]
    W --> E["Email addresses"]
    E --> P["People / employee names"]
    P --> S["Social profiles, usernames"]
    E --> B["Breach appearances"]
    C --> H
    D2 --> D
    IP --> PT["Open ports / banners"]
    H --> WEB["Web tech, headers, JS files"]
    WEB --> K["Leaked keys, internal endpoints"]

Read that diagram as a set of transforms. Domain → hostnames is a transform. Hostname → IP is a transform. IP → netblock → other domains is the pivot that turns one target into a whole estate. The loop from D2 back to D is the important one: recon is not a pipeline, it is a fixed-point computation. You keep applying transforms until no new entities appear. SpiderFoot automates exactly this loop; Maltego makes you do it by hand but lets you see it; Recon-ng gives you the tables and expects you to choose the next module.

Recon-ng's module namespace encodes the transform signature directly:

recon / domains-hosts / hackertarget
  │        │      │       └── the data source doing the work
  │        │      └───────── OUTPUT table (hosts)
  │        └──────────────── INPUT table (domains)
  └───────────────────────── module family

So recon/hosts-ports/shodan_ip means "read the hosts table, write the ports table, using Shodan." Once you internalise <input>-<output>/<source>, you can navigate hundreds of modules without documentation.

A worked pivot, in words. You are testing example.com. Certificate Transparency gives you legacy-vpn.example.com. Resolution gives 203.0.113.44. A reverse-IP lookup on that address returns example-partners.net — a domain nobody put in scope but which shares infrastructure. That is a scope question you must raise with the client, and it is also the single most common way a bug-bounty hunter finds an asset that fifty other hunters missed. The frameworks exist to make that three-hop chain take four seconds instead of forty minutes.

Scope discipline. The graph does not know your rules of engagement. Frameworks will happily walk from an in-scope domain to a third party's infrastructure. Discovering that a related asset exists is passive and generally fine; testing it is not. Chapter 2 of the Pentest Methodology notebook covers the authorisation boundary — re-read it before you act on a pivot that leaves the scoped domain.

Part 3: API Keys — The Real Gating Factor

Beginners assume the difference between a mediocre recon run and an excellent one is tool choice. It is not. It is API keys. Every one of these frameworks is mostly a polite HTTP client wrapped around other people's datasets, and the depth of your results is a direct function of which of those datasets will talk to you.

Two rules follow from this table.

Rule one: use a dedicated identity for every key. Chapter 3 covered sock-puppet discipline. API keys are the highest-risk place to break it, because the key ties every query you make to an account, and providers log queries. Register research accounts on a separate email, on infrastructure not linked to your primary identity, and never mix client engagements across one key if the provider exposes query history to anyone.

Rule two: keys are configuration, not secrets in your notes. Each framework has a proper place to store them, and none of that place is a shell history entry:

# Recon-ng — stored in ~/.recon-ng/keys.db
[recon-ng][default] > keys add shodan_api AAAABBBBCCCCDDDD
[recon-ng][default] > keys list
 
# theHarvester — YAML file, no keys on the command line
$ cat ~/.theHarvester/api-keys.yaml
apikeys:
  bevigil:
    key:
  binaryedge:
    key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  bufferoverun:
    key:
  github:
    key: ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  hunter:
    key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  intelx:
    key:
  shodan:
    key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
# SpiderFoot — Settings tab in the web UI, or a config file
$ python3 ./sf.py -l 127.0.0.1:5001    # then Settings → per-module API key fields

OPSEC note: several providers publish or sell aggregate query telemetry, and a few (Shodan among them) let account owners see who has queried their own netblocks via enterprise monitoring. Querying a target's infrastructure through a keyed account is passive with respect to the target's own logs but not anonymous with respect to the data provider. For red-team work where attribution matters, that distinction is the whole game.

Part 4: theHarvester From Scratch

What it is. theHarvester is an OSINT gathering tool written in Python, originally by Christian Martorella, that queries a long list of public sources for a single domain and returns emails, hostnames, IPs, ASNs, and sometimes employee names. It is the fastest way to go from "I know the domain" to "I have something to look at."

Why it exists. Before it, the first five minutes of an engagement were manual: Google, Bing, PGP key servers, LinkedIn, one at a time. theHarvester parallelises that into one command.

What it is not. It is not a database, it has no concept of a workspace, and it will not pivot for you. It is a one-shot collector. Treat its output as input to the other three frameworks.

4.1 Install

theHarvester ships in Kali:

$ sudo apt update && sudo apt install -y theharvester
$ theHarvester --version

For the current upstream version — worth having, because sources break and get fixed constantly — install from source into a virtual environment:

$ git clone https://github.com/laramies/theHarvester.git
$ cd theHarvester
$ python3 -m venv .venv && source .venv/bin/activate
(.venv)$ python3 -m pip install -r requirements/base.txt
(.venv)$ python3 theHarvester.py --version

The virtualenv matters. theHarvester pins fairly aggressive dependency versions and installing it system-wide with pip is a reliable way to break other Python tooling on the same box.

4.2 The flags, all of them

$ theHarvester -h

The distinction that matters for scope: -b sources are passive, -r/-n/-c/-t/--screenshot are active. Never run the second group against a target you have not been authorised to touch.

4.3 Listing and choosing sources

$ theHarvester -d example.com -b nonsense
[!] Invalid source. Available sources:
    anubis, baidu, bevigil, binaryedge, bing, bingapi, brave, bufferoverun,
    censys, certspotter, criminalip, crtsh, dehashed, dnsdumpster,
    duckduckgo, fullhunt, github-code, hackertarget, hunter, hunterhow,
    intelx, netlas, onyphe, otx, pentesttools, projectdiscovery, rapiddns,
    rocketreach, securityTrails, sitedossier, subdomaincenter, subdomainfinderc99,
    threatminer, tomba, urlscan, virustotal, yahoo, zoomeye

Source selection strategy, in practice:

• No keys at all: -b crtsh,hackertarget,rapiddns,otx,anubis,duckduckgo,bing,threatminer,urlscan,dnsdumpster. This is a genuinely strong passive set and costs nothing.
• Email hunting: -b hunter,bing,duckduckgo,intelx,github-code.
• Everything you have keys for: -b all — but expect several sources to time out or rate-limit, and expect the run to take minutes rather than seconds.

-b all is not the flex it looks like. Broken sources emit noisy tracebacks that bury real results, and one hard-failing source can stall the run. In an engagement, name your sources explicitly so the run is reproducible.

4.4 First real run

$ theHarvester -d example.com -b crtsh,hackertarget,otx,rapiddns,bing -l 1000 -f harvest-example

Annotated output:

*******************************************************************
*  _   _                                            _             *
* | |_| |__   ___    /\  /\__ _ _ ____   _____  ___| |_ ___ _ __  *
* | __|  _ \ / _ \  / /_/ / _` | '__\ \ / / _ \/ __| __/ _ \ '__| *
* | |_| | | |  __/ / __  / (_| | |   \ V /  __/\__ \ ||  __/ |    *
*  \__|_| |_|\___| \/ /_/ \__,_|_|    \_/ \___||___/\__\___|_|    *
*                                                                 *
* theHarvester 4.6.0                                              *
*******************************************************************
 
[*] Target: example.com
 
[*] Searching Crtsh.
[*] Searching Hackertarget.
[*] Searching Otx.
[*] Searching Rapiddns.
[*] Searching Bing.
 
[*] ASNS found: 1
--------------------
AS15133
 
[*] Interesting Urls found: 3
--------------------
https://example.com/wp-content/uploads/2019/staff-list.pdf
https://careers.example.com/apply
https://status.example.com/history
 
[*] Emails found: 6
----------------------
careers@example.com
info@example.com
j.rivera@example.com
noc@example.com
security@example.com
webmaster@example.com
 
[*] Hosts found: 41
---------------------
api.example.com:203.0.113.10
assets.example.com:198.51.100.7
autodiscover.example.com
careers.example.com:203.0.113.31
cpanel.example.com
dev-old.example.com:203.0.113.44
git.example.com:203.0.113.44
jenkins-legacy.example.com
mail.example.com:198.51.100.25
...
vpn.example.com:203.0.113.44
 
[*] Saved JSON file: harvest-example.json
[*] Saved XML file: harvest-example.xml

Three things in that output are worth an experienced eye:

• security@example.com — a security contact. On a bug-bounty engagement, that plus /.well-known/security.txt tells you where to report and often tells you the programme's maturity.
• dev-old.example.com, jenkins-legacy.example.com, git.example.com — the naming convention screams neglected infrastructure, and git/dev-old/vpn all share 203.0.113.44. One box, three services, at least one of them forgotten.
• staff-list.pdf in wp-content/uploads — document metadata is a whole OSINT sub-discipline, and a staff list in a public upload directory is an immediate email-pattern and phishing-pretext source.

Bug-bounty note: the fastest real-world use of theHarvester is exactly this — one command, scan the host list for anything with old, legacy, dev, staging, test, uat, bak, or a year in the name, and start there. Those hosts are outside the change-management process that hardened www.

4.5 Reading the JSON, not the terminal

Never parse the pretty output. The JSON file is the contract:

$ jq -r '.hosts[]' harvest-example.json | sed 's/:.*//' | sort -u > hosts.txt
$ jq -r '.emails[]' harvest-example.json | sort -u > emails.txt
$ jq -r '.ips[]' harvest-example.json | sort -u > ips.txt
$ wc -l hosts.txt emails.txt ips.txt

That hosts.txt is the exact input format Chapter 5's dnsx / httpx pipeline expects, and the exact format Recon-ng's hosts table imports. theHarvester's job is done the moment those three files exist.

Part 5: theHarvester in Anger — Chaining, Proxies and Pitfalls

5.1 Adding the active flags, deliberately

Once scope is confirmed, the active flags earn their keep:

$ theHarvester -d example.com \
    -b crtsh,otx,rapiddns,hackertarget \
    -l 1000 \
    -r \
    -e 1.1.1.1 \
    -t \
    -f harvest-example-active

• -r resolves everything found, so you get the host→IP mapping in one pass instead of piping to dnsx.
• -e 1.1.1.1 sends those resolutions to Cloudflare rather than whatever resolver DHCP handed you. This matters for two reasons: your ISP resolver may return NXDOMAIN-hijacked results, and your resolution pattern is visible to whoever runs your default resolver.
• -t performs subdomain-takeover fingerprinting — it fetches each host and matches the response body against known "no such bucket / app not found" signatures. This is HTTP traffic to the target's DNS-pointed third-party services, so it is active. Chapter 5 Part 12 covers what to do with a positive.

5.2 Proxying

$ cat ~/.theHarvester/proxies.yaml
http:
  - 127.0.0.1:8080
  - 127.0.0.1:9050
$ theHarvester -d example.com -b bing,duckduckgo -p -f harvest-proxied

Two legitimate uses: routing through Burp on 8080 so you can inspect exactly what the tool sends (excellent for learning, and for debugging a source that returns nothing), and routing through Tor on 9050 when you want search-engine queries not to correlate to your egress IP. Be aware that most commercial data-source APIs block Tor exit nodes outright, so proxying via Tor generally works for scraped sources and fails for keyed ones.

5.3 Pitfalls

The historic-record pitfall is the big one. Passive sources are archives. crtsh will tell you about a certificate issued four years ago for a host that was decommissioned three years ago. Reporting that host as live attack surface is a false positive that costs you credibility on a pentest and gets your bug-bounty report closed as informative. Resolve first, always.

Part 6: Recon-ng From Scratch

What it is. Recon-ng is a full reconnaissance framework written in Python by Tim Tomes, deliberately modelled on the Metasploit console: a modular architecture, an interactive shell with tab completion, per-engagement workspaces, a real database, and separate reporting modules. If you know msfconsole, you already know 80% of Recon-ng's ergonomics.

Why it exists. Because recon output needs to persist and join. Once your hosts, contacts, ports, credentials and vulnerabilities live in tables in one SQLite file, "which employee emails belong to hosts in the netblock we found" becomes a SQL query rather than an afternoon.

The v5 breaking change. Recon-ng v4 shipped every module bundled. v5 (and later) moved to a marketplace: the base install has almost no recon modules, and you install what you need. Every tutorial older than that will give you commands that no longer exist (use became modules load, show modules became modules search). If a command errors, that is almost always why.

6.1 Install and first launch

$ sudo apt install -y recon-ng          # Kali
# or from source:
$ git clone https://github.com/lanmaster53/recon-ng.git
$ cd recon-ng && python3 -m pip install -r REQUIREMENTS
$ ./recon-ng

    _/_/_/    _/_/_/_/    _/_/_/    _/_/    _/      _/            _/      _/    _/_/_/
   _/    _/  _/        _/        _/    _/  _/_/    _/            _/_/    _/  _/
  _/_/_/    _/_/_/    _/        _/    _/  _/  _/  _/  _/_/_/_/  _/  _/  _/  _/  _/_/
 _/    _/  _/        _/        _/    _/  _/    _/_/            _/    _/_/  _/    _/
_/    _/  _/_/_/_/    _/_/_/    _/_/    _/      _/            _/      _/    _/_/_/
 
                          [recon-ng v5.1.2, Tim Tomes]
 
[*] No modules enabled/installed.
 
[recon-ng][default] >

That "No modules enabled/installed" line is not an error. It is the marketplace design.

6.2 The command grammar

Recon-ng's commands are noun-first and consistent:

spool and script are the two most underrated. spool start ~/engagements/acme/reconng.log gives you a complete evidence trail of every command and every result, which is what a report annexe needs. script record lets you build a repeatable recon playbook you can re-run against the next target.

6.3 The database schema

[recon-ng][acme] > db schema
 
+---------------------+
|      domains        |
+---------------------+
| domain    | TEXT    |
| notes     | TEXT    |
| module    | TEXT    |
+---------------------+
+---------------------+
|      companies      |
+---------------------+
| company   | TEXT    |
| description| TEXT   |
+---------------------+
+---------------------+
|        hosts        |
+---------------------+
| host      | TEXT    |
| ip_address| TEXT    |
| region    | TEXT    |
| country   | TEXT    |
| latitude  | TEXT    |
| longitude | TEXT    |
| notes     | TEXT    |
+---------------------+
+---------------------+
|      contacts       |
+---------------------+
| first_name| TEXT    |
| middle_name| TEXT   |
| last_name | TEXT    |
| email     | TEXT    |
| title     | TEXT    |
| region    | TEXT    |
| country   | TEXT    |
+---------------------+
+---------------------+
|       ports         |
+---------------------+
| ip_address| TEXT    |
| host      | TEXT    |
| port      | TEXT    |
| protocol  | TEXT    |
+---------------------+
+---------------------+
|    credentials      |
+---------------------+
| username  | TEXT    |
| password  | TEXT    |
| hash      | TEXT    |
| type      | TEXT    |
| leak      | TEXT    |
+---------------------+

Plus netblocks, locations, vulnerabilities, pushpins, profiles, repositories and leaks. Every module declares which of these it reads and which it writes — which is exactly the <input>-<output> naming from Part 2.

6.4 Workspaces

One workspace per engagement, always:

[recon-ng][default] > workspaces create acme
[recon-ng][acme] > workspaces list
 
  +----------------------------------------------------+
  | Workspaces |              Modified                 |
  +----------------------------------------------------+
  | default    | 2 minutes ago                         |
  | acme       | 1 second ago                          |
  +----------------------------------------------------+

The workspace is a directory under ~/.recon-ng/workspaces/<name>/ containing data.db. That single file is the engagement's recon data — back it up, and delete it under your data-retention policy when the engagement closes. Mixing two clients in one workspace is a contract-breach waiting to happen.

Part 7: Recon-ng Workflow — Marketplace to Report

7.1 Installing modules

[recon-ng][acme] > marketplace search domains-hosts
 
  +---------------------------------------------------------------------------------+
  |                    Path                    | Version |     Status    | D | K |
  +---------------------------------------------------------------------------------+
  | recon/domains-hosts/binaryedge             |  1.1    | not installed | * | * |
  | recon/domains-hosts/bing_domain_api        |  1.0    | not installed |   | * |
  | recon/domains-hosts/bing_domain_web        |  1.1    | not installed |   |   |
  | recon/domains-hosts/brute_hosts            |  1.0    | not installed |   |   |
  | recon/domains-hosts/builtwith              |  1.0    | not installed |   | * |
  | recon/domains-hosts/certificate_transparency| 1.2    | not installed |   |   |
  | recon/domains-hosts/hackertarget           |  1.1    | not installed |   |   |
  | recon/domains-hosts/mx_spf_ip              |  1.0    | not installed |   |   |
  | recon/domains-hosts/shodan_hostname        |  1.1    | not installed | * | * |
  | recon/domains-hosts/threatcrowd            |  1.0    | not installed |   |   |
  | recon/domains-hosts/threatminer            |  1.0    | not installed |   |   |
  +---------------------------------------------------------------------------------+
 
  D = Has dependencies. See info for details.
  K = Requires keys. See info for details.

The D and K columns are the whole reason to read this table before installing. K means the module is inert without an API key.

[recon-ng][acme] > marketplace info recon/domains-hosts/certificate_transparency
 
  +---------------------------------------------------------------------------+
  | path        | recon/domains-hosts/certificate_transparency                 |
  | name        | Certificate Transparency Search                             |
  | author      | Andres Buitrago                                             |
  | version     | 1.2                                                         |
  | last_updated| 2021-08-24                                                  |
  | description | Retrieves domains from certificate transparency logs        |
  |             | (crt.sh). Updates the 'hosts' table.                        |
  | required_keys| []                                                         |
  | dependencies| []                                                          |
  +---------------------------------------------------------------------------+

Install a curated keyless set — this is the "works anywhere, costs nothing" baseline:

[recon-ng][acme] > marketplace install recon/domains-hosts/certificate_transparency
[recon-ng][acme] > marketplace install recon/domains-hosts/hackertarget
[recon-ng][acme] > marketplace install recon/domains-hosts/threatminer
[recon-ng][acme] > marketplace install recon/domains-hosts/mx_spf_ip
[recon-ng][acme] > marketplace install recon/hosts-hosts/resolve
[recon-ng][acme] > marketplace install recon/hosts-hosts/reverse_resolve
[recon-ng][acme] > marketplace install recon/hosts-ports/shodan_ip
[recon-ng][acme] > marketplace install recon/domains-contacts/whois_pocs
[recon-ng][acme] > marketplace install recon/domains-contacts/hunter_io
[recon-ng][acme] > marketplace install recon/contacts-contacts/mangle
[recon-ng][acme] > marketplace install recon/contacts-credentials/hibp_breach
[recon-ng][acme] > marketplace install recon/netblocks-hosts/virustotal
[recon-ng][acme] > marketplace install discovery/info_disclosure/interesting_files
[recon-ng][acme] > marketplace install reporting/html
[recon-ng][acme] > marketplace install reporting/csv
[recon-ng][acme] > marketplace install reporting/list

marketplace install all exists and is tempting. Resist it: it pulls hundreds of modules, many long-dead, and your modules search results become unusable.

7.2 Seeding and running

[recon-ng][acme] > db insert domains
domain (TEXT): example.com
notes (TEXT):
[*] 1 rows affected.
 
[recon-ng][acme] > modules load recon/domains-hosts/certificate_transparency
[recon-ng][acme][certificate_transparency] > info
 
  Name: Certificate Transparency Search
  Path: modules/recon/domains-hosts/certificate_transparency.py
  Author: Andres Buitrago
 
  Options:
    Name    Current Value  Required  Description
    ------  -------------  --------  -----------
    SOURCE  default        yes       source of input (see 'info' for details)
 
  Source Options:
    default        SELECT DISTINCT domain FROM domains WHERE domain IS NOT NULL
    <string>       string representing a single input
    <path>         path to a file containing a list of inputs
    query <sql>    database query returning one column of inputs
 
[recon-ng][acme][certificate_transparency] > run
 
-------------
EXAMPLE.COM
-------------
[*] URL: https://crt.sh/?q=%25.example.com&output=json
[*] [host] api.example.com (<blank>)
[*] [host] assets.example.com (<blank>)
[*] [host] dev-old.example.com (<blank>)
[*] [host] git.example.com (<blank>)
[*] [host] jenkins-legacy.example.com (<blank>)
[*] [host] vpn.example.com (<blank>)
...
-------
SUMMARY
-------
[*] 41 total (41 new) hosts found.

The SOURCE option is Recon-ng's superpower and the thing most people never touch. default runs a built-in SQL query. But you can hand it any SQL:

[recon-ng][acme][shodan_ip] > options set SOURCE query SELECT DISTINCT ip_address FROM hosts WHERE ip_address IS NOT NULL AND host LIKE '%vpn%'

That is a targeted second hop expressed declaratively. It is the difference between using Recon-ng as a tool and using it as a framework.

7.3 The full chain

# 1. domains -> hosts (three independent sources, results merge and dedupe)
[recon-ng][acme] > modules load recon/domains-hosts/hackertarget
[recon-ng][acme][hackertarget] > run
[recon-ng][acme][hackertarget] > back
[recon-ng][acme] > modules load recon/domains-hosts/threatminer
[recon-ng][acme][threatminer] > run
[recon-ng][acme][threatminer] > back
 
# 2. hosts -> IPs
[recon-ng][acme] > modules load recon/hosts-hosts/resolve
[recon-ng][acme][resolve] > run
 
# 3. IPs -> more hostnames (reverse DNS pivot)
[recon-ng][acme] > modules load recon/hosts-hosts/reverse_resolve
[recon-ng][acme][reverse_resolve] > run
 
# 4. IPs -> open ports (Shodan, no packets to target)
[recon-ng][acme] > modules load recon/hosts-ports/shodan_ip
[recon-ng][acme][shodan_ip] > options set SOURCE query SELECT DISTINCT ip_address FROM hosts WHERE ip_address IS NOT NULL
[recon-ng][acme][shodan_ip] > run
 
# 5. domains -> people
[recon-ng][acme] > modules load recon/domains-contacts/whois_pocs
[recon-ng][acme][whois_pocs] > run
[recon-ng][acme] > modules load recon/domains-contacts/hunter_io
[recon-ng][acme][hunter_io] > run
 
# 6. people -> more candidate emails via naming-convention mangling
[recon-ng][acme] > modules load recon/contacts-contacts/mangle
[recon-ng][acme][mangle] > options set PATTERN <fi>.<last>@example.com
[recon-ng][acme][mangle] > run
 
# 7. emails -> breach appearances
[recon-ng][acme] > modules load recon/contacts-credentials/hibp_breach
[recon-ng][acme][hibp_breach] > run

Step 6 deserves a note. The mangle module takes real names you found and generates addresses in the organisation's observed pattern. If j.rivera@example.com is real, the pattern is <fi>.<last>@, and every name you scraped from LinkedIn becomes a candidate address. That is the mechanical core of every credential-stuffing and phishing pretext operation, which is why it belongs in the report: "the organisation's email convention is trivially derivable, and 340 employee names are publicly enumerable."

7.4 Querying and reporting

[recon-ng][acme] > db query SELECT host, ip_address FROM hosts WHERE ip_address = '203.0.113.44'
 
  +--------------------------------------+
  |         host        |  ip_address    |
  +--------------------------------------+
  | dev-old.example.com | 203.0.113.44   |
  | git.example.com     | 203.0.113.44   |
  | vpn.example.com     | 203.0.113.44   |
  +--------------------------------------+
 
[recon-ng][acme] > db query SELECT h.host, p.port FROM hosts h JOIN ports p ON h.ip_address = p.ip_address ORDER BY p.port
 
[recon-ng][acme] > db query SELECT count(*) FROM hosts
[recon-ng][acme] > dashboard

Then export:

[recon-ng][acme] > modules load reporting/html
[recon-ng][acme][html] > options set CREATOR "Recon Team"
[recon-ng][acme][html] > options set CUSTOMER "Acme Corp"
[recon-ng][acme][html] > options set FILENAME /home/kali/engagements/acme/recon.html
[recon-ng][acme][html] > run
[*] Report generated at '/home/kali/engagements/acme/recon.html'.
 
[recon-ng][acme] > modules load reporting/list
[recon-ng][acme][list] > options set TABLE hosts
[recon-ng][acme][list] > options set COLUMN host
[recon-ng][acme][list] > options set FILENAME /home/kali/engagements/acme/hosts.txt
[recon-ng][acme][list] > run

reporting/list is how Recon-ng hands off to the rest of your toolchain — a bare newline-separated list of hosts, ready for httpx, nuclei, or Burp's scope configuration.

7.5 Non-interactive Recon-ng

For scheduled or scripted recon:

$ cat > /tmp/acme.rc <<'EOF'
workspaces load acme
modules load recon/domains-hosts/certificate_transparency
run
back
modules load recon/hosts-hosts/resolve
run
back
modules load reporting/list
options set TABLE hosts
options set COLUMN host
options set FILENAME /home/kali/engagements/acme/hosts.txt
run
exit
EOF
 
$ recon-ng -w acme -r /tmp/acme.rc

-w selects the workspace, -r runs a resource file. This turns Recon-ng into a cron-able attack-surface monitor — diff the current hosts.txt against the previous run's and you have a change-detection feed for free.

Part 8: SpiderFoot From Scratch

What it is. SpiderFoot is an OSINT automation engine by Steve Micallef. You give it a seed — a domain, IP, netblock, email, username, phone number, Bitcoin address — pick a use case, and it recursively runs every applicable module against every entity it discovers, until the graph closes. It has a web UI, a CLI, a REST API, and a correlation engine that flags interesting findings rather than making you read 40,000 rows.

Why it exists. Because the fixed-point computation from Part 2 is mechanical, and machines are better at it than humans. Recon-ng makes you choose the next module; SpiderFoot chooses all of them, forever.

The trade-off. SpiderFoot's completeness is also its problem: an unconstrained scan produces enormous volume, much of it low-signal, and some modules are actively noisy against the target. Controlling it is the skill.

8.1 Install and launch

$ git clone https://github.com/smicallef/spiderfoot.git
$ cd spiderfoot
$ python3 -m venv .venv && source .venv/bin/activate
(.venv)$ pip3 install -r requirements.txt
(.venv)$ python3 ./sf.py -l 127.0.0.1:5001

 Starting web server at http://127.0.0.1:5001 ...
 
*************************************************************
 Use SpiderFoot by starting your web browser of choice and
 browsing to http://127.0.0.1:5001
*************************************************************

Docker, if you prefer not to manage the Python environment:

$ docker run -p 5001:5001 --name spiderfoot ghcr.io/smicallef/spiderfoot

Bind to localhost. -l 0.0.0.0:5001 exposes an unauthenticated web application that can scan arbitrary targets from your host, to your entire network. If you must expose it, put it behind sf.py's passwd file authentication and a reverse proxy with TLS. An open SpiderFoot instance is a genuine SSRF-and-worse liability — and, on the other side of the fence, finding an exposed one during a pentest is a finding worth writing up.

8.2 The concepts

• Target/seed: what you scan. SpiderFoot auto-detects the type from the string you give it.
• Module: one collector, sfp_<name>. There are 200+.
• Event type: the typed data a module consumes and produces — DOMAIN_NAME, INTERNET_NAME, IP_ADDRESS, EMAILADDR, HUMAN_NAME, USERNAME, TCP_PORT_OPEN, VULNERABILITY_CVE_CRITICAL, LEAKSITE_URL, DARKNET_MENTION_URL, SSL_CERTIFICATE_ISSUED, MALICIOUS_IPADDR, and dozens more.
• Scan: one run over one seed.
• Correlation rule: a YAML rule that inspects a finished scan's events and raises a titled, severity-rated finding.

The event-type system is what makes the automation safe to reason about: a module declares what it consumes, so SpiderFoot only ever fires it when a matching entity appears. Nothing runs speculatively.

8.3 Use cases — the single most important setting

When you create a scan, you choose one of four modes. This choice matters more than every other option combined:

Get this wrong and you have generated unauthorised active traffic against a client. "I clicked All" is not a defence. Passive is the default you should have to consciously leave.

8.4 Running a scan in the UI

New Scan → name it after the engagement → seed example.com → By Use Case → Passive → Run Scan Now.

The scan view gives you five tabs:

• Status — per-module progress and errors. Read it: a module erroring on every call usually means a missing or rate-limited API key.
• Browse — findings grouped by event type, with counts. This is where you actually work.
• Correlations — the curated findings (see 9.2).
• Graph — a force-directed visualisation of the entity relationships.
• Scan Settings — the exact configuration used, which you need for the report's methodology section.

Typical passive-scan yield on a mid-size organisation:

Type                                    Unique  Total
--------------------------------------------------------
Internet Name                              412    1,204
Affiliate - Internet Name                  188      401
IP Address                                  63      877
Raw DNS Records                             94       94
SSL Certificate - Issued to                311      620
Email Address                               57      143
Human Name                                  88      110
Similar Domain                              41       41
Web Technology                              36      212
Junk File                                    9        9
Leak Site URL                                4        4
Account on External Site                    27       27
Hosting Provider                             6        6
BGP AS Ownership                             3        3
Physical Address                             2        2
Vulnerability - CVE Critical                 1        1

Similar Domain and Leak Site URL are the two rows an experienced analyst reads first. Similar domains are typosquats and potential phishing infrastructure — sometimes the client's own defensive registrations, sometimes not. Leak-site URLs are pastes and dumps mentioning the organisation.

Part 9: SpiderFoot Advanced — CLI, Correlations and Control

9.1 The command line

The UI is for exploring; the CLI is for engagements and automation.

# List every module
$ python3 ./sf.py -M
 
# List every event type
$ python3 ./sf.py -T
 
# Passive scan, tab-delimited to stdout
$ python3 ./sf.py -s example.com -u passive -q -o tab
 
# Only specific modules
$ python3 ./sf.py -s example.com -m sfp_crt,sfp_dnsresolve,sfp_hunter,sfp_shodan -q -o csv > sf-example.csv
 
# Only collect specific event types (everything needed to produce them still runs)
$ python3 ./sf.py -s example.com -t EMAILADDR,HUMAN_NAME,INTERNET_NAME -u passive -q -o json > sf-example.json
 
# Correlations only, from an existing scan
$ python3 ./sf.py -c <scanId>

-q -o json is the combination you want for anything that feeds another tool:

$ python3 ./sf.py -s example.com -u passive -q -o json \
  | jq -r 'select(.type=="INTERNET_NAME") | .data' | sort -u > sf-hosts.txt

9.2 Correlation rules

SpiderFoot v4 added a correlation engine, and it is the feature that converts volume into findings. Rules are YAML in correlations/, and shipped rules include things like "Hosts with an expired SSL certificate", "Open cloud storage buckets", "Email addresses found in breaches", "Hosts running vulnerable software with a critical CVE", and "Similar domains registered recently".

A rule looks like this:

id: outdated_tls_cert
version: 1
meta:
  name: Expired SSL/TLS certificate
  description: >
    A certificate presented by a host in scope has passed its expiry date.
    Expired certificates indicate unmanaged infrastructure and break clients.
  risk: MEDIUM
collections:
  - collect:
      - method: exact
        field: type
        value: SSL_CERTIFICATE_EXPIRED
headline: "Expired certificate on {entity.data}"

Writing your own is genuinely worth it. An engagement-specific rule such as "any INTERNET_NAME matching (dev|test|uat|staging|old|legacy) and having a TCP_PORT_OPEN event" encodes your own judgement about what matters and applies it to every future scan automatically. That is how you stop re-reading 40,000 rows by hand.

Blue-team usage: run SpiderFoot in investigate mode against a domain that appeared in a phishing email. The correlation output — malicious-IP hits, recent registration, similarity to your own brand, shared infrastructure with known-bad — is a triage verdict in about four minutes, and it produces the evidence trail for the ticket.

9.3 Controlling the blast radius

Three settings in Settings → Global:

• _fetchtimeout / _socks* — timeouts and proxying. Route through Tor or a VPS if attribution matters.
• _maxthreads — concurrency. Lower it for stealth, raise it for speed.
• User-Agent — the default announces SpiderFoot. Change it if you don't want that in the target's logs, but be aware that misrepresenting your scanner may conflict with a bug-bounty programme's rules; several explicitly require identifying headers.

And the single most important control: do not seed a scan with a netblock or company name unless you have verified every asset it will expand to is in scope. Seeding example.com scans one estate. Seeding AS15133 scans a CDN's entire customer base.

Part 10: Maltego From Scratch

What it is. Maltego is a desktop link-analysis application. Your data is a graph of typed entities; you run transforms (server-side functions) on selected entities to produce related entities and draw edges. Its unique value is not collection — the other three collect more, faster — it is human pattern recognition on a visual graph.

Why it exists. Some conclusions only exist in the shape of the data. Three unrelated-looking shell companies that share one registrant email; a cluster of typosquats that all resolve into one /29; an employee whose personal username reuses across GitHub, a forum, and a leaked database. You do not find those with grep. You find them by looking.

10.1 Editions and the CE limit

The CE 12-result cap is severe and it is not a bug you can work around — a domain with 400 subdomains returns 12. The practical consequence: use CE for structure and pivoting, not for collection. Collect with theHarvester/Recon-ng/SpiderFoot, import the results, and use Maltego to see the relationships.

10.2 Install and first run

$ sudo apt install -y maltego     # Kali
$ maltego

First launch walks you through: choose edition → register/login with a Maltego account → install Transform Hub items → select a start page. The Transform Hub is the equivalent of Recon-ng's marketplace: tiles for transform providers, some free, some paid, some requiring their own API key.

Free hub items worth installing immediately:

• Maltego Standard Transforms — the DNS/WHOIS/IP core. Non-negotiable.
• Have I Been Pwned (key required)
• VirusTotal Public API
• AlienVault OTX
• ThreatCrowd / ThreatMiner
• Shodan (key required)
• WhoisXML (limited free credits)
• Social Links / Hunchly / etc. — commercial, deep, expensive.

10.3 The interface

• Entity Palette (left) — drag an entity type onto the graph to seed it.
• Graph canvas (centre) — nodes and edges.
• Run View / Transform menu (right-click a node) — the transforms applicable to that entity type.
• Detail View / Property View (right) — every property of the selected entity, including which transform produced it.
• Overview (bottom-left) — minimap for large graphs.

Layouts (Organic, Block, Hierarchical, Circular) change how the graph is drawn. Organic is the default and the one where clusters become visually obvious — nodes with shared connections pull together, which is precisely the signal you're hunting.

10.4 A first graph, step by step

1. Drag a Domain entity onto the canvas. Double-click it, set the value to example.com.
2. Right-click → All Transforms → To DNS Name - SOA (Start of Authority). You get the primary nameserver and the SOA email — often a real administrator's address.
3. Right-click the domain → To DNS Name [Using Name Schema dictionary] and To Website [Quick lookup].
4. Right-click the domain → To Entities from WHOIS [IBM Watson] or the WhoisXML transforms. Registrant name, org, email, and registrar appear as separate entities.
5. Select all DNS Name entities (rubber-band select) → To IP Address [DNS]. Edges converge; shared hosting becomes visible instantly.
6. Select the IP addresses → To Netblock [Using natural boundaries] → To AS number → To Company [Owner].
7. Take the registrant email → To Domains [Reverse WHOIS]. This is the money transform: every other domain registered with that address.

sequenceDiagram
    participant A as Analyst
    participant M as Maltego Client
    participant TS as Transform Server (TDS/local)
    participant DS as Data Source (WHOIS, DNS, Shodan…)

    A->>M: Select entity "example.com", run transform
    M->>TS: POST entity + transform id + auth
    TS->>DS: Query provider API
    DS-->>TS: Raw response
    TS-->>M: Typed entities + links (max N per licence)
    M-->>A: Render new nodes on graph
    A->>M: Select 6 new IPs, run "To Netblock"
    M->>TS: Batched request (one call per entity)
    TS-->>M: Netblock entities
    A->>A: Spot that 3 "unrelated" domains share one /29

10.5 Machines

A Machine is a saved sequence of transforms that runs automatically — Maltego's answer to "I do the same eight pivots on every domain." Machines are written in a Spring-based DSL:

machine("acme.footprintL1",
        displayName:"Acme Footprint L1",
        author:"Analyst",
        description:"Domain to hosts to IPs to netblocks to ASN") {
 
  start {
    run "paterva.v2.DomainToDNSName_SOA"
    run "paterva.v2.DomainToDNSName_MX"
    run "paterva.v2.DomainToDNSName_NS"
    run "paterva.v2.DomainToWebsite_QuickLookup"
 
    paths {
      path {
        run "paterva.v2.DNSNameToIPAddress_DNS"
        run "paterva.v2.IPAddressToNetblock_Natural"
        run "paterva.v2.NetblockToASNumber"
      }
    }
  }
}

Built-in machines worth knowing: Company Stalker (domain → emails → people), Footprint L1/L2/L3 (increasingly aggressive infrastructure mapping), Person - Email Address, and URL to Network and Domain Information. Footprint L3 is genuinely aggressive; treat it like SpiderFoot's All and only run it in scope.

10.6 Getting your own data in

Because CE caps transforms, importing is how you actually use Maltego on a real engagement:

• Import → Import Graph from Table — feed it the CSV from Recon-ng's reporting/csv or SpiderFoot's -o csv, map columns to entity types, and Maltego builds the graph.
• Copy-paste — paste a newline-separated list of hostnames onto the canvas and Maltego offers to create entities from them.
• Local transforms — write your own transform in Python with the maltego-trx library and register it as a Local Transform. This lets you graph any data source, including internal ones.

# local transform: hostname -> IP, via your own resolver
from maltego_trx.entities import IPv4Address
from maltego_trx.transform import DiscoverableTransform
import socket
 
class HostToIP(DiscoverableTransform):
    @classmethod
    def create_entities(cls, request, response):
        host = request.Value
        try:
            for info in socket.getaddrinfo(host, None, socket.AF_INET):
                ip = info[4][0]
                ent = response.addEntity(IPv4Address, ip)
                ent.addProperty("source.host", "Source Host", "loose", host)
        except socket.gaierror:
            response.addUIMessage(f"No A record for {host}")

CTF/bug-bounty note: Maltego is overkill for a single-target CTF box. Where it earns its place is a wide-scope bug-bounty programme (*.example.com plus acquisitions) or an OSINT-category CTF where the challenge is explicitly about linking identities across platforms — TraceLabs-style missing-persons CTFs are built around exactly this workflow.

Part 11: Choosing the Right Framework

flowchart TD
    Q1{"What do you have?"} -->|"One domain, need something in 60s"| TH["theHarvester"]
    Q1 -->|"An engagement that needs a durable, queryable dataset"| RN["Recon-ng"]
    Q1 -->|"A seed and time, want maximum coverage hands-off"| SF["SpiderFoot"]
    Q1 -->|"Many entities, need to SEE relationships"| MT["Maltego"]

    TH --> OUT["hosts.txt / emails.txt / JSON"]
    RN --> DB["workspace SQLite + HTML report"]
    SF --> EV["typed events + correlations"]
    MT --> GR["graph (.mtgx)"]

    OUT --> MERGE["Merge + dedupe + resolve"]
    DB --> MERGE
    EV --> MERGE
    MERGE --> MT
    MERGE --> NEXT["httpx / nuclei / Burp scope"]

The honest answer for most engagements is: theHarvester for the first ninety seconds, SpiderFoot passive running in the background for the whole engagement, Recon-ng as the system of record you actually report from, and Maltego only when the finding is relational rather than enumerable. They are complements, not alternatives.

Part 12: The Complete Hands-On Lab — One Domain, Four Frameworks

Lab setup and ethics. Run this against a domain you own, a deliberately vulnerable target, or a bug-bounty programme whose policy permits automated recon. The commands below use example.com as a placeholder; substitute your authorised target. Everything in Stages 1–4 is passive except where explicitly labelled.

Stage 0: Workspace

$ export TARGET=example.com
$ export ENG=~/engagements/acme
$ mkdir -p $ENG/{raw,merged,reports}
$ cd $ENG
$ date -u +"%Y-%m-%dT%H:%M:%SZ" > raw/START.txt

That timestamp file matters more than it looks. Recon data is a snapshot; a report that says "as of collection" without saying when is not reproducible.

Stage 1: theHarvester — the fast pass

$ theHarvester -d $TARGET \
    -b crtsh,hackertarget,rapiddns,otx,threatminer,urlscan,anubis,bing,duckduckgo \
    -l 1000 \
    -f raw/th-$TARGET

[*] Target: example.com
[*] Searching Crtsh.
[*] Searching Hackertarget.
[*] Searching Rapiddns.
[*] Searching Otx.
[*] Searching Threatminer.
[*] Searching Urlscan.
[*] Searching Anubis.
[*] Searching Bing.
[*] Searching Duckduckgo.
 
[*] Emails found: 9
[*] Hosts found: 47
[*] ASNS found: 2
 
[*] Saved JSON file: raw/th-example.com.json

Extract:

$ jq -r '.hosts[]?' raw/th-$TARGET.json | cut -d: -f1 | sort -u > merged/hosts-th.txt
$ jq -r '.emails[]?' raw/th-$TARGET.json | sort -u    > merged/emails-th.txt
$ wc -l merged/hosts-th.txt merged/emails-th.txt
  47 merged/hosts-th.txt
   9 merged/emails-th.txt

Stage 2: Recon-ng — the system of record

$ cat > /tmp/acme-stage2.rc <<'EOF'
workspaces create acme
db insert domains
example.com
 
modules load recon/domains-hosts/certificate_transparency
run
back
modules load recon/domains-hosts/hackertarget
run
back
modules load recon/domains-hosts/threatminer
run
back
modules load recon/domains-hosts/mx_spf_ip
run
back
modules load recon/hosts-hosts/resolve
run
back
modules load recon/hosts-hosts/reverse_resolve
run
back
modules load recon/domains-contacts/whois_pocs
run
back
EOF
 
$ recon-ng -w acme -r /tmp/acme-stage2.rc

Import theHarvester's hosts so nothing is lost, then export a unified list:

$ recon-ng -w acme <<'EOF'
db query SELECT count(*) FROM hosts
modules load reporting/list
options set TABLE hosts
options set COLUMN host
options set FILENAME /home/kali/engagements/acme/merged/hosts-rng.txt
run
back
modules load reporting/html
options set CREATOR "Recon Team"
options set CUSTOMER "Acme Corp"
options set FILENAME /home/kali/engagements/acme/reports/recon-ng.html
run
exit
EOF

Sample of the interesting query:

[recon-ng][acme] > db query SELECT ip_address, count(*) AS n, group_concat(host, ', ') FROM hosts WHERE ip_address IS NOT NULL GROUP BY ip_address HAVING n > 2 ORDER BY n DESC
 
  +----------------------------------------------------------------------------------+
  | ip_address    | n |                    group_concat(host, ', ')                   |
  +----------------------------------------------------------------------------------+
  | 203.0.113.44  | 4 | dev-old.example.com, git.example.com, jenkins-legacy.example  |
  |               |   | .com, vpn.example.com                                         |
  | 198.51.100.7  | 3 | assets.example.com, cdn.example.com, img.example.com          |
  +----------------------------------------------------------------------------------+

203.0.113.44 is now the priority host of the engagement, and you found it with one SQL query. Four services, at least two of which (dev-old, jenkins-legacy) are named like nobody owns them, on one box, alongside the VPN.

Stage 3: SpiderFoot — the wide automated sweep

$ cd ~/tools/spiderfoot && source .venv/bin/activate
(.venv)$ python3 ./sf.py -s example.com -u passive -q -o json \
    > ~/engagements/acme/raw/sf-example.json

Pull the event types you care about:

$ cd $ENG
$ jq -r 'select(.type=="INTERNET_NAME") | .data' raw/sf-example.json | sort -u > merged/hosts-sf.txt
$ jq -r 'select(.type=="EMAILADDR")     | .data' raw/sf-example.json | sort -u > merged/emails-sf.txt
$ jq -r 'select(.type=="HUMAN_NAME")    | .data' raw/sf-example.json | sort -u > merged/people-sf.txt
$ jq -r 'select(.type=="SIMILARDOMAIN") | .data' raw/sf-example.json | sort -u > merged/typosquats.txt
$ jq -r 'select(.type|test("LEAKSITE"))  | .data' raw/sf-example.json | sort -u > merged/leaks.txt
$ jq -r '.type' raw/sf-example.json | sort | uniq -c | sort -rn | head -20

   1204 INTERNET_NAME
    877 IP_ADDRESS
    620 SSL_CERTIFICATE_ISSUED
    401 AFFILIATE_INTERNET_NAME
    212 WEBSERVER_TECHNOLOGY
    143 EMAILADDR
    110 HUMAN_NAME
     94 RAW_DNS_RECORDS
     41 SIMILARDOMAIN
     27 ACCOUNT_EXTERNAL_OWNED
      9 JUNK_FILE
      6 PROVIDER_HOSTING
      4 LEAKSITE_URL
      3 BGP_AS_OWNER
      2 PHYSICAL_ADDRESS
      1 VULNERABILITY_CVE_CRITICAL

VULNERABILITY_CVE_CRITICAL with a count of 1 is the line you jump to:

$ jq -r 'select(.type=="VULNERABILITY_CVE_CRITICAL") | [.data, .module] | @tsv' raw/sf-example.json
CVE-2021-44228    sfp_vuln_cve

That is a Shodan/passive-banner-derived claim, not a verified finding. It goes in the report as "passive data sources indicate a host presenting a banner associated with CVE-2021-44228; verification required" — never as a confirmed vulnerability. Passive intel is a lead, not evidence.

Stage 4: Merge, resolve, verify

$ cat merged/hosts-th.txt merged/hosts-rng.txt merged/hosts-sf.txt \
  | tr 'A-Z' 'a-z' | grep -E "\.example\.com$" | sort -u > merged/hosts-all.txt
$ wc -l merged/hosts-all.txt
  118 merged/hosts-all.txt
 
# Which tool contributed what — worth knowing for next time
$ comm -13 <(sort -u merged/hosts-th.txt) <(sort -u merged/hosts-all.txt) | wc -l
  71

Now resolve and probe (this is the boundary — everything below touches the target or its providers):

$ dnsx -l merged/hosts-all.txt -a -resp -silent -o merged/resolved.txt
$ wc -l merged/resolved.txt
  63 merged/resolved.txt
 
$ httpx -l <(cut -d' ' -f1 merged/resolved.txt) \
        -status-code -title -tech-detect -follow-redirects \
        -o merged/live.txt -silent

https://www.example.com          [200] [Acme Corp]                [Nginx,React]
https://api.example.com          [401] [Unauthorized]             [Kong]
https://careers.example.com      [200] [Careers - Acme]           [WordPress:6.4.2,PHP]
https://vpn.example.com          [200] [SSL VPN Service]          [Fortinet]
https://git.example.com          [200] [GitLab]                   [GitLab:14.2.1,Nginx]
https://jenkins-legacy.example.com [403] [Authentication required] [Jetty:9.4.39,Jenkins]
https://dev-old.example.com      [200] [Index of /]               [Apache:2.4.29]

118 passive hostnames → 63 that resolve → 7 that answer HTTP. That 118→63 collapse is the single most important number in this lab, and it is why Part 5's "resolve before you believe" rule exists. Reporting 118 hosts as attack surface would be wrong by a factor of nearly two.

The last three lines are the engagement:

• git.example.com running GitLab 14.2.1 — a specific, old, CVE-rich version.
• jenkins-legacy.example.com — a 403 from Jenkins/Jetty, which is an auth wall, not absence of a service.
• dev-old.example.com — Index of / means directory listing is enabled on Apache. That is a finding on its own and a doorway to more.

Stage 5: Maltego — see it

# Build a two-column CSV that Maltego can import as entity + link
$ echo "host,ip" > merged/maltego-import.csv
$ awk '{print $1","$2}' merged/resolved.txt | tr -d '[]' >> merged/maltego-import.csv
$ head -3 merged/maltego-import.csv
host,ip
www.example.com,203.0.113.10
git.example.com,203.0.113.44

In Maltego: Import → Import Graph from Table → select the CSV → map host to DNS Name, ip to IPv4 Address, and set the connection so each host links to its IP → Finish. Switch to Organic layout.

The cluster around 203.0.113.44 is now visible as a shape, not a table row. Right-click that IP → To Netblock → To AS Number → To Domains [Reverse DNS]. If any domain outside example.com appears on that netblock, you have either a scope conversation to have with the client, or a shared-hosting artefact to rule out.

Stage 6: Document

$ date -u +"%Y-%m-%dT%H:%M:%SZ" > raw/END.txt
$ tree -L 2 $ENG

/home/kali/engagements/acme
├── merged
│   ├── emails-sf.txt
│   ├── emails-th.txt
│   ├── hosts-all.txt
│   ├── hosts-rng.txt
│   ├── hosts-sf.txt
│   ├── hosts-th.txt
│   ├── live.txt
│   ├── maltego-import.csv
│   ├── people-sf.txt
│   ├── resolved.txt
│   └── typosquats.txt
├── raw
│   ├── END.txt
│   ├── START.txt
│   ├── sf-example.json
│   ├── th-example.com.json
│   └── th-example.com.xml
└── reports
    └── recon-ng.html

Raw output is immutable evidence, merged/ is derived working data, reports/ is what leaves the building. Keep the three separate and the "where did this host come from?" question during report review takes ten seconds instead of an hour.

Part 13: OPSEC, Rate Limits and the Legal Line

13.1 What "passive" actually protects you from

Passive means the target's own logs do not show your activity. It does not mean invisible. Three parties still see you:

1. The data provider. Shodan, Censys, SecurityTrails and Hunter all log queries against your key.
2. Your egress network. Your ISP or corporate proxy sees every request.
3. Anyone the provider notifies. A small number of services offer subscribers alerts on lookups of their own assets.

For a red-team engagement where the client is explicitly testing detection, this matters. Route research through infrastructure that is not attributable to you or the client, use research-only accounts, and record which identity was used for which query — that record is part of the deliverable.

13.2 Rate limits are a data-quality problem, not just an annoyance

When a source rate-limits you mid-run, most of these tools silently return partial results. You do not get an error; you get fewer hosts, and you have no way to tell the difference between "the organisation has 40 subdomains" and "the API cut me off at 40." Mitigations:

• Run each source separately and compare counts across runs.
• Watch SpiderFoot's Status tab for modules in an error state.
• Re-run passive collection a day later and diff — a large delta means the first run was truncated, not that the estate grew.
• Prefer keyed API access over scraping where you can; it fails loudly rather than silently.

13.3 The legal line

Broad strokes, and not legal advice — Chapter 5 of the Security Foundations notebook covers this properly:

• Querying public datasets about an organisation is generally lawful.
• Actively probing systems you are not authorised to probe is not. In many jurisdictions this includes DNS brute-force, port scanning, and HTTP requests to non-public paths. The UK Computer Misuse Act and the US CFAA are both written broadly enough that "I only sent one HTTP request" is not a reliable defence.
• Personal data is regulated. Under GDPR, an employee's name and work email are personal data. Collecting them for a scoped, contracted security assessment can be lawful; storing them indefinitely, or on personal equipment, or after the engagement closes, is a different question. Have a retention policy, apply it, and record that you applied it.
• Breach data is the sharpest edge. Some jurisdictions treat possession of leaked credential dumps as an offence regardless of intent. Using an API such as HIBP that answers "has this address appeared in a breach" without returning the credential is materially safer than downloading dumps, and it is what a professional engagement should use.
• Bug-bounty policy is a contract. If a programme prohibits automated scanning, running SpiderFoot in footprint mode against it is a policy breach even though every individual request looks innocuous.

Data handling. Everything these tools produce about people — names, emails, phone numbers, addresses, social profiles — is subject-access-request territory. Encrypt engagement directories at rest, do not sync them to personal cloud storage, and delete them on the schedule your contract specifies.

Part 14: Detection & Defense Angle

Everything above is also a defensive playbook. If you run security for an organisation, the frameworks are more useful pointed inward than they are as a threat.

14.1 What you can and cannot detect

flowchart LR
    subgraph Invisible["Invisible to you"]
        A["CT log queries"]
        B["Passive DNS lookups"]
        C["Shodan/Censys historic data"]
        D["Search-engine scraping"]
        E["Breach-data lookups"]
    end
    subgraph Visible["Visible in your logs"]
        F["DNS brute-force at your authoritative NS"]
        G["HTTP requests with tool User-Agents"]
        H["Sequential vhost probing"]
        I["Takeover-check requests to dangling CNAMEs"]
        J["Port scans"]
    end
    Invisible --> R["Reduce the data, not the detection"]
    Visible --> S["Alert, rate-limit, and correlate"]

The left column is the point: you cannot detect most OSINT, so the defence is to shrink what is collectable. The right column is where detection engineering applies.

14.2 Detections worth building

The certificate row is the standout. CT monitoring (certstream, or your CA's own alerting) is cheap, near-zero-false-positive, and catches both attacker-issued certs for lookalike domains and shadow-IT teams standing up hosts nobody told you about.

14.3 Reducing the collectable surface

This is where the real defensive win is:

• Run these tools against yourself, on a schedule. A weekly SpiderFoot passive scan plus a Recon-ng resource file, diffed against last week, is a free attack-surface-management programme. recon-ng -w selfscan -r monitor.rc && diff hosts-last.txt hosts.txt is genuinely most of the product.
• Kill dangling DNS. Every decommissioning ticket must include "remove the DNS record." Subdomain takeover exists entirely because this step gets skipped.
• Do not put internal hostnames in public certificates. A SAN list containing jenkins.internal.example.com publishes your internal naming convention to CT logs permanently. Use a private CA internally, or wildcards where appropriate.
• Audit WHOIS and SOA contacts. A named individual's email in a public record is a phishing target and a reverse-WHOIS pivot into every other domain you own. Use role addresses and registrar privacy.
• Strip document metadata. Author names, usernames, internal paths and software versions in published PDFs and Office files are a free org chart. Sanitise on publish.
• Assume your email convention is public. It is. Defend with MFA, phishing-resistant factors, and impossible-travel detection — not with obscurity.
• Monitor for typosquats. SpiderFoot's SIMILARDOMAIN events, run against your own brand, are a phishing-infrastructure early-warning feed.

IR use case: when an incident starts with a suspicious domain, a SpiderFoot investigate scan on that domain gives you registration age, hosting, shared infrastructure, reputation hits and related domains in one pass. Recon-ng's reverse_resolve on the attacker IP frequently surfaces the rest of their estate. This is the same tooling, pointed the other way.

Part 15: Final Revision / Summary

• An OSINT framework differs from a recon tool by having a data model: it stores what it finds, normalises it, and lets you take the second hop. Entities + transforms + storage = pivoting.
• Recon is a fixed-point computation over a graph, not a linear pipeline. You apply transforms until nothing new appears. SpiderFoot automates that loop, Recon-ng makes you drive it, Maltego makes you see it.
• API keys, not tool choice, determine result depth. Register research-only accounts, store keys in each tool's proper config, never on the command line.
• theHarvester — one command, ~40 sources, emails + hosts + IPs in seconds. -b sources are passive; -r, -n, -c, -t, --screenshot are active. Always -f to JSON and parse the JSON, not the terminal.
• Recon-ng — Metasploit-shaped, marketplace-based since v5 (marketplace install, modules load, not use). One workspace per engagement. The SOURCE option accepts arbitrary SQL, which is how you express targeted pivots. -w/-r make it scriptable and cron-able.
• SpiderFoot — automated recursive collection over 200+ modules with typed events and a correlation engine. The use case (passive/footprint/investigate/all) is the most consequential setting you choose. Bind the UI to localhost.
• Maltego — visual link analysis. CE's 12-result transform cap means you collect elsewhere and import, then use the graph for the relational insight that tables hide.
• Passive sources are archives. Always resolve before believing a hostname. In the lab, 118 passive hostnames became 63 resolving and 7 answering HTTP.
• Passive ≠ invisible. The data provider and your egress network still see you. Passive ≠ evidence either — a passive CVE hit is a lead requiring verification.
• Shared IPs are the highest-value pattern. One SQL query grouping hosts by IP found the box hosting vpn, git, dev-old and jenkins-legacy together.
• Defensively, most OSINT is undetectable, so the win is reducing what is collectable: kill dangling DNS, keep internal names out of public certificates, monitor CT logs, use role-based WHOIS contacts, strip document metadata — and run these exact tools against yourself on a schedule, diffing week over week.
• Legally, querying public datasets is generally fine; actively probing without authorisation is not; personal data is regulated; breach dumps are the sharpest edge — prefer APIs that answer yes/no over possessing the data.

Part 16: Cheat Sheet / Quick Reference

theHarvester

# Fast passive pass, no keys required
theHarvester -d TARGET -b crtsh,hackertarget,rapiddns,otx,threatminer,urlscan,anubis,bing,duckduckgo -l 1000 -f out
 
# Everything you have keys for
theHarvester -d TARGET -b all -l 1000 -f out-all
 
# Email focus
theHarvester -d TARGET -b hunter,intelx,bing,duckduckgo,github-code -l 500 -f emails
 
# Active: resolve + takeover check (SCOPE REQUIRED)
theHarvester -d TARGET -b crtsh,otx -r -e 1.1.1.1 -t -f out-active
 
# DNS brute force (LOUD, SCOPE REQUIRED)
theHarvester -d TARGET -c -e 1.1.1.1 -f out-brute
 
# Through Burp for inspection
theHarvester -d TARGET -b bing -p -f out-proxied
 
# Parse
jq -r '.hosts[]?'  out.json | cut -d: -f1 | sort -u > hosts.txt
jq -r '.emails[]?' out.json | sort -u              > emails.txt
jq -r '.ips[]?'    out.json | sort -u              > ips.txt

Recon-ng

# Setup
workspaces create ENGAGEMENT
keys add shodan_api YOURKEY
keys add hunter_io  YOURKEY
keys list
marketplace refresh
 
# Find and install
marketplace search domains-hosts
marketplace info  recon/domains-hosts/certificate_transparency
marketplace install recon/domains-hosts/certificate_transparency
 
# Seed
db insert domains
 
# Run
modules search hosts
modules load recon/domains-hosts/certificate_transparency
info
options set SOURCE example.com
run
back
 
# Targeted pivot with SQL
options set SOURCE query SELECT DISTINCT ip_address FROM hosts WHERE ip_address IS NOT NULL
 
# Inspect
show hosts
show contacts
db schema
db query SELECT ip_address, count(*) n FROM hosts GROUP BY ip_address HAVING n>2 ORDER BY n DESC
dashboard
 
# Evidence + report
spool start ~/eng/recon.log
modules load reporting/html
options set FILENAME ~/eng/report.html
run
snapshots take
 
# Non-interactive
recon-ng -w ENGAGEMENT -r script.rc

SpiderFoot

python3 ./sf.py -l 127.0.0.1:5001                    # web UI (localhost only!)
python3 ./sf.py -M                                    # list modules
python3 ./sf.py -T                                    # list event types
python3 ./sf.py -s TARGET -u passive -q -o json       # passive scan, JSON
python3 ./sf.py -s TARGET -u footprint -q -o csv      # ACTIVE, scope required
python3 ./sf.py -s TARGET -m sfp_crt,sfp_dnsresolve -q -o tab
python3 ./sf.py -s TARGET -t EMAILADDR,HUMAN_NAME -u passive -q -o json
python3 ./sf.py -c SCANID                             # correlations for a scan
 
# Extract
jq -r 'select(.type=="INTERNET_NAME") | .data' out.json | sort -u
jq -r '.type' out.json | sort | uniq -c | sort -rn

Event types to check first: INTERNET_NAME, EMAILADDR, HUMAN_NAME, SIMILARDOMAIN, LEAKSITE_URL, VULNERABILITY_CVE_CRITICAL, MALICIOUS_IPADDR, ACCOUNT_EXTERNAL_OWNED, CLOUD_STORAGE_BUCKET_OPEN, SSL_CERTIFICATE_EXPIRED, JUNK_FILE.

Maltego

Decision rule

Need something in 60 seconds ................ theHarvester
Need a durable, queryable, reportable dataset  Recon-ng
Need maximum coverage, hands-off ............ SpiderFoot (passive!)
Need to SEE a relationship .................. Maltego
Doing a real engagement ..................... all four, in that order

Part 17: Common Pitfalls

1. Believing passive hostnames without resolving them. Passive sources are historical archives. Resolve everything before it enters a report.
2. Running SpiderFoot in all or footprint mode before authorisation. This generates active traffic against the target. Passive should be your muscle-memory default.
3. Following v4 Recon-ng tutorials on v5+. use and show modules no longer exist. Use modules load and marketplace search.
4. marketplace install all. Hundreds of modules, many dead, unusable search results.
5. Mixing engagements in one Recon-ng workspace. One workspace per client, always. This is a data-protection issue, not a tidiness issue.
6. Exposing the SpiderFoot UI on 0.0.0.0. Unauthenticated remote-scan capability on your network.
7. Trusting a passive CVE hit as a finding. It is a banner match. Verify or label it clearly as unverified.
8. Silent rate-limit truncation. Fewer results, no error. Re-run and diff.
9. Reporting affiliate/third-party assets as in scope. Frameworks walk off the target estate freely. Check every asset against the scope document before it enters the report.
10. Keys on the command line. They land in shell history and process listings. Use each tool's config store.
11. Ignoring the shared-IP pattern. Grouping hosts by IP is a one-line query and repeatedly the highest-value observation in a recon dataset.
12. Collecting personal data with no retention plan. Names and emails are regulated personal data. Decide the deletion date before you collect.

Part 18: Practice Labs & Resources

Deliberately safe targets for the tools

• Run the entire Part 12 lab against a domain you own — a personal domain with a couple of subdomains is ideal, because you know ground truth and can measure each tool's false-negative rate.
• TryHackMe — "OhSINT": a short room built around extracting metadata and pivoting from a single image; excellent for the entity→transform mindset before you touch a framework.
• TryHackMe — "Searchlight - IMINT": image intelligence and geolocation pivots; the human-judgement half of Maltego's value.
• TryHackMe — "Sakura Room": a full OSINT investigation that walks username → social account → GitHub → infrastructure. This is the exact pivot chain Recon-ng's profiles-profiles modules automate.
• TryHackMe — "Red Team Recon" and "Passive Reconnaissance": cover theHarvester and Recon-ng directly with guided tasks.
• HackTheBox — "Reconnaissance" / "Footprinting" modules on HTB Academy: paid, but the footprinting module is the most rigorous structured treatment of this material available.

OSINT-specific competition

• TraceLabs Search Party CTF: real missing-persons OSINT, run several times a year. Maltego and SpiderFoot are the standard toolkits, and the scoring rewards exactly the pivot discipline in Part 2. It is also the best possible practice in handling sensitive personal data responsibly.
• picoCTF — the Forensics/General Skills categories include metadata and search-based challenges that reward theHarvester-style thinking.
• Cyber Detective CTF (open-sourced challenge sets) — general OSINT with a strong link-analysis component.

Bug-bounty application

• Pick a wide-scope public programme on HackerOne or Bugcrowd (*.example.com plus acquisitions), read its policy for automated-scanning rules, then run the Part 12 pipeline in passive mode only. Compare your merged host list with the programme's published scope — assets you found that are not listed are a scope question to ask, not a target.
• Read disclosed reports tagged subdomain takeover and information disclosure; trace backwards from the finding to which recon step would have surfaced it. Most of them start exactly where this chapter's lab ends: an old/dev/legacy hostname on a shared IP.

Extending the tooling

• Write one SpiderFoot correlation rule encoding your own "what matters" judgement — e.g. non-production-looking hostnames with open ports — and run it against your previous scans.
• Write one Recon-ng resource file that runs your standard passive chain end-to-end, then schedule it and diff the host list weekly. That single file is a working attack-surface-monitoring programme.
• Write one Maltego local transform with maltego-trx that graphs a data source none of the hub items cover — an internal asset inventory, or the JSON from your SpiderFoot scans.

Reading

• The Recon-ng wiki on the project repository — the only authoritative source for current v5+ command syntax.
• SpiderFoot's module documentation — worth reading the full -M list once, so you know what exists.
• Maltego's transform and Machine documentation, plus the maltego-trx library docs for local transforms.
• OSINT Framework (osintframework.com) — a taxonomy of sources rather than a tool, useful for finding the dataset a framework doesn't cover.