>_ shadow.red

HTTP Enumeration

Reconnaissance Service Enumeration linux

Metasploit modules

search type:auxiliary name:http
use auxiliary/scanner/http/http_version
use auxiliary/scanner/http/http_header
use auxiliary/scanner/http/robots_txt
use auxiliary/scanner/http/dir_scanner
use auxiliary/scanner/http/files_dir
use auxiliary/scanner/http/http_login
use auxiliary/scanner/http/http_put

Nmap

nmap -sV --script=http-enum 10.10.23.24
nmap -sV -p 80 --script=http-enum 10.10.4.12

Direct request

curl http://10.10.2.14/secure/

robots.txt and sitemaps

curl google.com/robots.txt

Important files to check: robots.txt, sitemap.xml, sitemaps.xml.