Expired Domain Scanner

A web application that crawls websites and identifies outbound links pointing to expired or unresolvable domains. Useful for SEO audits, link-building research, and finding domain acquisition opportunities.

---

How It Works

The scanner crawls a target website page by page, extracts all outbound (external) links, and checks whether each linked domain resolves via DNS. Domains with no DNS records (NXDOMAIN) are flagged as potentially expired. For each expired domain found, it optionally queries WHMCS to retrieve the domain's registration expiry date via WHOIS.

Results are stored in MongoDB and accessible through a filterable, sortable UI.

---

Features

• Website crawler — respects robots.txt, configurable batch size and concurrency
• DNS checking — checks A, AAAA, CNAME records to confirm a domain is dead
• WHOIS lookup — retrieves domain expiry date via WHMCS API (optional)
• Background scanning — navigate between pages without interrupting an active scan
• Live status bar — real-time crawl stats visible from any page
• Results page — sortable columns, TLD filter, expiry date filters, shortlist with localStorage persistence
• Summary page — per-website overview with drill-down to see all expired domains found on that site
• Scan resume — pause and resume large crawls from where they left off

---

Prerequisites

Requirement	Version	Notes
Node.js	v18+	Required for both API and UI
MongoDB	v6+	Local or remote instance
WHMCS	Any	Optional — needed for WHOIS expiry dates only

---

Project Structure

expired-domain-scan/
├── api/                        # Express API + crawler
│   ├── src/
│   │   ├── server.js           # HTTP server, REST endpoints
│   │   └── worker.js           # Crawler, DNS checker, WHOIS lookup
│   ├── .env                    # Environment variables (create this)
│   └── package.json
│
└── expired-scanner-ui/         # React frontend (Vite)
    ├── src/
    │   ├── App.jsx             # Root layout, nav, status bar
    │   ├── ScanContext.jsx     # Global scan state (React context)
    │   ├── Scanner.jsx         # Scan control + live results
    │   ├── Results.jsx         # Stored results with filtering
    │   └── Summary.jsx         # Per-website summary + drill-down
    └── package.json

---

Installation

1. Clone the repository

git clone <your-repo-url>
cd expired-domain-scan

2. Install API dependencies

cd api
npm install

3. Install UI dependencies

cd ../expired-scanner-ui
npm install

4. Start MongoDB

If you installed MongoDB via Homebrew on macOS:

brew services start mongodb-community

Verify it's running:

mongosh --eval "db.runCommand({ping:1})"
# Expected: { ok: 1 }

5. Configure environment variables

Create a .env file inside the api/ directory:

cd api
cp .env.example .env   # or create it manually

Edit api/.env:

# MongoDB connection (optional — defaults to localhost:27017)
MONGO_URL=mongodb://localhost:27017

# Port for the API server (optional — defaults to 4000)
PORT=4000

# WHMCS API credentials (optional — only needed for WHOIS expiry date lookups)
WHMCS_API_URL=https://your-whmcs-domain.com/includes/api.php
WHMCS_API_IDENTIFIER=your_api_identifier
WHMCS_API_SECRET=your_api_secret

Note: The scanner works without WHMCS credentials — expired domains will still be detected via DNS, but registration expiry dates won't be available.

---

Running the Application

You need two terminals — one for the API server and one for the UI.

Terminal 1 — Start the API

cd api
npm run dev        # development (auto-restarts on file changes)
# or
npm start          # production

Expected output:

API listening on 4000
Connected to MongoDB

Terminal 2 — Start the UI

cd expired-scanner-ui
npm run dev

Expected output:

VITE ready in Xms
➜  Local: http://localhost:5173/

Open http://localhost:5173 in your browser.

---

Usage

Running a Scan

1. Go to the Scanner tab
2. Enter the URL of the website you want to crawl (e.g. https://example.com)
3. Optionally adjust:
   • Batch size — number of pages to crawl per run (default: 1000)
   • Concurrency — number of pages fetched in parallel (default: 5, max: 10)
4. Click Start Scan

The scanner will crawl the site page by page, checking every outbound link. Expired domains appear in the table below as they are found.

Resuming a scan

If a scan is paused (batch limit reached before the site was fully crawled), you'll see a Resume button the next time you enter the same URL. This continues from where it left off.

Navigating During a Scan

You can switch to the Results or Summary tabs while a scan is running. The scan continues in the background — a status bar at the bottom of the screen shows live progress from any page.

Results Page

View all expired domains stored in the database.

Feature	How to use
Filter by website	Type a website name in the search box
Filter by TLD	Select a TLD from the dropdown (.com, .net, etc.)
Filter by expiry	Click a pill: All / Expired / < 30 days / < 90 days / No Date
Sort columns	Click any column header; click again to reverse
Shortlist a domain	Click the ☆ star on any row — turns gold and persists across sessions
View source page	Click the website name — opens the exact page where the expired link was found

Summary Page

View a high-level overview of all scanned websites.

• Shows each scanned website, the number of expired domains found, and the date last scanned
• Click any row to drill down into the full details for that website, including:
  • Stats (total found, already expired, with/without expiry date)
  • Full table with domain, TLD, source page, expiry badge, reason, and scan date

---

API Endpoints

The API runs on http://localhost:4000 by default.

Method	Endpoint	Description
POST	/scan	Start a new scan
GET	/scan/status?startUrl=	Check if a previous scan exists for a URL
GET	/events/:id	SSE stream for live scan progress
GET	/results	Fetch stored expired domains (optional ?website= and ?tld= filters)
GET	/summary	Aggregate count of expired domains per scanned website

POST /scan — request body

{
  "startUrl": "https://example.com",
  "maxPages": 1000,
  "concurrency": 5,
  "mode": "new"
}

• mode: "new" starts fresh, "resume" continues a paused scan

---

Database

MongoDB database: expired_domain_scanner

Collection	Contents
results	One document per unique (website, expired domain) pair
scans	Crawl state — visited URLs, queue, status — used for resume

Result document schema

{
  "website":          "example.com",
  "domain":           "expireddomain.com",
  "tld":              "com",
  "status":           "no-dns",
  "code":             "NXDOMAIN",
  "sourceUrl":        "https://example.com/blog/some-post",
  "expiryDate":       "2023-11-15",
  "expiryDateReason": null,
  "foundAt":          "2024-01-20T10:30:00.000Z"
}

---

Troubleshooting

Connection refused on port 4000

• Make sure the API server is running: cd api && npm run dev
• Check MongoDB is running: brew services list | grep mongo

MongoDB won't connect

• Start it: brew services start mongodb-community
• Verify: mongosh --eval "db.runCommand({ping:1})"

No expiry dates showing

• WHMCS credentials are missing or incorrect in api/.env
• Many TLDs (e.g. .io, .co) don't expose expiry dates in WHOIS — this is normal

Scan stops immediately

• The target site may block crawlers or return errors on most pages
• Try reducing concurrency to 1–2 and check the status bar for error messages

UI shows blank / white text

• Hard-refresh the browser: Cmd+Shift+R (macOS) or Ctrl+Shift+R (Windows)
• Clear browser cache and reload

---

Tech Stack

Layer	Technology
Frontend	React 19, Vite 7
API	Node.js, Express 4
Database	MongoDB 7
Crawler	got, cheerio, p-queue, robots-txt-parser
DNS	Node.js built-in dns module
WHOIS	WHMCS API