Web page inspector for Elixir.
Funkspector is a web scraper that lets you extract data from web pages and XML or TXT sitemaps.
Add funkspector to your list of dependencies in mix.exs:
def deps do
[{:funkspector, "~> 2.0"}]
endFunkspector 2.0 uses Req (Finch/Mint) as its default HTTP transport. HTTPoison remains available as an opt-in adapter — see HTTP Adapter below.
Pass Funkspector a URL to resolve and it will return the final URL after following redirections:
{:ok, final_url, response} = Funkspector.resolve("http://example.com")Pass Funkspector the URL of a web page and it will return a Funkspector.Document with the scraped data:
{:ok, document} = Funkspector.page_scrape("https://example.com")The returned document.data contains:
%{
urls: %{
original: "https://example.com",
base: "https://example.com",
canonical: nil,
parsed: %{scheme: "https", host: "example.com", ...},
root: "https://example.com/"
},
links: %{
raw: ["/about", "https://other.com", ...],
http: %{
internal: ["https://example.com/about", ...],
external: ["https://other.com", ...]
},
non_http: ["mailto:hi@example.com", ...]
},
headers: %{"content-type" => "text/html;charset=utf-8", ...}
}Funkspector can extract the locations from XML sitemaps:
{:ok, document} = Funkspector.sitemap_scrape("https://example.com/sitemap.xml")
document.data.locs
# => ["https://example.com/", "https://example.com/about", ...]It also supports plain text sitemaps:
{:ok, document} = Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt")
document.data.lines
# => ["https://example.com/", "https://example.com/about", ...]Funkspector.page_scrape("https://example.com", %{user_agent: "My Bot"})Funkspector.page_scrape("https://example.com", %{basic_auth: {"user", "secret"}})Use recv_timeout to set a custom timeout in milliseconds:
Funkspector.page_scrape("https://example.com", %{recv_timeout: 5_000})Note:
recv_timeoutis a per-chunk receive timeout, not a bound on total request time.
TLS certificates are verified by default. To scrape a host with a broken, self-signed, or expired certificate, opt out explicitly (this disables MITM protection for that request, so use it deliberately):
Funkspector.page_scrape("https://self-signed.example.com", %{insecure: true})max_body_size (default 100 MB) bounds both the raw response body and gzip
decompression, protecting against memory exhaustion and decompression bombs
when scraping untrusted URLs. Oversized responses return
%Funkspector.Error{reason: :body_too_large}. Set :infinity to disable:
Funkspector.page_scrape("https://example.com", %{max_body_size: 5_000_000})You can skip the HTTP request if you already have the document contents:
Funkspector.page_scrape("https://example.com", %{contents: "<html>...</html>"})
Funkspector.sitemap_scrape("https://example.com/sitemap.xml", %{contents: "<xml>...</xml>"})
Funkspector.text_sitemap_scrape("https://example.com/sitemap.txt", %{contents: "..."})In case of error, Funkspector returns the original URL and the reason:
case Funkspector.page_scrape("https://example.com") do
{:ok, document} ->
IO.inspect(document.data)
{:error, url, reason} ->
IO.puts("Could not scrape #{url} because of #{inspect(reason)}")
endThe reason is one of:
:invalid_url— the URL failed Funkspector's syntax/TLD validation.- A
%Funkspector.Error{reason: atom, adapter: module}— a transport failure (e.g.,:nxdomain,:timeout,:closed,{:tls_alert, _}). The:reasonatom comes fromgen_tcp/inet, so the contract is the same across adapters. - A
%Funkspector.Response{status_code: integer}— a non-2xx HTTP response (e.g., 404, 500, 300).
Funkspector 2.0 ships with two HTTP adapters:
Funkspector.HTTP.Adapters.Req(default) — backed by Req/Finch/Mint. Nohackneydependency, so it is unaffected by the hackney 1.21 issues (CVE-2026-47075, CVE-2026-47076) whose fix in hackney 4.0.1 is blocked for HTTPoison by httpoison#501.Funkspector.HTTP.Adapters.HTTPoison— backed by HTTPoison/hackney. Opt-in for users who want to preserve the pre-2.0 transport; note it carries the hackney 1.21 CVEs above.
Set the adapter app-wide in your config/config.exs:
config :funkspector, :http_adapter, Funkspector.HTTP.Adapters.HTTPoisonYou'll also need to declare httpoison (and hackney) in your own
mix.exs deps since Funkspector marks them optional: true:
{:httpoison, "~> 2.3"}Pass :adapter in the options map to override on a single call:
Funkspector.resolve("https://example.com", %{
adapter: Funkspector.HTTP.Adapters.HTTPoison
})The error and response structs are normalized across adapters:
# Funkspector 1.x
{:error, url, %HTTPoison.Error{reason: :nxdomain, id: nil}}
# Funkspector 2.0
{:error, url, %Funkspector.Error{reason: :nxdomain, adapter: _}}The :reason atom is unchanged, so most pattern matches require only a
struct rename.
Funkspector fetches whatever URL you give it and follows redirects, so when you point it at untrusted or user-supplied URLs, keep in mind:
- No SSRF filtering.
valid_url?/1acceptslocalhost, IP-literal hosts (including private ranges like10.0.0.0/8, link-local169.254.169.254, and IPv6 loopback), and these are reachable both directly and via a redirect from a public URL. Funkspector does not block them — enforce your own allow/deny policy or network-level egress controls before fetching attacker-influenced URLs. Note that input validation alone is insufficient, because a public URL can redirect to an internal one. - TLS is verified by default. Disable it only deliberately, per call, with
%{insecure: true}. - Bound the response size with
max_body_size(on by default at 100 MB) when scraping untrusted hosts. links.http.internalis not a security boundary — it is a host-string match against the final fetched host and honors the page's<base href>. Re-validate any extracted URL before fetching it.