Headless Crawl: performAdditionalAnalysis uses an uninitialized parser - JS/redirect/form extraction may be silently disabled

[awerqo]

Hey, when I read the katana code for integration as a lib, I noticed this in the headless engine

katana/pkg/engine/headless/headless.go

Lines 237 to 238 in 0f80ea8

	responseParser := parser.NewResponseParser()
	newNavigations := responseParser.ParseResponse(rr.Response)

So we don't provide init options as in the hybrid and basic crawl

katana/pkg/types/crawler_options.go

Lines 60 to 68 in 0f80ea8

	parserOptions := &parser.Options{
	AutomaticFormFill: options.AutomaticFormFill,
	ScrapeJSLuiceResponses: options.ScrapeJSLuiceResponses,
	ScrapeJSResponses: options.ScrapeJSResponses,
	DisableRedirects: options.DisableRedirects,
	}
	responseParser := parser.NewResponseParser()
	responseParser.InitWithOptions(parserOptions)

Moreover, we have an inited parser in the headless crawler struct

katana/internal/runner/runner.go

Line 88 in 0f80ea8

crawlerOptions, err := types.NewCrawlerOptions(options)

katana/internal/runner/runner.go

Line 101 in 0f80ea8

crawler, err = headless.New(crawlerOptions)

So maybe in the performAdditionalAnalysis func we should just call this, or am I missing something?
newNavigations := h.options.Parser.ParseResponse(rr.Response)

Is the missing InitWithOptions here intentional, or am I missing something? I haven't tested this change yet, but I can try when I have free time.

@ehsandeep @Ice3man543 - if you happen to recall the reasoning behind this, that would be very helpful. Happy to open a PR with the change (and a test) if it's just an oversight.

Thanks!