CI should validate on real production traffic.

[monperrus]

CI should validate on real production traffic.

algorithm:

for each dataset of production requests:
  measure percentage of bot detection
compute average percentage

a pull request should never increase detection rate by more than X%, say 1%.

[mwnciau]

I've been collecting user agents datasets here, which might be of interest: https://github.com/mwnciau/user_agent_dumps/

The Intoli dataset seems to have already filtered out most bots already, unfortunately, but it still might be useful in verifying new patterns don't match common valid user agents.

The top_user_agents dataset does contain bots, I think, but since it only includes the top 100 user agents in a given week, there are actually very few bots in the dataset.

[monperrus]

thanks for the pointer.