Add post merge build test

[maksymiuks]

Recently we are discovering plenty of bugs trying to create a traceability matrix for big and low-level packages like rlang. What do you think about adding a job that would run main covtracer branch against rlang and maybe some other big package every day on a schedule? This way we gonna be proactive about possible breaking changes and will be able to identify and fix them earlier. What do you think?

[dgkf]

I think this text got garbled some how, but I think I get you.

Yeah, I like the idea. I'm not sure how many issues we're going to dig up by repeatedly testing the same package, though. Would it be too crazy to pull a random package from, say, the top 1000 most frequently downloaded CRAN packages?

[maksymiuks]

Sorry, fixed the formatting already.

Well, recent updates to rlang (or to covtracer that were later tested against rlang) prompted 3 or 4 hotfixes. I think the primary reason for that is that rlang is one of the lowest-level packages and that's why it is gonna include structures that are not seen in any other package.

I thought about doing a random sampling but decided it might be too wide a range. Among those 1000 probably 80% will be simple packages with not much possibilities to stress covtracer. But maybe a mix of both would work? Let's have 2-3 predefined packages we think it is important to always test against and then sample 3-4 more? Does it sound better?

[dgkf]

Yeah, you make a good point. When we identify edge cases, they're almost always while testing rlang. Then let's start there and we can see whether we want to do something more stochastic later on.