🚀 [NEW] Test Case Order Manipulation

[matejicko]

🫴 Proposal

By default, runner takes all test cases and run them in alphabetical order by their names. However, user should be able to manipulate with the order. This possibility should be even during run-time programmatically from the scripts. For these purposes, these methods should be added:

public void NextTestCase(string testCasePath); // Test case that will run as next
public void NextCollection(string subCollectionPath); // Sub-collection that will run as next

public void SkipTestCase(string testCasePath); // Test case to be ignored/skipped
public void SkipTestCases(params string[] testCasesPaths); // Test cases to be ignored/skipped
public void SkipCollection(string subCollectionPath); // Sub-collection to be skipped at all

Apart of this dynamic changes of order, there should be some initialization method, which will give the user ability to configure the order of the executed test cases (not exclusively) in initialization script. This way the order can be changed programmatically, but in the advance:

tp.ConfigureTestRunner(testRunner => {
    testRunner.SetOrder(
        "init-test-case-req.http",
        "./001-cars/001-add-car-req.http",
        "./001-cars/003-edit-car-req.http");

    testRunner.Skip(
        "./001-cars/004-edit-car-2-req.http",
        "clean-up-test-case-req.http",
    );
})

✨ Motivation

These methods can be especially useful, when Response contains values that alter next behavior, or when tests failed and formerly scheduled next request(s) don't make sense anymore.

🔀 Alternatives

Instead of providing paths to test cases or sub-collections, with this issue #59 being implemented, it can be possible to reference test cases by their unique names.

Apart of this programmatic solution, the test cases order determination may be part of configuration files, which will be probably added in the future as well. There, in the JSON format can be specified order of the test cases.

💡 Possible implementation

There should be added new pipeline step type, e.g. SetNextTestCaseStep, which would set next test case to be executed. Each IPipelineStep has method ShouldExecute which should decide according to current ApplicationContext whether it should be executed. By having these two information, it should not be that difficult to set proper condition to ShouldExecute of the step InitializeTestCaseStep. This way the step will be aware of whether it should generate steps for this particular test case or it should be just skipped. Such an approach will prevent from unnecessary steps generation in advance.

Since we are adding free will to user, maybe it would be great to implement some loop detection mechanism, in order to prevent endless loops triggered by user.

➕ Additional context

This issue is re-newed version of #6.