Nested Describe Functions Fail

[FlippieCoetser]

According to Testhat Documentation, nesting describe() functions should be possible: link

Below is a copy of the example in the documentation

# Nested specs:
## code
addition <- function(a, b) a + b
division <- function(a, b) a / b

## specs
describe("math library", {
  describe("addition()", {
    it("can add two numbers", {
      expect_equal(1 + 1, addition(1, 1))
    })
  })
  describe("division()", {
    it("can divide two numbers", {
      expect_equal(10 / 2, division(10, 2))
    })
    it("can handle division by 0") #not yet implemented
  })
})

However, when trying to run the example in VS code, an error is thrown:

Removing the nesting like so:

## specs
describe("math library", { 
  it("can add two numbers", {
    expect_equal(1 + 1, addition(1, 1))
  })
  it("can divide two numbers", {
    expect_equal(10 / 2, division(10, 2))
  })
  it("can handle division by 0") #not yet implemented
})

Works as expected.

[meakbiyik]

Oof, finally someone stumbled upon some edge case of my hack of running tests by themselves 😅 Thanks for the issue, although, note that testthat has file atomicity, so anything finer than that might not be trustworthy. I will check this out at some point, but no promises.

[kubajal]

Here is a proposal for the fix for this issue in the testthat repo: r-lib/testthat#2118.

[kubajal]

tldr: upvote r-lib/testthat#2220 (comment). Future maintainers of VSCode R Test Explorer, RStudio and testthat will thank you.

The mechanism for running nested describe/testthat/it statments in VSCode has been partially enabled upstream in r-lib/testthat#2118 and now awaits implementation on VSCode Test Explorer side. Let me provide my initial analysis of how to implement that and perhaps someone could implement that before I find spare time to do that.

To start with, please note that the current approach in R Test Explorer for test discovery is that we have a custom tree sitter parser for testthat tests. The problem is that tree sitter doesn't support arbitrary recursive grammar rules, see tree-sitter-grammars/tree-sitter-lua#24 (comment). We need an improvement for discovery of testthat tests. This could be done in 2 ways.

1. Easier dirtier way

Implement a workaround around tree-sitter grammar recursion limitation.
We could improve the tree-sitter parser for testthat tests such that there is a fixed maximal nesting level, let's say 5 or 10 as I don't believe anyone would write tests with higher nesting levels. The rest of the VSCode side logic stays essentially the same. This isn't a generic solution and this should be considered a workaround.

2. Harder cleaner way

We could try to encourage the maintainers of testthat to properly implement test discovery on their side such similarily to how other popular unit test frameworks do it in other programming languages (e.g. gtestBinary --list_tests). We could then remove test discovery from our side which would otherwise require the workaround above.
By the way, there is a broader benefit to it because not only would it radically simplify the code on VSCode R Explorer side but also would enabled easier intergration of other IDEs as well. I believe that ironically RStudio would benefit from that too (i.e. effective maintainers of testthat).
Unfortunately, the code changes implementing better test discovery upstream in testthat were rejected in r-lib/testthat#2220 for reasons I don't understand. Please support our case by upvoting r-lib/testthat#2220 (comment).

Personally I would prefer option 2 as it is better long term but if anyone implemets 1 I'm fine with that too.

[meakbiyik]

Thanks a lot for the summary @kubajal, totally agreed. If someone would like to tackle option 1, I would be happy to dust off my tree-sitter grammar knowledge, but it might be a tad challenging.