CI build not running OSGi tests

[andrewfg]

In #4913 I added an OSGi test class in org.openhab.core.thing.tests in order to validate the new functionality. However if I add an assertion to that test class that should fail (e.g. something like assertTrue(false)) the CI build does not actually fail. Which proves that the whole test class is not being run. I tried this with not only my added test class but also with some of the prior existing classes, and none of those fail either. So I think that the CI build is not actually running any of those OSGi tests at all. (And indeed this error may extend to other modules than just org.openhab.core.thing.tests..)

Originally posted by @andrewfg in #4913 (comment)

[jimtng]

I tried building on my computer, but it appears that openhab.org is currently down / experiencing DNS issues?

[WARNING] Error while parsing schema(s).Location [ file:/home/openhab/projects/openhab-core/bundles/org.openhab.core.thing/schema/update/update-description-1.0.0.xsd{9,78}].
org.xml.sax.SAXParseException: java.net.UnknownHostException: openhab.org

[andrewfg]

I asked CoPilot and it suggested that the following include may be required. => WDYT??

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0</version>
  <configuration>
    <includes>
      <include>**/*OSGiTest.java</include>
    </includes>
  </configuration>
</plugin>

Or alternatively..

mvn clean verify -Posgi-tests

[andrewfg]

it appears that openhab.org is currently down / experiencing DNS issues?

@jimtng it is fixed again.. I.e. the build. However the tests are still not running.

[andrewfg]

@openhab/core-maintainers could someone please look at this?

[andrewfg]

@jimtng could you also confirm that the OSGi tests are not running on your system? I wonder if I am hallucinating..

[andrewfg]

Maybe because the .bndrun file contains this ??

-runblacklist: \
	bnd.identity;id='org.osgi.service.cm'

[wborn]

Are you on the latest git commit?

Maybe the runbundles need to be resolved properly?
See also: #4091

There are issues with newer JUnit versions: #4642 (comment)

[andrewfg]

Are you on the latest git commit?

Yes I am on today's v5.1 snapshot.

The issues that you refer to seem to confirm what I observe. It would be a good first step if someone else (i.e. you) can confirm that the tests are not running. I think this is beyond my capabilities to fix. But I think it is absolutely essential to OH that its tests shall run. IMHO the risk of ignoring this issue is life critical to OH.

PS for the avoidance of doubt: my observation is that OSGI tests are not running on OH CI build on the cloud. And they also do not run on local maven builds. The latter is IMHO not 100% critical. But the former is life critical.

[jimtng]

@andrewfg, are you referring to itests? Normal tests are running fine on local builds.

Example: I added a failing test case in bundles/org.openhab.core.thing/src/test/java/org/openhab/core/thing/link/events/LinkEventFactoryTest.java and when I ran a build with mvn clean spotless:apply install -T1C -pl :org.openhab.core.thing it definitely caused a build failure.

ERROR] Failures: 
[ERROR]   LinkEventFactoryTest.testCreateItemChannelLinkAddedEvent:57 expected: <false> but was: <true>
[INFO] 
[ERROR] Tests run: 406, Failures: 1, Errors: 0, Skipped: 0

"OSGi tests" in itests are not run during this part, and are usually run during ci build. I however don't have a lot of experience dealing with them. For curiosity's sake, I'd like to know how to trigger them on my dev machine too, or be pointed to where to read more about them.

[andrewfg]

Normal tests are running fine on local builds

Indeed normal Junit test are running fine.

are you referring to itests?

I am. These are live OSGi tests where actual components are created and run against each other. These test classes are in the itest folder vs the bundles folder, and have names like xyzOSGiTest.java. Often they are called integration tests. They seem never to be run during CI build. But they are critical to test .. well .. the integration of OH component classes.

[jimtng]

It seems that itests are running during ci build: https://github.com/openhab/openhab-core/actions/runs/16510944422/job/46692680478?pr=4928

[andrewfg]

@jimtng yes I see that the tests are running (and now failing) in #4928 for org.openhab.core.model.item.tests. My issue is that the new test suite that I had added in #4913 in org.openhab.core.thing.tests seemed not to be running. However my test suite does depend of the correct item parsing covered by the code that was broken prior to #4928, so I wonder if my tests had been magically suppressed in relation to pre #4928 being wrong. So let’s wait until your PR passes and is merged, and then we can see if the tests for my PR then run and also pass.

[jimtng]

@andrewfg please see the updated original post in #4928. Is your test done by using DSL .items parser? If so, you could probably get around it by prefixing with ^ as explained in #4928.