onlyif_cmds_exist should get a kwarg to fail when on CI services

[jankatins]

In this case the an incomplete latex setup on travis lead to missing tests in nbconvert, which prevented the tests from registering a bug. See jupyter/nbconvert#356 for the story...

IMO a testsuite should not skip any tests in a CI environment, so it would be nice if the decorator could raise a failure instead of skipping if it is run on a CI service (=change the default to fail if the command is not present on travis/appveyor/circleci).

CI like travis set some environment variables, so a simple test for such env variables could be used to change a skip to a fail.

[takluyver]

I think previously it was harder to install things like Tex on Travis, which was why we had some skipped tests.

I don't think I'd go for checking for specific CI services, but rather I'd allow setting some environment variable like FAIL_ON_TEST_SKIP=1. Maybe nose/py.test already support an option for something like this?

[jankatins]

I asked for pytest on twitter, but it seems they don't have such an switch. https://twitter.com/janschulz/status/763650601796395008

[takluyver]

It sounds like it should be doable as a py.test plugin rather than changing project code, though. Based on this gist in a response.

Nose has a --no-skip option which I imagine would do what we want.