Move from setuptools to scikit-build-core by volkm · Pull Request #217 · stormchecker/stormpy

volkm · 2025-03-24T09:10:26Z

Updated build system by using scikit-build-core.
Build should be significantly more streamlined now and can be performed with pip install .

One thing which can be improved in the future is the CI for stable versions. Before, we relied on a custom exit code to detect build failure due to version mismatch. I did not find a good solution for this yet and currently we consider exit code 1 as okay.

The build step via pip install . currently rebuilds all files for me, but with the ccache the building time is acceptable.

This PR should fix the following issues:

fixes stormpy passes storm location as CLI flag to setup.py #168
fixes Support for pyproject.toml #162
fixes Warning on setup.py install #95
fixes python setup.py test is soon to be deprecated #10

and make progress towards:

…e available

sjunges · 2025-03-24T11:55:26Z

@lukovdm @linusheck Could the two of you try this PR once on your system?

linusheck · 2025-03-25T16:02:23Z

This seems to have a hard requirement for carl-parser? It complains about it not being there, while the setup.py version dynamically skips compiling everything that depends on carl-parser.

      /Users/linusheck/git/stormpy/stormpy/src/pycarl/typed_parse/parser.h:7:10: fatal error: 'carl-parser/Parser.h' file not found
      #include <carl-parser/Parser.h>

volkm · 2025-03-25T16:07:35Z

In principle, it should also work without the carl-parser, see here. At least in some of the CI tests it seemed to work. Can you check whether what value PYCARL_HAS_PARSE has for you?

sjunges · 2025-03-29T13:56:54Z

When combined with stormchecker/storm#602, I get the following error compiling...

FAILED: CMakeFiles/storage.dir/src/storage/dd.cpp.o
/Library/Developer/CommandLineTools/usr/bin/clang++ -Dstorage_EXPORTS -I/Users/junges/stormpy -I/Users/junges/stormpy/build/cp311-cp311-macosx_14_0_arm64/src -isystem /opt/homebrew/include -isystem /Users/junges/.pyenv/versions/3.11.4/include/python3.11 -isystem /Users/junges/stormpy/build/cp311-cp311-macosx_14_0_arm64/deps/pybind11-src/include -isystem /usr/local/include/storm -isystem /usr/local/include/storm/resources/l3pp -isystem /usr/local/include/storm/resources/gmm -isystem /usr/local/include/storm/resources -isystem /usr/local/include/storm/resources/exprtk/exprtk -isystem /usr/local/include/storm/resources/parallel_hashmap -isystem /usr/local/include/storm/resources/cpphoafparser-0.99.2 -isystem /usr/local/include/storm/resources/modernjson -isystem /usr/local/include -isystem /Library/gurobi1102/macos_universal2/include -isystem /usr/local/include/storm/resources/cudd -isystem /opt/mathsat/include -isystem /usr/local/include/storm/carl/resources/Eigen -isystem /opt/homebrew/include/ginac -isystem /usr/local/include/storm/resources/sylvan -DBOOST_PHOENIX_STL_TUPLE_H -O3 -DNDEBUG -std=gnu++20 -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk -fPIC -fvisibility=default -MD -MT CMakeFiles/storage.dir/src/storage/dd.cpp.o -MF CMakeFiles/storage.dir/src/storage/dd.cpp.o.d -o CMakeFiles/storage.dir/src/storage/dd.cpp.o -c /Users/junges/stormpy/src/storage/dd.cpp
In file included from /Users/junges/stormpy/src/storage/dd.cpp:1:
In file included from /Users/junges/stormpy/src/storage/dd.h:3:
In file included from /Users/junges/stormpy/src/storage/common.h:1:
In file included from /Users/junges/stormpy/src/common.h:5:
/opt/homebrew/include/pybind11/pybind11.h:2415:49: error: call to deleted constructor of 'storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>'
Extra...>(first, last, std::forward(extra)...);
^~~~~
/Users/junges/stormpy/src/storage/dd.cpp:32:82: note: in instantiation of function template specialization 'pybind11::make_iterator<pybind11::return_value_policy::reference_internal, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, std::pair<storm::expressions::SimpleValuation, double>>' requested here
add.def("iter", [](const storm::dd::Add<DdType, double> &s) { return py::make_iterator(s.begin(), s.end()); },
^
/usr/local/include/storm/storm/storage/dd/sylvan/SylvanAddIterator.h:29:5: note: 'AddIterator' has been explicitly marked deleted here
AddIterator(AddIterator<DdType::Sylvan, ValueType> const& other) = delete;
^
/opt/homebrew/include/pybind11/pybind11.h:2372:38: note: passing argument to parameter 'first' here
iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&...extra) {
^
/opt/homebrew/include/pybind11/pybind11.h:2398:23: error: call to deleted constructor of 'storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>'
return cast(state{first, last, true});
^~~~~
/opt/homebrew/include/pybind11/pybind11.h:2410:20: note: in instantiation of function template specialization 'pybind11::detail::make_iterator_impl<pybind11::detail::iterator_access<storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>>, pybind11::return_value_policy::reference_internal, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, std::pair<storm::expressions::SimpleValuation, double>>' requested here
return detail::make_iterator_impl<detail::iterator_access,
^
/Users/junges/stormpy/src/storage/dd.cpp:32:82: note: in instantiation of function template specialization 'pybind11::make_iterator<pybind11::return_value_policy::reference_internal, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>, std::pair<storm::expressions::SimpleValuation, double>>' requested here
add.def("iter", [](const storm::dd::Add<DdType, double> &s) { return py::make_iterator(s.begin(), s.end()); },
^
/usr/local/include/storm/storm/storage/dd/sylvan/SylvanAddIterator.h:29:5: note: 'AddIterator' has been explicitly marked deleted here
AddIterator(AddIterator<DdType::Sylvan, ValueType> const& other) = delete;
^
In file included from /Users/junges/stormpy/src/storage/dd.cpp:1:
In file included from /Users/junges/stormpy/src/storage/dd.h:3:
In file included from /Users/junges/stormpy/src/storage/common.h:1:
In file included from /Users/junges/stormpy/src/common.h:5:
/opt/homebrew/include/pybind11/pybind11.h:2398:30: error: call to deleted constructor of 'storm::dd::AddIterator<storm::dd::DdType::Sylvan, double>'
return cast(state{first, last, true});
^~~~
/usr/local/include/storm/storm/storage/dd/sylvan/SylvanAddIterator.h:29:5: note: 'AddIterator' has been explicitly marked deleted here
AddIterator(AddIterator<DdType::Sylvan, ValueType> const& other) = delete;
^
3 errors generated.

sjunges · 2025-03-29T14:05:01Z

I noticed that the pybind header is an installed header and removing pybind there seems to help. Question is how we avoid this ...

sjunges · 2025-03-29T14:08:38Z

This seems to have a hard requirement for carl-parser? It complains about it not being there, while the setup.py version dynamically skips compiling everything that depends on carl-parser.
      /Users/linusheck/git/stormpy/stormpy/src/pycarl/typed_parse/parser.h:7:10: fatal error: 'carl-parser/Parser.h' file not found
      #include <carl-parser/Parser.h>

I also get this...

[100/113] Building CXX object CMakeFiles/pycarl.parse-gmp.dir/src/pycarl/mod_typed_parse.cpp.o
  FAILED: CMakeFiles/pycarl.parse-gmp.dir/src/pycarl/mod_typed_parse.cpp.o
  /Library/Developer/CommandLineTools/usr/bin/clang++ -Dpycarl_parse_gmp_EXPORTS -I/Users/junges/stormpy -I/Users/junges/stormpy/build/cp311-cp311-macosx_14_0_arm64/src -isystem /opt/homebrew/include -isystem /Users/junges/.pyenv/versions/3.11.4/include/python3.11 -isystem /Users/junges/stormpy/build/cp311-cp311-macosx_14_0_arm64/_deps/pybind11-src/include -isystem /usr/local/include/storm -isystem /usr/local/include/storm/carl/resources/Eigen -isystem /opt/homebrew/include/ginac -DBOOST_PHOENIX_STL_TUPLE_H_ -O3 -DNDEBUG -std=gnu++20 -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk -fPIC -fvisibility=default -MD -MT CMakeFiles/pycarl.parse-gmp.dir/src/pycarl/mod_typed_parse.cpp.o -MF CMakeFiles/pycarl.parse-gmp.dir/src/pycarl/mod_typed_parse.cpp.o.d -o CMakeFiles/pycarl.parse-gmp.dir/src/pycarl/mod_typed_parse.cpp.o -c /Users/junges/stormpy/src/pycarl/mod_typed_parse.cpp
  In file included from /Users/junges/stormpy/src/pycarl/mod_typed_parse.cpp:5:
  /Users/junges/stormpy/src/pycarl/typed_parse/parser.h:7:10: fatal error: 'carl-parser/Parser.h' file not found
  #include <carl-parser/Parser.h>
           ^~~~~~~~~~~~~~~~~~~~~~
  1 error generated.

Indeed, it says:

  -- Stormpy - Support for carl-parser found and included.

Which I believe should not happen. I will investigate.

sjunges · 2025-03-29T14:18:20Z

Yes, the problem is that

set(PYCARL_HAS_PARSE ((USE_PARSER) AND (carlparser_FOUND)))

is not valid CMAKE.

sjunges · 2025-03-29T14:56:21Z

The next error that i run into is

================================================================================================== ERRORS ==================================================================================================
______________________________________________________________________________________ ERROR collecting test session _______________________________________________________________________________________
../.pyenv/versions/3.11.4/envs/storm/lib/python3.11/site-packages/_pytest/config/__init__.py:641: in _importconftest
    mod = import_path(conftestpath, mode=importmode, root=rootpath)
../.pyenv/versions/3.11.4/envs/storm/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
../.pyenv/versions/3.11.4/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
../.pyenv/versions/3.11.4/envs/storm/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
tests/pycarl/parse/conftest.py:4: in <module>
    from stormpy.pycarl import parse
lib/stormpy/pycarl/parse/__init__.py:6: in <module>
    from . import parse
E   ImportError: cannot import name 'parse' from partially initialized module 'stormpy.pycarl.parse' (most likely due to a circular import) (/Users/junges/stormpy/lib/stormpy/pycarl/parse/__init__.py)
========================================================================================= short test summary info ==========================================================================================
ERROR  - ImportError: cannot import name 'parse' from partially initialized module 'stormpy.pycarl.parse' (most likely due to a circular import) (/Users/junges/stormpy/lib/stormpy/pycarl/parse/__init__.py)

volkm · 2025-03-29T22:19:17Z

Thanks for the fixes @sjunges. Did this resolve both your issues or should I take a look into it?

sjunges · 2025-05-05T20:46:43Z

This is my next error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/junges/.pyenv/versions/storm/lib/python3.11/site-packages/stormpy/__init__.py", line 6, in <module>
    from ._config import *
  File "/Users/junges/.pyenv/versions/storm/lib/python3.11/site-packages/stormpy/_config.py", line 3, in <module>
    from stormpy import pycarl
  File "/Users/junges/.pyenv/versions/storm/lib/python3.11/site-packages/stormpy/pycarl/__init__.py", line 6, in <module>
    from . import pycarl_core
ImportError: cannot import name 'pycarl_core' from partially initialized module 'stormpy.pycarl' (most likely due to a circular import) (/Users/junges/.pyenv/versions/storm/lib/python3.11/site-packages/stormpy/pycarl/__init__.py)

volkm · 2025-05-06T07:08:36Z

Thanks for the update. Good to know that you get this error message as well. I got it a couple of times myself but thought it was due to my nix setup. I will look deeper into it.

…s used

volkm · 2025-05-09T19:12:55Z

I restructured the building of the modules to avoid code duplication.
The .so files should also be installed now and at least the error message is different now:

_______________ ERROR collecting tests/core/test_bisimulation.py _______________
ImportError while importing test module '/opt/stormpy/tests/core/test_bisimulation.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/core/test_bisimulation.py:1: in <module>
    import stormpy
../venv/lib/python3.12/site-packages/stormpy/__init__.py:6: in <module>
    from ._config import *
../venv/lib/python3.12/site-packages/stormpy/_config.py:3: in <module>
    from stormpy import pycarl
../venv/lib/python3.12/site-packages/stormpy/pycarl/__init__.py:6: in <module>
    from . import pycarl_core
E   ImportError: libcarl.so.14.28: cannot open shared object file: No such file or directory

volkm · 2025-05-09T20:16:37Z

@sjunges Can you check at some point whether you can build it now?

volkm · 2025-06-06T13:15:50Z

Can we already merge this PR? Then we can add the remaining changes from #229 in a second step.

linusheck · 2025-06-06T13:59:15Z

Works for me!

volkm · 2025-06-06T14:02:07Z

Great to hear and thanks for testing!

volkm added 7 commits March 23, 2025 15:03

Use scikit-build-core for build instead of setuptools

3e4fa46

Compile with CCache

6e9f9db

Bump Black Python versions

c97b9a3

Enable builds for pycarl.cln and pycarl.parse only if dependencies ar…

b404a2a

…e available

Adapted Dockerfile and CI to new setup

c4a1115

Explicitly set hint for carl-parser

086cafa

Updated documentation to new build steps

d6638e8

sjunges reviewed Mar 24, 2025

View reviewed changes

Comment thread CMakeLists.txt Outdated

sjunges reviewed Mar 24, 2025

View reviewed changes

Comment thread doc/source/installation.rst Outdated

Comments by sjunges

cd4d8cf

Update CMakeLists.txt

d36fbf6

volkm added 6 commits May 9, 2025 20:49

Output paths for Storm, carl and carlparser and check whether hint wa…

145b730

…s used

Helper function for building stormpy modules

381364f

Restructured build

6e6f479

Decreased number of configuration files

261a7d6

Install modules

1c0eaf0

Build carl-parser portable

b6479d6

Set rpath

76439b1

volkm added this to the 1.11 milestone May 27, 2025

volkm added 2 commits June 6, 2025 14:07

Merge from master

bf5c8d9

Added Deprecated as dependency

be00aec

volkm merged commit b61c9ed into stormchecker:master Jun 6, 2025
10 checks passed

volkm deleted the setup branch June 6, 2025 14:04

volkm mentioned this pull request Aug 20, 2025

Duplicated _config information #261

Closed

Conversation

volkm commented Mar 24, 2025 • edited by sjunges Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sjunges commented Mar 24, 2025

Uh oh!

linusheck commented Mar 25, 2025

Uh oh!

volkm commented Mar 25, 2025

Uh oh!

sjunges commented Mar 29, 2025

Uh oh!

sjunges commented Mar 29, 2025

Uh oh!

sjunges commented Mar 29, 2025

Uh oh!

sjunges commented Mar 29, 2025

Uh oh!

sjunges commented Mar 29, 2025

Uh oh!

volkm commented Mar 29, 2025

Uh oh!

sjunges commented May 5, 2025

Uh oh!

volkm commented May 6, 2025

Uh oh!

volkm commented May 9, 2025

Uh oh!

volkm commented May 9, 2025

Uh oh!

volkm commented Jun 6, 2025

Uh oh!

linusheck commented Jun 6, 2025

Uh oh!

volkm commented Jun 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

volkm commented Mar 24, 2025 •

edited by sjunges

Loading