Can't create the conda env in Kestrel

[rajeee]

Installation

conda env update -f geothermal_env.yml

Failed with:

Collecting blinker==1.8.2 (from -r /home/radhikar/weather_regression/buildstock-weatherreg/condaenv._p4d360y.requirements.txt (line 4))
  Downloading blinker-1.8.2-py3-none-any.whl.metadata (1.6 kB)

Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement buildstock-query==2024.5.9 (from versions: none)
ERROR: No matching distribution found for buildstock-query==2024.5.9

failed

CondaEnvException: Pip failed

I tried again after removing buildstock-query from the requirement list (thinking that I can install it manually later). This again led to the following error:

(base) [radhikar@kl2 buildstock-weatherreg] CPU $ conda env create -f geothermal_env.yml
Collecting package metadata (repodata.json): done
Solving environment: done



... <removed for brevity> ...
  Using cached tensorboard-2.16.2-py3-none-any.whl.metadata (1.6 kB)
Collecting tensorboard-data-server==0.7.2 (from -r /home/radhikar/weather_regression/buildstock-weatherreg/condaenv.3wq5zbxt.requirements.txt (line 66))
  Using cached tensorboard_data_server-0.7.2-py3-none-any.whl.metadata (1.1 kB)
Collecting tensorflow==2.16.1 (from -r /home/radhikar/weather_regression/buildstock-weatherreg/condaenv.3wq5zbxt.requirements.txt (line 67))
  Using cached tensorflow-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.3 kB)

Pip subprocess error:
ERROR: Ignored the following versions that require a different python version: 1.1.1rc3 Requires-Python >=3.12; 1.1.1rc4 Requires-Python >=3.12; 1.4.2.220622 Requires-Python >=3.8,<3.11; 1.4.2.220626 Requires-Python >=3.8,<3.11; 1.4.3.220702 Requires-Python >=3.8,<3.11; 1.4.3.220703 Requires-Python >=3.8,<3.11; 1.4.3.220704 Requires-Python >=3.8,<3.11; 1.4.3.220710 Requires-Python >=3.8,<3.11; 1.4.3.220718 Requires-Python >=3.8,<3.11; 1.4.3.220724 Requires-Python >=3.8,<3.11; 1.4.3.220801 Requires-Python >=3.8,<3.11; 1.4.3.220807 Requires-Python >=3.8,<3.11; 1.4.3.220815 Requires-Python >=3.8,<3.11; 1.4.3.220822 Requires-Python >=3.8,<3.11; 1.4.3.220829 Requires-Python >=3.8,<3.11; 1.4.4.220906 Requires-Python >=3.8,<3.11; 1.4.4.220912 Requires-Python >=3.8,<3.11; 1.4.4.220919 Requires-Python >=3.8,<3.11; 1.5.0.220926 Requires-Python >=3.8,<3.11; 1.5.0.221003 Requires-Python >=3.8,<3.11; 1.5.0.221010 Requires-Python >=3.8,<3.11; 1.5.0.221012 Requires-Python >=3.8,<3.11; 1.5.1.221024 Requires-Python >=3.8,<3.11; 2.0.0 Requires-Python >=3.12; 2.0.1 Requires-Python >=3.12
ERROR: Could not find a version that satisfies the requirement tensorflow-intel==2.16.1 (from versions: 0.0.1)
ERROR: No matching distribution found for tensorflow-intel==2.16.1

failed

CondaEnvException: Pip failed

[joemowers]

Did you try pip install tensorflow? I wonder if this is the same issue that I encountered when creating the Quickstart.

[rajeee]

Doing pip install tensorflow results in this:

Installing collected packages: namex, libclang, flatbuffers, typing_extensions, termcolor, tensorboard-data-server, protobuf, opt_einsum, numpy, mdurl, markupsafe, markdown, google_pasta, gast, astunparse, absl-py, werkzeug, optree, ml_dtypes, markdown-it-py, h5py, grpcio, tensorboard, rich, keras, tensorflow
  Attempting uninstall: typing_extensions
    Found existing installation: typing_extensions 4.11.0
    Uninstalling typing_extensions-4.11.0:
      Successfully uninstalled typing_extensions-4.11.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.24.3
    Uninstalling numpy-1.24.3:
      Successfully uninstalled numpy-1.24.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
bokeh 3.7.3 requires Jinja2>=2.9, which is not installed.
bokeh 3.7.3 requires PyYAML>=3.10, which is not installed.
buildstockbatch 2024.11.0 requires lxml, which is not installed.
buildstockbatch 2024.11.0 requires pyyaml, which is not installed.
buildstockbatch 2024.11.0 requires semver, which is not installed.
resstockpostproc 0.1.0 requires plotly, which is not installed.
buildstock-query 0.1.0 requires PyYAML, which is not installed.
contourpy 1.2.0 requires numpy<2.0,>=1.20, but you have numpy 2.4.2 which is incompatible.
matplotlib 3.8.0 requires numpy<2,>=1.21, but you have numpy 2.4.2 which is incompatible.
scipy 1.10.0 requires numpy<1.27.0,>=1.19.5, but you have numpy 2.4.2 which is incompatible.
pandas 2.1.4 requires numpy<2,>=1.23.2; python_version == "3.11", but you have numpy 2.4.2 which is incompatible.
Successfully installed absl-py-2.4.0 astunparse-1.6.3 flatbuffers-25.12.19 gast-0.7.0 google_pasta-0.2.0 grpcio-1.76.0 h5py-3.15.1 keras-3.13.2 libclang-18.1.1 markdown-3.10.1 markdown-it-py-4.0.0 markupsafe-3.0.3 mdurl-0.1.2 ml_dtypes-0.5.4 namex-0.1.0 numpy-2.4.2 opt_einsum-3.4.0 optree-0.18.0 protobuf-6.33.5 rich-14.3.2 tensorboard-2.20.0 tensorboard-data-server-0.7.2 tensorflow-2.20.0 termcolor-3.3.0 typing_extensions-4.15.0 werkzeug-3.1.5

pip install --upgrade "numpy<2.0"

      Successfully uninstalled numpy-2.4.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
bokeh 3.7.3 requires Jinja2>=2.9, which is not installed.
bokeh 3.7.3 requires PyYAML>=3.10, which is not installed.
buildstockbatch 2024.11.0 requires lxml, which is not installed.
buildstockbatch 2024.11.0 requires pyyaml, which is not installed.
buildstockbatch 2024.11.0 requires semver, which is not installed.
resstockpostproc 0.1.0 requires plotly, which is not installed.
buildstock-query 0.1.0 requires PyYAML, which is not installed.
Successfully installed numpy-1.26.4

[rajeee]

Following the readme - after doing:
python -c "import site; print(site.getsitepackages())"
I get:
['/nopt/nrel/apps/cpu_stack/utilities/06-24/linux-rhel8-sapphirerapids/gcc-12.3.0/python-3.11.4-ynsdda7xpqgkyqbmriytqunnakdoym2i/lib/python3.11/site-packages']

But can't copy comstock_toml to there since buildstock-query doesn't exist there. Probably a quirk of Kestrel since I get this when install buildstock-query:

(/home/radhikar/.conda-envs/geothermal) [radhikar@kl2 buildstock-weatherreg] CPU $ pip install git+https://github.com/NREL/buildstock-query#egg=buildstock-query[full]
Defaulting to user installation because normal site-packages is not writeable

[rajeee]

(/home/radhikar/.conda-envs/geothermal) [radhikar@kl1 buildstock-weatherreg] CPU $ python B_building_stock_parallel_agg.py
Traceback (most recent call last):
  File "/home/radhikar/weather_regression/buildstock-weatherreg/B_building_stock_parallel_agg.py", line 7, in <module>
    import pandas as pd
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/__init__.py", line 46, in <module>
    from pandas.core.api import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/api.py", line 47, in <module>
    from pandas.core.groupby import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/groupby/generic.py", line 67, in <module>
    from pandas.core.frame import DataFrame
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/frame.py", line 142, in <module>
    from pandas.core.generic import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/generic.py", line 187, in <module>
    from pandas.core.window import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/window/__init__.py", line 1, in <module>
    from pandas.core.window.ewm import (
  File "/home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/core/window/ewm.py", line 11, in <module>
    import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/radhikar/.conda-envs/geothermal/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)

[mmowers]

Hi @rajeee, is buildstock-query==2024.5.9 no longer available? I get the same buildstock-query installation error that you have at the top of this issue when I install the conda environment from geothermal_env.yml.

Note that I was able to run B_building_stock_parallel_agg.py after following the installation instructions (ignoring installation errors). I ran into a different error in D_process_chunk_agg.py, perhaps because my version of buildstock-query (0.2.0) is different from the specified version (2024.5.9).

Show slurm output

INFO:buildstock_query.query_core:Loading ('resstock_amy2018_r1_2025_md_by_state_parquet', 'resstock_amy2018_r1_2025_ts_by_state', 'resstock_amy2018_r1_2025_md_by_state_parquet') ...
Script start time: 2026-02-04 10:27:32.389755
start_index: 0
end_index: 150
meta_path: /home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46/res_meta_master_upgrade0.csv
upgrade: 0
prefix: res_
output_dir: /home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46
script_dir: /home/mmowers/buildstock-weatherreg
counties_str: G5000010_G5000030_G5000050_G5000070_G5000090_G5000110_G5000130_G5000150_G5000170_G5000190_G5000210_G5000230_G5000250_G5000270
Script to rerun this file with the same arguments:
sbatch --job-name=chunk_res_0_0-150 ./C_run_bldg_chunk_agg.sh 0 150 /home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46/res_meta_master_upgrade0.csv 0 res_ /home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46 /home/mmowers/buildstock-weatherreg G5000010_G5000030_G5000050_G5000070_G5000090_G5000110_G5000130_G5000150_G5000170_G5000190_G5000210_G5000230_G5000250_G5000270
Traceback (most recent call last):
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/common.py", line 374, in _get_table_metadata
    response = retry_api_call(
               ^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/util.py", line 196, in retry_api_call
    return retry(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/tenacity/__init__.py", line 477, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/botocore/client.py", line 936, in _make_api_call
    request_dict = self._convert_to_request_dict(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/botocore/client.py", line 1010, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "WorkGroup", must be one of: CatalogName, DatabaseName, TableName

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46/inputs/D_process_chunk_agg.py", line 1027, in <module>
    ts_agg = process_chunk_agg(
             ^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/buildstock-weatherreg/outputs/outputs_2026-02-04-10-26-46/inputs/D_process_chunk_agg.py", line 297, in process_chunk_agg
    my_run = BuildStockQuery(**aws_run_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/buildstock_query/main.py", line 94, in __init__
    super(BuildStockQuery, self).__init__(params=self._run_params)
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/buildstock_query/query_core.py", line 151, in __init__
    self._initialize_tables()
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/buildstock_query/query_core.py", line 215, in _initialize_tables
    self.bs_table, self.ts_table, self.up_table = self._get_tables(self.table_name)
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/buildstock_query/query_core.py", line 314, in _get_tables
    baseline_table = self._get_table(f"{table_name[0]}")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/buildstock_query/query_core.py", line 251, in _get_table
    return self._tables.setdefault(table_name, sa.Table(table_name, self._meta, autoload_with=self._engine))
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in __new__
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 429, in __new__
    return cls._new(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 483, in _new
    with util.safe_reraise():
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 479, in _new
    table.__init__(name, metadata, *args, _no_init=False, **kw)  # type: ignore[misc] # noqa: E501
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 861, in __init__
    self._autoload(
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 893, in _autoload
    conn_insp.reflect_table(
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 1536, in reflect_table
    _reflect_info = self._get_reflection_info(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 2019, in _get_reflection_info
    columns=run(
            ^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 2005, in run
    res = meth(filter_names=_fn, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 931, in get_multi_columns
    table_col_defs = dict(
                     ^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 1136, in _default_multi_reflect
    single_tbl_method(
  File "<string>", line 2, in get_columns
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 106, in cache
    ret = fn(self, con, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/sqlalchemy/base.py", line 361, in get_columns
    metadata = self._get_table(connection, table_name, schema=schema, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in _get_table
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 106, in cache
    ret = fn(self, con, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/sqlalchemy/base.py", line 275, in _get_table
    return cursor.get_table_metadata(table_name, schema_name=schema, logging_=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/common.py", line 394, in get_table_metadata
    return self._get_table_metadata(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmowers/.conda-envs/geothermal/lib/python3.11/site-packages/pyathena/common.py", line 383, in _get_table_metadata
    raise OperationalError(*e.args) from e
pyathena.error.OperationalError: Parameter validation failed:
Unknown parameter in input: "WorkGroup", must be one of: CatalogName, DatabaseName, TableName

[mmowers]

@rajeee, I was able to successfully run the regression tool after installing the specific buildstock-query version with

pip install "buildstock-query[full] @ git+https://github.com/NatLabRockies/buildstock-query@2024.05.09"

I have updated the README accordingly.

[rajeee]

Yes, I am able to install buildstock-query using that command. However, I still get the same error above when trying to install the conda env, and get the same error about GLIBCXX_3.4.29 when trying to run the python B_building_stock_parallel_agg.py command.