pg_config executable not found

[firmianay]

#0 39.46 Collecting psycopg2-binary==2.8.4
#0 39.60   Downloading psycopg2-binary-2.8.4.tar.gz (378 kB)
#0 40.40      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 378.7/378.7 kB 471.7 kB/s eta 0:00:00
#0 40.42   Preparing metadata (setup.py): started
#0 40.52   Preparing metadata (setup.py): finished with status 'error'
#0 40.53   error: subprocess-exited-with-error
#0 40.53   
#0 40.53   × python setup.py egg_info did not run successfully.
#0 40.53   │ exit code: 1
#0 40.53   ╰─> [23 lines of output]
#0 40.53       running egg_info
#0 40.53       creating /tmp/pip-pip-egg-info-wtffupsx/psycopg2_binary.egg-info
#0 40.53       writing /tmp/pip-pip-egg-info-wtffupsx/psycopg2_binary.egg-info/PKG-INFO
#0 40.53       writing dependency_links to /tmp/pip-pip-egg-info-wtffupsx/psycopg2_binary.egg-info/dependency_links.txt
#0 40.53       writing top-level names to /tmp/pip-pip-egg-info-wtffupsx/psycopg2_binary.egg-info/top_level.txt
#0 40.53       writing manifest file '/tmp/pip-pip-egg-info-wtffupsx/psycopg2_binary.egg-info/SOURCES.txt'
#0 40.53       
#0 40.53       Error: pg_config executable not found.
#0 40.53       
#0 40.53       pg_config is required to build psycopg2 from source.  Please add the directory
#0 40.53       containing pg_config to the $PATH or specify the full executable path with the
#0 40.53       option:
#0 40.53       
#0 40.53           python setup.py build_ext --pg-config /path/to/pg_config build ...
#0 40.53       
#0 40.53       or with the pg_config option in 'setup.cfg'.
#0 40.53       
#0 40.53       If you prefer to avoid building psycopg2 from source, please install the PyPI
#0 40.53       'psycopg2-binary' package instead.
#0 40.53       
#0 40.53       For further information please check the 'doc/src/install.rst' file (also at
#0 40.53       <http://initd.org/psycopg/docs/install.html>).
#0 40.53       
#0 40.53       [end of output]
#0 40.53   
#0 40.53   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 40.53 error: metadata-generation-failed
#0 40.53 
#0 40.53 × Encountered error while generating package metadata.
#0 40.53 ╰─> See above for output.
#0 40.53 
#0 40.53 note: This is an issue with the package mentioned above, not pip.
#0 40.53 hint: See above for details.
------
failed to solve: process "/bin/sh -c python3 -m pip install /opencve/" did not complete successfully: exit code: 1

[ldurnez]

Closing, cf #38 answer.

[nreisbeck]

This is reproducible on aarch64

@firmianay are you using an M1/M2 Mac?
@ldurnez can you confirm that you were not using an M1/M2 Mac?

I was able to verify that when using DOCKER_DEFAULT_PLATFORM=linux/amd64 docker compose build that I was able to run everything, on the master branch as well as the tagged v1.4.0 release.

$ docker info

Client:
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  compose: Docker Compose (Docker Inc., 2.12.2)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

 Server Version: 20.10.23
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64

$ docker compose build

[+] Building 60.4s (14/17)                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                         0.0s
 => => transferring dockerfile: 1.15kB                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim-buster                                                                                    2.0s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                0.0s
 => [internal] load build context                                                                                                                            0.0s
 => => transferring context: 505B                                                                                                                            0.0s
 => [base 1/1] FROM docker.io/library/python:3.8-slim-buster@sha256:3fe962ec7350d8af5345913206e30731d96198945147c57c20b4965285b94882                        12.4s
 => => resolve docker.io/library/python:3.8-slim-buster@sha256:3fe962ec7350d8af5345913206e30731d96198945147c57c20b4965285b94882                              0.0s
 => => sha256:a8e11a7826c6ce9b3366e1d2c4dfbdec39133e62fe15bca830f2947fd1064b6c 2.65MB / 2.65MB                                                               8.7s
 => => sha256:0c86d6539fb88bb8980d2ba18d417e8f62dcc3f8b33dc75c0c15d3f3dddcc90d 10.97MB / 10.97MB                                                            11.6s
 => => sha256:3fe962ec7350d8af5345913206e30731d96198945147c57c20b4965285b94882 988B / 988B                                                                   0.0s
 => => sha256:c128dd81d790bc785444941dd356ffb405738c251080dec5822e7562a2431057 1.37kB / 1.37kB                                                               0.0s
 => => sha256:7adc90e60de2524ac763a0da876eb8b2a7da13e0517e1f9bbab371a77fe49b67 6.95kB / 6.95kB                                                               0.0s
 => => sha256:cfe0f778e53ce031ef449bc02975a07f2568716dea807ed2f390d352c0001972 25.92MB / 25.92MB                                                             5.2s
 => => extracting sha256:cfe0f778e53ce031ef449bc02975a07f2568716dea807ed2f390d352c0001972                                                                    0.9s
 => => sha256:3fb5a24eaa70bff488f5852c80622b9e598274fe6dbf212503fcfc1d147a1f80 244B / 244B                                                                   5.6s
 => => sha256:28fd5eb3414c5315232b446139d48461239367110c188fa0f1ee2368035d17f5 3.20MB / 3.20MB                                                               6.9s
 => => extracting sha256:a8e11a7826c6ce9b3366e1d2c4dfbdec39133e62fe15bca830f2947fd1064b6c                                                                    0.1s
 => => extracting sha256:0c86d6539fb88bb8980d2ba18d417e8f62dcc3f8b33dc75c0c15d3f3dddcc90d                                                                    0.3s
 => => extracting sha256:3fb5a24eaa70bff488f5852c80622b9e598274fe6dbf212503fcfc1d147a1f80                                                                    0.0s
 => => extracting sha256:28fd5eb3414c5315232b446139d48461239367110c188fa0f1ee2368035d17f5                                                                    0.2s
 => [stage-2 1/3] RUN apt-get update && apt-get upgrade -y     && apt-get clean     && rm -rf /var/lib/apt/lists/*                                          12.0s
 => [builder 1/8] RUN apt-get update && apt-get upgrade -y && apt-get install -y     git     && rm -rf /var/lib/apt/lists/*                                 14.6s
 => [builder 2/8] WORKDIR /opencve                                                                                                                           0.0s
 => [builder 3/8] RUN git clone --depth 1 -b v1.4.0 "https://github.com/opencve/opencve.git" . || git clone --depth 1 -b 1.4.0 "https://github.com/opencve/  2.9s
 => [builder 4/8] WORKDIR /app                                                                                                                               0.0s 
 => [builder 5/8] RUN python3 -m venv /app/venv                                                                                                              2.8s 
 => [builder 6/8] RUN python3 -m pip install --upgrade pip                                                                                                   5.0s 
 => ERROR [builder 7/8] RUN python3 -m pip install /opencve/                                                                                                20.6s 
------                                                                                                                                                            
 > [builder 7/8] RUN python3 -m pip install /opencve/:
#0 0.421 Processing /opencve
#0 0.422   Preparing metadata (setup.py): started
#0 0.595   Preparing metadata (setup.py): finished with status 'done'
#0 1.932 Collecting Flask==1.0.2
#0 2.284   Downloading Flask-1.0.2-py2.py3-none-any.whl (91 kB)
#0 2.630      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.4/91.4 kB 239.8 kB/s eta 0:00:00
#0 2.797 Collecting Flask-Admin==1.5.5
#0 2.892   Downloading Flask_Admin-1.5.5-py3-none-any.whl (1.9 MB)
#0 3.662      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 2.4 MB/s eta 0:00:00
#0 3.761 Collecting Flask-Assets==0.12
#0 3.826   Downloading Flask-Assets-0.12.tar.gz (22 kB)
#0 3.837   Preparing metadata (setup.py): started
#0 3.957   Preparing metadata (setup.py): finished with status 'done'
#0 4.074 Collecting Flask-DebugToolbar==0.10.1
#0 4.145   Downloading Flask_DebugToolbar-0.10.1-py2.py3-none-any.whl (326 kB)
#0 4.224      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 326.8/326.8 kB 4.2 MB/s eta 0:00:00
#0 4.388 Collecting Flask-SQLAlchemy==2.3.2
#0 4.455   Downloading Flask_SQLAlchemy-2.3.2-py2.py3-none-any.whl (16 kB)
#0 4.649 Collecting Flask-Migrate==3.0.1
#0 4.706   Downloading Flask_Migrate-3.0.1-py2.py3-none-any.whl (12 kB)
#0 4.783 Collecting Flask-Login==0.4.1
#0 4.856   Downloading Flask-Login-0.4.1.tar.gz (14 kB)
#0 4.894   Preparing metadata (setup.py): started
#0 5.009   Preparing metadata (setup.py): finished with status 'done'
#0 5.104 Collecting Flask-WTF==0.14.3
#0 5.169   Downloading Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
#0 5.235 Collecting Flask-User==1.0.2.2
#0 5.295   Downloading Flask-User-1.0.2.2.tar.gz (74 kB)
#0 5.313      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.9/74.9 kB 4.3 MB/s eta 0:00:00
#0 5.330   Preparing metadata (setup.py): started
#0 14.52   Preparing metadata (setup.py): finished with status 'done'
#0 14.62 Collecting Flask-Gravatar==0.5.0
#0 14.68   Downloading Flask_Gravatar-0.5.0-py2.py3-none-any.whl (7.5 kB)
#0 14.75 Collecting Flask-Paginate==0.5.2
#0 14.86   Downloading flask-paginate-0.5.2.tar.gz (5.5 kB)
#0 14.86   Preparing metadata (setup.py): started
#0 14.98   Preparing metadata (setup.py): finished with status 'done'
#0 15.07 Collecting Flask-RESTful==0.3.8
#0 15.12   Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB)
#0 16.43 Collecting Flask-Limiter==1.4
#0 16.50   Downloading Flask_Limiter-1.4-py3-none-any.whl (15 kB)
#0 17.03 Collecting SQLAlchemy==1.3.1
#0 17.13   Downloading SQLAlchemy-1.3.1.tar.gz (5.9 MB)
#0 18.96      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 3.2 MB/s eta 0:00:00
#0 19.33   Preparing metadata (setup.py): started
#0 19.56   Preparing metadata (setup.py): finished with status 'done'
#0 19.66 Collecting SQLAlchemy-Utils==0.33.11
#0 19.72   Downloading SQLAlchemy-Utils-0.33.11.tar.gz (128 kB)
#0 19.74      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.0/128.0 kB 5.6 MB/s eta 0:00:00
#0 19.77   Preparing metadata (setup.py): started
#0 19.92   Preparing metadata (setup.py): finished with status 'done'
#0 20.15 Collecting psycopg2-binary==2.8.4
#0 20.23   Downloading psycopg2-binary-2.8.4.tar.gz (378 kB)
#0 20.30      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 378.7/378.7 kB 5.7 MB/s eta 0:00:00
#0 20.33   Preparing metadata (setup.py): started
#0 20.45   Preparing metadata (setup.py): finished with status 'error'
#0 20.46   error: subprocess-exited-with-error
#0 20.46   
#0 20.46   × python setup.py egg_info did not run successfully.
#0 20.46   │ exit code: 1
#0 20.46   ╰─> [23 lines of output]
#0 20.46       running egg_info
#0 20.46       creating /tmp/pip-pip-egg-info-_r0a8th8/psycopg2_binary.egg-info
#0 20.46       writing /tmp/pip-pip-egg-info-_r0a8th8/psycopg2_binary.egg-info/PKG-INFO
#0 20.46       writing dependency_links to /tmp/pip-pip-egg-info-_r0a8th8/psycopg2_binary.egg-info/dependency_links.txt
#0 20.46       writing top-level names to /tmp/pip-pip-egg-info-_r0a8th8/psycopg2_binary.egg-info/top_level.txt
#0 20.46       writing manifest file '/tmp/pip-pip-egg-info-_r0a8th8/psycopg2_binary.egg-info/SOURCES.txt'
#0 20.46       
#0 20.46       Error: pg_config executable not found.
#0 20.46       
#0 20.46       pg_config is required to build psycopg2 from source.  Please add the directory
#0 20.46       containing pg_config to the $PATH or specify the full executable path with the
#0 20.46       option:
#0 20.46       
#0 20.46           python setup.py build_ext --pg-config /path/to/pg_config build ...
#0 20.46       
#0 20.46       or with the pg_config option in 'setup.cfg'.
#0 20.46       
#0 20.46       If you prefer to avoid building psycopg2 from source, please install the PyPI
#0 20.46       'psycopg2-binary' package instead.
#0 20.46       
#0 20.46       For further information please check the 'doc/src/install.rst' file (also at
#0 20.46       <http://initd.org/psycopg/docs/install.html>).
#0 20.46       
#0 20.46       [end of output]
#0 20.46   
#0 20.46   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 20.46 error: metadata-generation-failed
#0 20.46 
#0 20.46 × Encountered error while generating package metadata.
#0 20.46 ╰─> See above for output.
#0 20.46 
#0 20.46 note: This is an issue with the package mentioned above, not pip.
#0 20.46 hint: See above for details.
------
failed to solve: executor failed running [/bin/sh -c python3 -m pip install /opencve/]: exit code: 1

psycopg2-binary and psycopg2 do not appear to provide a binary for aarch64 based on PyPI Files available, at least not for the version pinned in the project's requirements.txt file.

This is also confirmed by discussion(s) here and here

https://pypi.org/project/psycopg2-binary/2.8.2/#files

I'll save the overly verbose output, but you can confirm that pip via setuptools will try to build psycopg2 from source because it could not get a arm64/aarch64 wheel with the following:

docker run -it -i python:3.8-slim-buster /bin/bash
pip install psycopg2-binary==2.8.2

# You can also confirm there is no aarch64 build for any 2.8.x with the below, which will attempt to install the last 2.8.x release of 2.8.6
pip install psycopg2-binary~=2.8.2

Until opencve/opencve itself upgrades their pscycopg2 version pin to 2.9 you have a couple of options, given @ldurnez's reason for closing your PR, which I must admit, I don't entirely understand (or agree with), if you add libpq-dev and gcc to the build stage, they're not carried over into the final image... That's entirely the whole point of a multi-stage docker build. Although I will concede that including libpq-dev and gcc in the Dockerfile solely for aarch64 reasons is a bit of a kludge, but it would have been a nice gesture.

Option 1: fork this repo, and make the fixes to your own Dockerfile => build & run

Option 2: prefix your docker compose build command with DOCKER_DEFAULT_PLATFORM=linux/amd64 to force the artifact to be built with a target architecture of amd64. Warning: This will be slower to build with QEMU

Option 3: I have a PR on a branch ready to review & merge that will split this Dockerfile into a more robust multi-stage multi-arch build, such that when running docker compose build locally on a linux/amd64 device (read: Macbook Pro w/ Intel chip the artifact built will be linux/amd64 and when run on Apple Silicon (M1/M2) it will build a linux/arm64 artifact, and pull the required sources & libs to successfully build psycopg2-binary.

@ldurnez would you entertain another PR to update the Dockerfile to add arm64 support?
PR #41