deptry fails to parse requirements.txt files with filepath entries in

[sambrown3-nhs]

Environment

• deptry version: 0.24.0
• Python version: 3.11
• Operating system (e.g. Ubuntu 22.04, Windows 11): Ubuntu 22.04

Describe the issue

Our requirements.txt file contains a filepath to a locally held python wheel file, as is specified as an option in the Requirements File Format specification, but this causes deptry to bail out with a crashdump when trying to parse this file.

Sample Output:

$ deptry . --verbose
Running with the following configuration:
root: (PosixPath('.'),)
config: pyproject.toml
no_ansi: False
per_rule_ignores: {}
ignore: ()
exclude: ('venv', '\\.venv', '\\.direnv', 'tests', '\\.git', 'setup\\.py')
extend_exclude: ()
using_default_exclude: True
ignore_notebooks: False
requirements_files: ('requirements.txt',)
using_default_requirements_files: False
requirements_files_dev: ('ide-requirements-dev.txt',)
known_first_party: ()
json_output: None
package_module_name_map: {}
pep621_dev_dependency_groups: ()
experimental_namespace_package: False
github_output: False
github_warning_errors: ()

pyproject.toml found!
pyproject.toml does not contain a [tool.poetry] section, so Poetry is not used to specify the project's dependencies.
pyproject.toml does not contain a [tool.uv.dev-dependencies] section, so uv is not used to specify the project's dependencies.
pyproject.toml does not contain a [tool.pdm.dev-dependencies] section, so PDM is not used to specify the project's dependencies.
pyproject.toml contains a [project] section, so PEP 621 is used to specify the project's dependencies.
pyproject.toml has the entry build-system.build-backend == 'setuptools.build_meta', so setuptoolsis used to specify the project's dependencies.
Scanning requirements.txt for dependencies
Traceback (most recent call last):
  File "/opt/project-env/lib/python3.11/site-packages/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/packaging/_parser.py", line 71, in _parse_requirement
    name_token = tokenizer.expect(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/packaging/_tokenizer.py", line 143, in expect
    raise self.raise_syntax_error(f"Expected {expected}")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/packaging/_tokenizer.py", line 168, in raise_syntax_error
    raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier
    ./basket/our_internal_package.whl
    ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/project-env/bin/deptry", line 7, in <module>
    sys.exit(deptry())
             ^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/cli.py", line 318, in deptry
    cli(max_content_width=column_size)
  File "/opt/project-env/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/cli.py", line 312, in cli
    ).run()
      ^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/core.py", line 58, in run
    dependencies_extract = dependency_getter.get()
                           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/dependency_getter/pep621/base.py", line 48, in get
    dependencies = self._get_dependencies()
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/dependency_getter/pep621/base.py", line 71, in _get_dependencies
    return get_dependencies_from_requirements_files(dependencies_files, self.package_module_name_map)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/deptry/dependency_getter/requirements_files.py", line 56, in get_dependencies_from_requirements_files
    itertools.chain(
  File "/opt/project-env/lib/python3.11/site-packages/deptry/dependency_getter/requirements_files.py", line 58, in <genexpr>
    get_dependencies_from_requirements_file(file_name, package_module_name_map, is_dev)
  File "/opt/project-env/lib/python3.11/site-packages/deptry/dependency_getter/requirements_files.py", line 74, in get_dependencies_from_requirements_file
    for requirement in requirements.parse(requirements_file_content):
  File "/opt/project-env/lib/python3.11/site-packages/requirements/parser.py", line 90, in parse
    yield Requirement.parse(line)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/requirements/requirement.py", line 274, in parse
    return cls.parse_line(line)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/requirements/requirement.py", line 240, in parse_line
    pkg_req = Req(line_without_comment)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-env/lib/python3.11/site-packages/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
    ./basket/our_internal_package.whl
    ^

Minimal way to reproduce the issue

Download any python wheel, and add a path to that file to a requirements.txt file. When deptry parses that reqs file, it fails to do so.

Expected behavior

Deptry should parse the requirements file successfully, and run as normal

Additional context

When I remove that path from our requirements file, deptry successfully runs, but throws a lot of DEP003 'our_internal_package' imported but it is a transitive dependency as it hasn't detected that our_internal_package is specified as a wheel file.

[mkniewallner]

We use requirements-parser to parse requirements files, which indeed crashes on those (madpah/requirements-parser#76). Handling that upstream would be nice. Alternatively, we still have the option to manually remove dependencies that requirements-parser does not support on our side before calling it.

[mkniewallner]

Note that even if requirements-parser would not crash on the dependency definition, it would not be able to determine the name of the package (thus neither would deptry), but you should be able to do this:

# Assuming your package name is "our-internal-package"
our-internal-package @ file://./our_internal_package.whl

<PACKAGE_NAME> @ has to match the actual name of your package, and this will allow deptry to correctly retrieve the dependency.

[sambrown3-nhs]

# Assuming your package name is "our-internal-package"
our-internal-package @ file://./our_internal_package.whl

<PACKAGE_NAME> @ has to match the actual name of your package, and this will allow deptry to correctly retrieve the dependency.

Thanks for this, feels like I spent most of yesterday afternoon looking for this syntax. I was aware you could use @ for URLs, I'm not sure why I didn't put together that it could be used for file references too!

EDIT:

Ah, That's worked for running deptry, but pip doesn't accept that format (see pypa/pip#6658 ). I'll be able to put in a bit of file manipulation as part of my build step that runs Deptry to make this work though, so I'm able to use the tool now.

[mkniewallner]

Ah, That's worked for running deptry, but pip doesn't accept that format (see pypa/pip#6658 ). I'll be able to put in a bit of file manipulation as part of my build step that runs Deptry to make this work though, so I'm able to use the tool now.

Oh sorry for that, I tested that locally before suggesting it, but I must have used uv pip install (which does support that) instead of pip install by mistake when testing.

madpah/requirements-parser#54 suggests something that, while more of a hack, could work:

./our_internal_package.whl #egg=our-internal-package