Merge pull request #278 from ghantoos/f/engine-v2

Migrates lshell to the canonical engine/config pipeline and updates Python support policy.

Author: ghantoos

.github/workflows/lshell-tests.yml

@@ -14,13 +14,17 @@ jobs:
   pytest:
     name: Pytest Unit/Integration Tests
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.14"]
 
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python 3.10
+    - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: ${{ matrix.python-version }}
     - name: Set up Python path
       run: echo "PYTHONPATH=$PWD" >> $GITHUB_ENV
     - name: Install dependencies
@@ -38,8 +42,9 @@ jobs:
     name: Lint + Flake8
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: ["3.8", "3.9", "3.10"]
+        python-version: ["3.10", "3.14"]
 
     steps:
     - uses: actions/checkout@v4

CHANGELOG.md

@@ -3,6 +3,18 @@
 Contact: [ghantoos@ghantoos.org](mailto:ghantoos@ghantoos.org)  
 [https://github.com/ghantoos/lshell](https://github.com/ghantoos/lshell)
 
+### v0.12.0 (UNRELEASED)
+- Packaging/CI: Raised minimum supported Python version to 3.10 (`requires-python >=3.10`), removed EOL Python versions from CI, and aligned docs/package metadata with the new baseline; CI/classifiers now track active CPython release branches 3.10-3.14 (Python 3.6 reached EOL on 23/12/2021).
+- Security: Removed regex-driven shell parsing from the authorization flow.
+- Engine: Migrated security parsing to a deterministic scanner.
+- Refactor: Removed legacy `lshell.parser` compatibility wrapper; runtime and diagnostics now rely on canonical `lshell.engine.*` parsing paths only.
+- Refactor: Reorganized configuration code into `lshell/config/` with focused modules (`runtime.py`, `diagnostics.py`, `resolve.py`, `schema.py`) and updated imports accordingly.
+- Config: Unified runtime (`CheckConfig`) and diagnostics (`policy-show`) merge logic into a shared resolver to keep section precedence, include overlays, +/- list operations, `all` expansion, and glob-path handling aligned.
+- CLI: Restored split diagnostics naming: `policy-show` as the external subcommand and `lshow` as the in-shell builtin.
+- CLI: Removed legacy/extra diagnostics commands `lpath`, `lsudo`, `policy-path`, and `policy-sudo`.
+- UX: Extended `policy-show` output to include path allow/deny details and sudo policy details directly.
+- Tests: Added runtime-vs-diagnostics parity coverage for precedence, include overlays, `allowed=all` minus operations, and schema error behavior.
+
 ### v0.11.1 21/03/2026
 - Feature: Added `lshell setup-system` to provision logging paths/permissions and user/group wiring for deployments.
 - Feature: Added `lshell harden-init` with hardened templates (`sftp-only`, `rsync-backup`, `deploy-minimal`, `readonly-support`) plus `--dry-run`, scoped `[grp:*]`/`[user:*]`, and validation checks.
@@ -19,7 +31,7 @@ Contact: [ghantoos@ghantoos.org](mailto:ghantoos@ghantoos.org)
 
 ### v0.11.0 10/03/2026
 - Reworked command parsing with a new `pyparsing`-based parser for more reliable command handling.
-- Added policy diagnostics and built-ins: `policy-show`, `policy-path`, and `policy-sudo`.
+- Added policy diagnostics commands: `policy-show` (CLI) and `lshow` (in-shell).
 - Added customizable user-facing messages via the `messages` configuration section.
 - Added session `umask` configuration support from `lshell.conf` and CLI overrides.
 - Improved `sudo` behavior and command execution handling.
@@ -143,8 +155,8 @@ Contact: [ghantoos@ghantoos.org](mailto:ghantoos@ghantoos.org)
 
 ### v0.9.14 27/10/2010
 - Corrected `get_aliases` function, as it was looping when aliases were "recursive" (e.g. `ls:ls --color=auto`)
-- Added `lsudo` built-in command to list allowed sudo commands.
-- Corrected completion function when 2 strings collided (e.g. `ls` and `lsudo`)
+- Added diagnostics sudo-policy visibility for allowed sudo commands.
+- Corrected completion function when 2 strings collided (for example `ls` with diagnostics commands)
 - Corrected the README's installation part (adding `--prefix`).
 - Added possibility to log via syslog.
 - Corrected warning counter (was counting minus 1).
@@ -179,7 +191,7 @@ Contact: [ghantoos@ghantoos.org](mailto:ghantoos@ghantoos.org)
 - Added the possibility to configure introduction prompt.
 - Replaced "joker" by "warnings" (more elegant)
 - Possibility of limiting the history file size.
-- Added `lpath` built-in command to list allowed and denied path. Thanks to Adrien Urban.
+- Added diagnostics path-policy visibility for allowed and denied paths. Thanks to Adrien Urban.
 - Corrected bug when using `~` was not parsed as "home directory" when used in a command other than `cd`. Thank you Adrien Urban for finding this.
 - Corrected minor typo when warning for a forbidden path.
 - If `$(foo)` is present in the line, check if `foo` is allowed before executing the line. Thank you Adrien Urban for pointing this out!

README.md

@@ -10,6 +10,8 @@ PyPI project page: https://pypi.org/project/limited-shell/
 
 ## Installation
 
+Supported Python versions: `3.10`, `3.11`, `3.12`, `3.13`, `3.14` (minimum `3.10`).
+
 Install from PyPI:
 
 ```bash
@@ -95,9 +97,10 @@ lshell policy-show \
 
 Inside an interactive session:
 
-- `policy-show [<command...>]`
-- `policy-path` (`lpath` alias)
-- `policy-sudo` (`lsudo` alias)
+- `lshow [<command...>]`
+
+`lshow` includes effective command policy, allowed/denied paths, and sudo
+policy in one output.
 
 Hide these built-ins if needed:
 
@@ -296,6 +299,7 @@ just test-fuzz-security-parser 20000
 Optional local run (if you want to fuzz outside Docker):
 
 ```bash
+python3 --version  # should be >= 3.10
 pip install -r requirements-fuzz.txt
 python3 fuzz/fuzz_parser_policy.py -runs=20000
 ```

debian/control

@@ -2,8 +2,8 @@ Source: lshell
 Section: shells
 Priority: optional
 Maintainer: Ignace Mouzannar <mouzannar@gmail.com>
-Build-Depends: debhelper-compat (= 13), python3 (>= 3.4), pybuild-plugin-pyproject
-X-Python3-Version: >= 3.4
+Build-Depends: debhelper-compat (= 13), python3 (>= 3.10), pybuild-plugin-pyproject
+X-Python3-Version: >= 3.10
 Standards-Version: 3.9.7
 Homepage: https://github.com/ghantoos/lshell
 

debian/lshell.deb-test.conf

@@ -13,7 +13,7 @@ allowed_file_extensions : ['.conf', '.log', '.txt']
 aliases                 : {'ll': 'ls -l', 'la': 'ls -la'}
 env_vars                : {'LSHELL_LAYER': 'default', 'LSHELL_ENV': 'deb-test'}
 prompt_short            : 1
-intro                   : "\033[1;95mDEB Test Profile: layered default/group/user policy\033[0m\n\n\033[1;92mAllowed examples\033[0m\n\033[90m---------------\033[0m\nid\nwhoami\ndate\nll\ncat /home/testuser/lshell/test/testfiles/test.conf\npolicy-show\npolicy-show cat /home/testuser/lshell/test/testfiles/test.conf\n\n\033[1;91mForbidden examples\033[0m\n\033[90m------------------\033[0m\necho hello\ncat /etc/passwd\ncat /tmp/test.log\n"
+intro                   : "\033[1;95mDEB Test Profile: layered default/group/user policy\033[0m\n\n\033[1;92mAllowed examples\033[0m\n\033[90m---------------\033[0m\nid\nwhoami\ndate\nll\ncat /home/testuser/lshell/test/testfiles/test.conf\nlshow\nlshow cat /home/testuser/lshell/test/testfiles/test.conf\n\n\033[1;91mForbidden examples\033[0m\n\033[90m------------------\033[0m\necho hello\ncat /etc/passwd\ncat /tmp/test.log\n"
 messages                : {
                             'unknown_syntax': 'deb-test: unknown syntax -> {command}',
                             'forbidden_command': 'deb-test: command blocked -> "{command}"',
@@ -35,7 +35,7 @@ env_vars                : {'LSHELL_LAYER': 'group'}
 
 [testuser]
 # User layer applies on top of default + group for testuser.
-allowed                 : + ['cat', 'head', 'tail', 'policy-show'] - ['echo']
+allowed                 : + ['cat', 'head', 'tail', 'lshow'] - ['echo']
 path                    : + ['/home/testuser/lshell/test/testfiles'] - ['/home']
 overssh                 : + ['ls'] - ['rsync']
 allowed_file_extensions : + ['.yaml'] - ['.txt']

docs/engine-migration.md

@@ -0,0 +1,59 @@
+# Canonical Engine Architecture
+
+## Overview
+
+Lshell now uses a single canonical command engine for both runtime execution
+and policy diagnostics. There is no legacy engine toggle or fallback path.
+
+## Pipeline
+
+The shared flow is:
+
+1. `parse(line)` -> `lshell.engine.ast.ParsedAST`
+2. `normalize(parsed_ast)` -> `lshell.engine.ast.CanonicalAST`
+3. `authorize(canonical_ast, policy)` -> structured allow/deny decision
+4. `execute(decisions, runtime)` -> retcode + audit outcome
+
+Runtime entrypoint:
+
+- `lshell.utils.cmd_parse_execute` -> `lshell.engine.executor.execute_for_shell`
+
+Policy diagnostics entrypoint:
+
+- `lshell.config.diagnostics.policy_command_decision` -> canonical authorizer path
+
+## Engine Modules
+
+- `lshell/engine/ast.py`: canonical AST structures.
+- `lshell/engine/parser.py`: top-level command/operator sequence parser.
+- `lshell/engine/normalizer.py`: canonical command normalization and assignment splitting.
+- `lshell/engine/authorizer.py`: centralized command/path/security authorization.
+- `lshell/engine/reasons.py`: structured reason codes and user/audit mappings.
+- `lshell/engine/executor.py`: runtime execution with strict-mode and audit semantics.
+
+## Reason Codes
+
+Primary reason codes are defined in `lshell.engine.reasons`, including:
+
+- `allowed`
+- `unknown_syntax`
+- `forbidden_control_char`
+- `forbidden_character`
+- `forbidden_path`
+- `forbidden_command`
+- `forbidden_sudo_command`
+- `forbidden_file_extension`
+- `forbidden_env_assignment`
+- `forbidden_trusted_protocol`
+- `command_not_found`
+
+Mappings:
+
+- `to_policy_message(reason)` for policy-show text.
+- `to_audit_reason(reason)` for runtime audit logging.
+- `warning_payload(reason)` for strict/warning-counter compatibility hooks.
+
+## Validation
+
+- Unit tests: parser/normalizer/authorizer behavior.
+- Security regressions: parser smuggling, substitution checks, and path ACL edge cases.

etc/lshell.conf

@@ -223,6 +223,6 @@ strict          : 0
 #disable_exit      : 0
 
 ## show/hide policy introspection builtins:
-## policy-show, policy-path, policy-sudo (+ aliases lpath, lsudo)
+## lshow
 ## set to 0 to hide these commands from users
 #policy_commands   : 0

fuzz/fuzz_parser_policy.py

@@ -12,8 +12,9 @@
     ) from exc
 
 with atheris.instrument_imports():
-    from lshell import parser as lshell_parser
-    from lshell import policy
+    from lshell.engine import normalizer as engine_normalizer
+    from lshell.engine import parser as engine_parser
+    from lshell.config import diagnostics as policy
     from lshell import sec
     from lshell import utils
 
@@ -39,8 +40,6 @@ def info(self, _message):
 
 
 _FUZZ_TMP = tempfile.mkdtemp(prefix="lshell-fuzz-")
-_FUZZ_PARSER = lshell_parser.LshellParser()
-
 
 def _base_conf():
     """Build an isolated, permissive config for parser/policy fuzz entrypoints."""
@@ -80,9 +79,8 @@ def _fuzz_one_line(line):
         "path": conf["path"],
     }
     try:
-        parsed = _FUZZ_PARSER.parse(line)
-        if parsed is not None:
-            _FUZZ_PARSER.validate_command(parsed)
+        parsed = engine_parser.parse(line)
+        engine_normalizer.normalize(parsed)
 
         utils.split_command_sequence(line)
         utils.split_commands(line)

lshell/builtincmd.py

@@ -3,25 +3,21 @@
 import glob
 import sys
 import os
-import re
 import shlex
 import readline
 import signal
 
 # import lshell specifics
 from lshell import variables
 from lshell import utils
+from lshell import sec as sec_policy
 
 
 # Store background jobs
 BACKGROUND_JOBS = []
 
 POLICY_COMMANDS = [
-    "policy-show",
-    "policy-path",
-    "policy-sudo",
-    "lpath",
-    "lsudo",
+    "lshow",
 ]
 
 builtins_list = [
@@ -31,11 +27,7 @@
     "exit",
     "export",
     "history",
-    "policy-show",
-    "policy-path",
-    "policy-sudo",
-    "lpath",
-    "lsudo",
+    "lshow",
     "help",
     "fg",
     "bg",
@@ -54,12 +46,15 @@ def _cancel_job_timeout(job):
 def cmd_lpath(conf):
     """Show path policy in a concise, readable format."""
     current_dir = os.path.realpath(os.getcwd())
-    current_match = current_dir if current_dir.endswith("/") else f"{current_dir}/"
-    allowed_re = str(conf["path"][0])
-    denied_re = str(conf["path"][1][:-1])
-    current_allowed = bool(re.findall(allowed_re, current_match))
-    current_denied = bool(re.findall(denied_re, current_match)) if denied_re else False
-    current_status = "allowed" if current_allowed and not current_denied else "denied"
+    path_acl = conf.get("path", ["", ""])
+    allow = path_acl[0] if len(path_acl) > 0 else ""
+    deny = path_acl[1] if len(path_acl) > 1 else ""
+    allowed_roots = sec_policy._split_path_acl_entries(allow)
+    denied_roots = sec_policy._split_path_acl_entries(deny)
+    current_allowed = sec_policy._is_path_allowed(
+        current_dir, allowed_roots, denied_roots
+    )
+    current_status = "allowed" if current_allowed else "denied"
 
     sys.stdout.write("Path Policy\n")
     sys.stdout.write("-----------\n")

lshell/cli.py

@@ -6,12 +6,12 @@
 import sys
 import uuid
 
-from lshell import policy as policy_mode
+from lshell.config import diagnostics as policy_mode
 from lshell import systemsetup as system_setup
 from lshell import hardeninit as harden_init
 from lshell import audit
 from lshell import containment
-from lshell.checkconfig import CheckConfig
+from lshell.config.runtime import CheckConfig
 from lshell.shellcmd import LshellTimeOut, ShellCmd