Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/contributing/contributing-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ Currently, depending on the flags used, `spacegrep` is invoked both independentl

## Making a change

Semgrep runs on Python versions >= 3.8. If you don't have one of these versions installed, please do so before proceeding.
Semgrep runs on Python versions >= 3.10. If you don't have one of these versions installed, please do so before proceeding.

Because the Python and OCaml development paths are relatively independent, the instructions are divided into Python ([semgrep-cli contributing](semgrep-contributing.md)) and OCaml ([semgrep-core contributing](semgrep-core-contributing.md)).

Expand Down
73 changes: 31 additions & 42 deletions docs/contributing/semgrep-contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ The `semgrep-cli` name refers to the project that exposes the actual `semgrep` c
## Prerequisite

- Python >= 3.10 installed in your local machine.
- [`pipenv`](https://github.com/pypa/pipenv) for managing your virtual
- [`uv`](https://github.com/astral-sh/uv) for managing your virtual
environment.
- Install it by following the `pipenv` [documentation](https://pipenv.pypa.io/en/latest/installation.html).
- Ensure `pipenv` is on your `$PATH` before proceeding.
- Install it by following the `uv` [documentation](https://docs.astral.sh/uv/getting-started/installation/)
- Ensure `uv` is on your `$PATH` before proceeding.

## Set up the environment

Expand All @@ -20,25 +20,22 @@ Most Python development is done inside the `cli` directory:
cd cli
```

Next, initialize and enter the virtual environment. The following command installs developer dependencies, such as `pytest`, and installs `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command:
Next, initialize the virtual environment. The following commands installs both the required dependencies and the developer dependencies, such as `pytest`, and installs `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Next, initialize the virtual environment. The following commands installs both the required dependencies and the developer dependencies, such as `pytest`, and installs `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command:
Next, initialize the virtual environment. The following commands install both the required dependencies and the developer dependencies, such as `pytest`, and install `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command:


```bash
pipenv shell
uv sync
uv build
uv pip install dist/*.whl
```

By convention, your shell prompt is prepended with `(cli)` when the virtual environment is active.

Next, install the Python dependencies:
To execute `semgrep` in this virtual environiment, you could activate the virtual environment as follows:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To execute `semgrep` in this virtual environiment, you could activate the virtual environment as follows:
To execute `semgrep` in the virtual environment, activate the virtual environment as follows:


```bash
SEMGREP_SKIP_BIN=true pipenv install --dev
source .venv/bin/activate
semgrep --help
```

:::info
`SEMGREP_SKIP_BIN` tells the installer that you'll use your own `semgrep-core`; see below.*
:::

Running `which semgrep` should return a path within your virtual environment. On macOS, this is likely contained within `$HOME/.local/share/virtualenvs/`.
Running `which semgrep` should return a path within your local virtual environment. On macOS, this is likely contained within `cli/.venv/bin/`.

## Get the `semgrep-core` binary

Expand Down Expand Up @@ -66,36 +63,28 @@ export PATH="${SEMGREP_BREW_CORE_BINARY_PATH}:${PATH}"

## Run `semgrep-cli`

Ensure that you are in the `cli/` directory, and then issue the following command:
Ensure that you are in the `cli/` directory and that you have activated the virtual environment with the installed wheel, then issue the following command:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ensure that you are in the `cli/` directory and that you have activated the virtual environment with the installed wheel, then issue the following command:
Ensure that you are in the `cli/` directory and that you have activated the virtual environment with the installed wheel. Then, run the following command:


```
pipenv run semgrep --help
```bash
semgrep --help
```

To try a simple analysis, run:

```
```bash
echo 'if 1 == 1: pass' | semgrep --lang python --pattern '$X == $X' -
```

You now have Semgrep running locally.

## Install `semgrep`

You can always run `semgrep` from `cli/`, which will use your latest changes in that directory, but you may also want to install the `semgrep` binary. To do this, run

```
pipenv install --dev
```

If you encounter difficulties, reach out to the [`semgrep` team on Slack](https://go.semgrep.dev/slack).

Now you can run `semgrep --help` from anywhere.

If you have installed `semgrep-core` from source, there are convenient targets in the root Makefile that let you update all binaries. After you pull, run:

```
make rebuild
```bash
make
```

See the Makefile in `cli/`
Expand All @@ -104,9 +93,9 @@ See the Makefile in `cli/`

Semgrep uses `mypy` to do static type-checking of its Python code. Therefore, when adding a new Python package, you also need to add typing stubs for that package. This can be done in three steps. For example, suppose you are adding the package `pyyaml` to Semgrep.

1. Install the corresponding package with typing stubs. For this `pyyaml` example, the corresponding package is `types-pyyaml`. In the following command, `--dev` specifies that this package is needed for development but not in production. This command updates `cli/Pipfile` with the typing stubs package, and adds both the typing stubs and the package itself to your `Pipfile.lock`. This allows you to import the package in your code (for example, `import yaml as pyyaml`).
1. Install the corresponding package with typing stubs. For this `pyyaml` example, the corresponding package is `types-pyyaml`. In the following command, `--group dev` specifies that this package is needed for development but not in production. This command updates `cli/pyproject.toml` with the typing stubs package, and adds both the typing stubs and the package itself to your `uv.lock`. This allows you to import the package in your code (for example, `import yaml as pyyaml`).
```
pipenv install --dev types-pyyaml
uv add --group dev types-pyyaml
```
2. Add the typing stubs package to `.pre-commit-config.yaml` so that the pre-commit `mypy` hook can find the package.
```
Expand All @@ -115,9 +104,9 @@ Semgrep uses `mypy` to do static type-checking of its Python code. Therefore, wh
- ...
- types-PyYAML
```
3. Add the original package to `cli/setup.py` in the `install_requires` list variable. You can find the version number either in the `Pipfile.lock` file or by looking up the most recent major version of the package online.
3. Add the original package to `cli/pyproject.toml` in the `dependencies` list. You can find the version number either in the `Pipfile.lock` file or by looking up the most recent major version of the package online.
```
install_requires = [
dependencies = [
...
"pyyaml~=6.0",
]
Expand All @@ -130,7 +119,7 @@ This change makes your package a dependency of published Semgrep. Without this c
For a reference build that's known to work, consult the root `Dockerfile`
to build Semgrep inside a container. You can check that it builds with

```
```sh
docker build -t semgrep .
```

Expand All @@ -140,14 +129,14 @@ docker build -t semgrep .

To run tests, run the following command:

```
pipenv run pytest
```sh
uv run pytest
```

There are some much slower tests that run Semgrep on many open source projects. To run these slow tests, run:

```sh
pipenv run pytest tests/qa
uv run pytest tests/qa
```

If you want to update the tests to match the current output:
Expand All @@ -158,19 +147,19 @@ make regenerate-tests

If you want to run a single test file:

```
pipenv run pytest path/to/test.py
```sh
uv run pytest path/to/test.py
```

Or run an individual test function:

```
pipenv run pytest path/to/test.py::test_func_name
```sh
uv run pytest path/to/test.py::test_func_name
```

`semgrep-cli` also includes [`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/en/latest/)
to allow for basic benchmarking functionality. Run the following command:

```
pipenv run pytest --benchmark-only
```sh
uv run pytest --benchmark-only
```
36 changes: 8 additions & 28 deletions docs/contributing/semgrep-core-contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,6 @@ brew install pkg-config bash
Lastly, you will almost certainly want the Python environment for `semgrep-cli`
configured before proceeding. Please refer to the [Set up the environment](/contributing/semgrep-contributing#set-up-the-environment) documentation.

Once you've returned here, ensure that your shell is able to enter the Python
virtual environment.

```bash
cd cli; pipenv shell # enter the virtual environment
cd .. # from within the virtual environment, return to the repo root
```

### First-time installation

The root `Makefile` contains targets that take care of building the
Expand Down Expand Up @@ -83,16 +75,10 @@ Unless there is a significant dependency change, you won't need to run `make dev

The Semgrep team has provided useful targets to help you build and link the entire semgrep project, including both `semgrep-core` and `semgrep`. You may find these helpful.

To install the latest OCaml binaries and `semgrep` binary after pulling source code changes from Git, run:
To build the latest OCaml binaries and `semgrep` binary after pulling source code changes from Git, run:

```
make rebuild
```

To install after you make a change locally, run

```
make build # or just `make`
```sh
make
```

After making either of these targets, `semgrep` runs with all your local changes, OCaml and Python both.
Expand All @@ -115,12 +101,6 @@ git submodule update --recursive

This will update internal dependencies. (We suggest aliasing it to `uu`)

After `tree-sitter` is updated, you may need to reconfigure it. If so, run

```
make config
```

### Develop `semgrep-core`

If you are developing `semgrep-core`, Use `Makefile` in the repository root for `core` and `core-test` targets; the code is primarily in `src/`.
Expand All @@ -129,25 +109,25 @@ The following assumes you are in the repository root.

After you pull or make a change, compile using

```
```sh
make
```

This will build an executable for `semgrep-core` in `_build/default/src/main/Main.exe` (we suggest aliasing this to `sc`). Try it out by running

```
```sh
_build/default/src/main/Main.exe -help
```

When you are done, test your changes with

```
```sh
make core-test
```

Finally, to update the `semgrep-core` binary used by `semgrep`, run

```
```sh
make copy-core-for-cli
```

Expand Down Expand Up @@ -325,7 +305,7 @@ information, for example:
```bash
export SEMGREP_CORE_DEBUG=1
export SEMGREP_CORE_PROFILE=1
pipenv run semgrep -f ../semgrep-core/tests/PERF/ajin.yaml ../semgrep-core/tests/PERF/three.js
uv run semgrep -f ../semgrep-core/tests/PERF/ajin.yaml ../semgrep-core/tests/PERF/three.js
```
will output:
```bash
Expand Down