Skip to content

Prevent file descriptor leak in util.just_run #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mariokostelac wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Prevent file descriptor leak in util.just_run #237

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9a877a1
Add a test ensuring that just_run doesn't leak fds
mariokostelac Jun 28, 2022
818f2a7
Make sure just_run doesn't leak file descriptors of new loops.
mariokostelac Jun 28, 2022
47283dc
Update nbclient/tests/test_util.py
mario-at-intercom Jun 29, 2022
fbf02e9
Call sleep without helper function
mariokostelac Jun 29, 2022
11fea9b
Rewrite test to avoid mypy issues
mariokostelac Jun 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion nbclient/tests/test_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
import asyncio
import unittest.mock
from unittest.mock import MagicMock

import psutil
import pytest
import tornado

from nbclient.util import run_hook, run_sync
from nbclient.util import just_run, run_hook, run_sync


@run_sync
Expand Down Expand Up @@ -75,3 +77,36 @@ async def test_run_hook_async():
hook = MagicMock(return_value=some_async_function())
await run_hook(hook)
assert hook.call_count == 1


def test_just_run_doesnt_leak_fds():
proc = psutil.Process()

async def async_sleep():
await asyncio.sleep(0.01)

# Warmup, just to make sure we're not failing on some initial fds being opened for the first time.
for _ in range(10):
just_run(async_sleep())
fds_count = proc.num_fds()

diff = []
for _ in range(10):
just_run(async_sleep())
Comment thread
mariokostelac marked this conversation as resolved.
Outdated
diff.append(proc.num_fds() - fds_count)
assert diff == [0] * 10


def test_just_run_clears_new_loop():
async def async_sleep():
await asyncio.sleep(0.1)

loop = asyncio.new_event_loop()
loop.stop = MagicMock(wraps=loop.stop)
loop.close = MagicMock(wraps=loop.close)

@davidbrochart davidbrochart Jun 29, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just ignore type checks on these.

Suggested change
loop.stop = MagicMock(wraps=loop.stop)
loop.close = MagicMock(wraps=loop.close)
loop.stop = MagicMock(wraps=loop.stop) # type: ignore
loop.close = MagicMock(wraps=loop.close) # type: ignore

@mariokostelac mariokostelac Jun 29, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I might have a better solution for these, without ignoring type checks.

I can't run tests locally, though.

(venv) dev-envs-ad\mario@a-2zurb4h4bbny ➜  nbclient git:(mario/fix_fd_leak) ls
binder  CHANGELOG.md  CONTRIBUTING.md  docs  LICENSE  MANIFEST.in  nbclient  nbclient.egg-info  pyproject.toml  pytest.ini  README.md  RELEASING.md  requirements-dev.txt  requirements.txt  setup.cfg  setup.py  venv
(venv) dev-envs-ad\mario@a-2zurb4h4bbny ➜  nbclient git:(mario/fix_fd_leak) tox -e py38
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found

@davidbrochart davidbrochart Jun 29, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're not using tox, you should just run pytest.

@mariokostelac mariokostelac Jun 29, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was running pytest, but it was missing the type checking piece that failed. I think https://github.com/jupyter/nbclient/blob/main/CONTRIBUTING.md might need an update.

Re: type checking. Would it make sense to put it as pre-commit check?

@davidbrochart davidbrochart Jun 29, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was running pytest, but it was missing the type checking piece that failed. I think https://github.com/jupyter/nbclient/blob/main/CONTRIBUTING.md might need an update.

Indeed!

Re: type checking. Would it make sense to put it as pre-commit check?

Definitely, if you want to open a PR for that, it would be great.


with unittest.mock.patch.object(asyncio, "new_event_loop", return_value=loop):
just_run(async_sleep())

loop.stop.assert_called_once
loop.close.assert_called_once
6 changes: 5 additions & 1 deletion nbclient/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,11 @@ def just_run(coro: Awaitable) -> Any:

nest_asyncio.apply()
check_patch_tornado()
return loop.run_until_complete(coro)
res = loop.run_until_complete(coro)
if not had_running_loop:
loop.stop()
loop.close()
return res


T = TypeVar("T")
Expand Down
1 change: 1 addition & 0 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ ipywidgets<8.0.0
mypy
pip>=18.1
pre-commit
psutil
pytest>=4.1
pytest-asyncio
pytest-cov>=2.6.1
Expand Down