Handle `ZombieProcess` and `NoSuchProcess` from `psutil` for OSX systems by flipdazed · Pull Request #1599 · eth-brownie/brownie

flipdazed · 2022-08-21T21:08:55Z

…://github.com//issues/1588

What I did

I added a check to ensure that a process in is actually an active process. There are many circumstances in OSX where one can have hanging/zombie processes. In this instance, the process name does not exist as the process has been terminated. This causes a really confusing psutil error for the user

  ...

  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/brownie/network/main.py", line 48, in connect
    rpc.attach(host)
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/brownie/network/rpc/__init__.py", line 114, in attach
    pid = self._find_rpc_process_pid(resolved_addr)
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/brownie/network/rpc/__init__.py", line 193, in _find_rpc_process_pid
    return self._get_pid_from_docker_backend()
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/brownie/network/rpc/__init__.py", line 241, in _get_pid_from_docker_backend
    proc = self._find_proc_by_name("com.docker.backend")
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/brownie/network/rpc/__init__.py", line 255, in _find_proc_by_name
    if process_name.lower() in proc.name().lower():
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py", line 628, in name
    cmdline = self.cmdline()
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py", line 681, in cmdline
    return self._proc.cmdline()
  File "/opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py", line 348, in wrapper
    raise NoSuchProcess(self.pid, self._name)
psutil.NoSuchProcess: process no longer exists (pid=696)

Related issue: #1588

How I did it

As with many bugs, this fix is quite simple when the origin of the bug has been pinned down. The following code

brownie/brownie/network/rpc/__init__.py

Lines 253 to 256 in 4ae5f52

    
           def _find_proc_by_name(self, process_name: str) -> psutil.Process: 
        
               for proc in psutil.process_iter(): 
        
                   if process_name.lower() in proc.name().lower(): 
        
                       return proc

will fail for certain psutil cases (yielding ZombieProcess and NoSuchProcess shown below for ease of viewing and linked)

        except ProcessLookupError:
            if is_zombie(self.pid):
                raise ZombieProcess(self.pid, self._name, self._ppid)
            else:
                raise NoSuchProcess(self.pid, self._name)
        except PermissionError:
            raise AccessDenied(self.pid, self._name)
        except cext.ZombieProcessError:
            raise ZombieProcess(self.pid, self._name, self._ppid)

Added a try/except loop to catch the specific error and skip it.

    def _find_proc_by_name(self, process_name: str) -> psutil.Process:
        for proc in psutil.process_iter():
            try:
                proc_name = proc.name().lower()
            except (psutil.NoSuchProcess, psutil.ZombieProcess):
                continue

            if process_name.lower() in proc_name:
                return proc

This is the most intuitive way of catching this error as there is no way to filter the list of iterated processes to improve efficiency.

How to verify it

This is really tough - you need to somehow create a hanging process. It should be obvious from the code within psutil

https://github.com/giampaolo/psutil/blob/4446e3bd5fd29e0598f2ca6672d5ffe8ca40d442/psutil/_psosx.py#L336-L353

It will be difficult to create zombie processes or hanging processes for a full regression test. I don't think there should be test cases for such an obscure issue as a result. Satisfying the existing tests should be sufficient.

simple verification using existing bad system setup

as shown by ..

In [19]: import psutil
In [22]: [proc for proc in psutil.process_iter() if proc.name()]
---------------------------------------------------------------------------
ProcessLookupError                        Traceback (most recent call last)
File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:343, in wrap_exceptions.<locals>.wrapper(self, *args, **kwargs)
    342 try:
--> 343     return fun(self, *args, **kwargs)
    344 except ProcessLookupError:

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:401, in Process.cmdline(self)
    399 @wrap_exceptions
    400 def cmdline(self):
--> 401     return cext.proc_cmdline(self.pid)

ProcessLookupError: [Errno 3] assume no such process (originated from sysctl(KERN_PROCARGS2) -> EINVAL)

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
Input In [22], in <cell line: 1>()
----> 1 [proc for proc in psutil.process_iter() if proc.name()]

Input In [22], in <listcomp>(.0)
----> 1 [proc for proc in psutil.process_iter() if proc.name()]

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py:628, in Process.name(self)
    622 if POSIX and len(name) >= 15:
    623     # On UNIX the name gets truncated to the first 15 characters.
    624     # If it matches the first part of the cmdline we return that
    625     # one instead because it's usually more explicative.
    626     # Examples are "gnome-keyring-d" vs. "gnome-keyring-daemon".
    627     try:
--> 628         cmdline = self.cmdline()
    629     except AccessDenied:
    630         pass

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py:681, in Process.cmdline(self)
    679 def cmdline(self):
    680     """The command line this process has been called with."""
--> 681     return self._proc.cmdline()

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:348, in wrap_exceptions.<locals>.wrapper(self, *args, **kwargs)
    346         raise ZombieProcess(self.pid, self._name, self._ppid)
    347     else:
--> 348         raise NoSuchProcess(self.pid, self._name)
    349 except PermissionError:
    350     raise AccessDenied(self.pid, self._name)

NoSuchProcess: process no longer exists (pid=696)

The fix avoids this problem ...

In [15]:         for proc in psutil.process_iter():
    ...:             try:
    ...:                 proc_name = proc.name().lower()
    ...:             except (psutil.NoSuchProcess, psutil.ZombieProcess):
    ...:                 continue
    ...: 
    ...:             if process_name.lower() in proc_name:
    ...:                 print(proc)
    ...: 
psutil.Process(pid=92742, name='com.docker.backend', status='running', started='21:38:50')
psutil.Process(pid=92745, name='com.docker.backend', status='running', started='21:38:50')

For now it can be manually corrected by removing the bad processes like

In [24]: !kill 696

In [25]: In [19]: import psutil
    ...: In [22]: [proc for proc in psutil.process_iter() if proc.name()]
---------------------------------------------------------------------------
ProcessLookupError                        Traceback (most recent call last)
File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:343, in wrap_exceptions.<locals>.wrapper(self, *args, **kwargs)
    342 try:
--> 343     return fun(self, *args, **kwargs)
    344 except ProcessLookupError:

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:401, in Process.cmdline(self)
    399 @wrap_exceptions
    400 def cmdline(self):
--> 401     return cext.proc_cmdline(self.pid)

ProcessLookupError: [Errno 3] assume no such process (originated from sysctl(KERN_PROCARGS2) -> EINVAL)

During handling of the above exception, another exception occurred:

NoSuchProcess                             Traceback (most recent call last)
Input In [25], in <cell line: 2>()
      1 import psutil
----> 2 [proc for proc in psutil.process_iter() if proc.name()]

Input In [25], in <listcomp>(.0)
      1 import psutil
----> 2 [proc for proc in psutil.process_iter() if proc.name()]

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py:628, in Process.name(self)
    622 if POSIX and len(name) >= 15:
    623     # On UNIX the name gets truncated to the first 15 characters.
    624     # If it matches the first part of the cmdline we return that
    625     # one instead because it's usually more explicative.
    626     # Examples are "gnome-keyring-d" vs. "gnome-keyring-daemon".
    627     try:
--> 628         cmdline = self.cmdline()
    629     except AccessDenied:
    630         pass

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/__init__.py:681, in Process.cmdline(self)
    679 def cmdline(self):
    680     """The command line this process has been called with."""
--> 681     return self._proc.cmdline()

File /opt/homebrew/anaconda3/envs/keyring-verifier-api/lib/python3.8/site-packages/psutil/_psosx.py:348, in wrap_exceptions.<locals>.wrapper(self, *args, **kwargs)
    346         raise ZombieProcess(self.pid, self._name, self._ppid)
    347     else:
--> 348         raise NoSuchProcess(self.pid, self._name)
    349 except PermissionError:
    350     raise AccessDenied(self.pid, self._name)

NoSuchProcess: process no longer exists (pid=20033)

In [26]: !kill 20033

In [27]: In [19]: import psutil
    ...: In [22]: [proc for proc in psutil.process_iter() if proc.name()]
Out[27]: 
[psutil.Process(pid=0, name='kernel_task', status='running', started='2022-08-18 21:48:21'),
 psutil.Process(pid=1, name='launchd', status='running', started='2022-08-18 21:48:21'),
 psutil.Process(pid=303, name='logd', status='running', started='2022-08-18 21:49:03'),
 psutil.Process(pid=304, name='UserEventAgent', status='running', started='2022-08-18 21:49:03'),
 psutil.Process(pid=306, name='uninstalld', status='running', started='2022-08-18 21:49:03'),
 psutil.Process(pid=307, name='fseventsd', status='running', started='2022-08-18 21:49:03'),
 psutil.Process(pid=308, name='mediaremoted', status='running', started='2022-08-18 21:49:03'),
 psutil.Process(pid=310, name='Karabiner-Driver', status='running', started='2022-08-18 21:49:03'),

...

Checklist

I have confirmed that my PR passes all linting checks
I have included test cases
I have updated the documentation
I have added an entry to the changelog

…rownie#1588

flipdazed · 2022-08-21T21:21:38Z

This function shouldn't have test cases or documentation as it's so niche it would add code bloat

fix for Darwin systems that cont find a specific process as per eth-b…

3c00cb8

…rownie#1588

flipdazed and others added 3 commits August 21, 2022 22:22

added ZombieProcess as well

b93be5f

changelog

a1a2553

Merge branch 'master' into feature/NoSuchProcess

ff9b1d0

flipdazed changed the title ~~fix for Darwin systems that cont find a specific process as per https…~~ Handle ZombieProcess and NoSuchProcess from psutil Aug 21, 2022

flipdazed changed the title ~~Handle ZombieProcess and NoSuchProcess from psutil~~ Handle ZombieProcess and NoSuchProcess from psutil when connecting rpc Aug 21, 2022

flipdazed changed the title ~~Handle ZombieProcess and NoSuchProcess from psutil when connecting rpc~~ Handle ZombieProcess and NoSuchProcess from psutil for OSX systems Aug 21, 2022

better log title

11a4a05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle `ZombieProcess` and `NoSuchProcess` from `psutil` for OSX systems#1599

Handle `ZombieProcess` and `NoSuchProcess` from `psutil` for OSX systems#1599
flipdazed wants to merge 5 commits intoeth-brownie:masterfrom
flipdazed:feature/NoSuchProcess

flipdazed commented Aug 21, 2022 •

edited

Loading

Uh oh!

flipdazed commented Aug 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	def _find_proc_by_name(self, process_name: str) -> psutil.Process:
	for proc in psutil.process_iter():
	if process_name.lower() in proc.name().lower():
	return proc

Conversation

flipdazed commented Aug 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

How I did it

How to verify it

simple verification using existing bad system setup

Checklist

Uh oh!

flipdazed commented Aug 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

flipdazed commented Aug 21, 2022 •

edited

Loading