We recently had a memory leak on one of our servers and it stopped as soon as we commented out the line of code assigning the Diagnostics attribute of an IntegrityError to a local variable within the exception handler.
The code below replicates the issue for Python 3.5.2, Psycopg 2.6.1. When I run this the amount of resident memory used by the process steadily increases, although not every time func() is called.
#!/usr/bin/env python3
import psycopg2
import itertools
import gc
import os
conn = psycopg2.connect(os.getenv('DSN'))
def main():
with conn:
with conn.cursor() as cur:
cur.execute('CREATE TABLE IF NOT EXISTS leaky (id integer UNIQUE)')
for n in itertools.count(1):
print(n, resident_memory_usage())
func()
def func():
with conn:
with conn.cursor() as cur:
try:
cur.execute("INSERT INTO leaky VALUES (1)")
except psycopg2.IntegrityError as e:
diag = e.diag # this assignment results in a memory leak unless it's explicitly deleted
#e.diag # just accessing the attribute does not result in a memory leak
#del diag # explicitly deleting stops the leak
#gc.collect() # <-- garbage collecting doesn't help even though the only reference should be circular
def resident_memory_usage():
with open('/proc/self/status') as status:
for line in status:
parts = line.split()
if parts[0][2:-1].lower() == 'rss':
return int(parts[1])
if __name__ == '__main__':
main()
We recently had a memory leak on one of our servers and it stopped as soon as we commented out the line of code assigning the Diagnostics attribute of an IntegrityError to a local variable within the exception handler.
The code below replicates the issue for Python 3.5.2, Psycopg 2.6.1. When I run this the amount of resident memory used by the process steadily increases, although not every time func() is called.