Home » Python » Total memory used by Python process?

Total memory used by Python process?

Posted by: admin November 1, 2017 Leave a comment

Questions:

Is there a way for a Python program to determine how much memory it’s currently using? I’ve seen discussions about memory usage for a single object, but what I need is total memory usage for the process, so that I can determine when it’s necessary to start discarding cached data.

Answers:

On Windows, you can use WMI (home page, cheeseshop):


def memory():
    import os
    from wmi import WMI
    w = WMI('.')
    result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())
    return int(result[0].WorkingSet)

On Linux (from python cookbook http://code.activestate.com/recipes/286222/:

import os
_proc_status = '/proc/%d/status' % os.getpid()

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
          'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
    '''Private.
    '''
    global _proc_status, _scale
     # get pseudo file  /proc/<pid>/status
    try:
        t = open(_proc_status)
        v = t.read()
        t.close()
    except:
        return 0.0  # non-Linux?
     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
    i = v.index(VmKey)
    v = v[i:].split(None, 3)  # whitespace
    if len(v) < 3:
        return 0.0  # invalid format?
     # convert Vm value to bytes
    return float(v[1]) * _scale[v[2]]


def memory(since=0.0):
    '''Return memory usage in bytes.
    '''
    return _VmB('VmSize:') - since


def resident(since=0.0):
    '''Return resident memory usage in bytes.
    '''
    return _VmB('VmRSS:') - since


def stacksize(since=0.0):
    '''Return stack size in bytes.
    '''
    return _VmB('VmStk:') - since

Questions:
Answers:

For Unixes (Linux, Mac OS X, Solaris) you could also use the getrusage() function from the standard library module resource. The resulting object has the attribute ru_maxrss, which gives peak memory usage for the calling process:

>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
2656 # peak memory usage (bytes on OS X, kilobytes on Linux)

The Python docs aren’t clear on what the units are exactly, but the Mac OS X man page for getrusage(2) describes the units as bytes. The Linux man page isn’t clear, but it seems to be equivalent to the information from /proc/self/status, which is in kilobytes.

The getrusage() function can also be given resource.RUSAGE_CHILDREN to get the usage for child processes, and (on some systems) resource.RUSAGE_BOTH for total (self and child) process usage.

resource is a standard library module.

If you only care about Linux, you can just check the /proc/self/status file as described in a similar question.

Questions:
Answers:

Here is a useful solution that works for various operating systems, including Windows 7 x64:

import os
import psutil
process = psutil.Process(os.getpid())
print(process.memory_info().rss)

Questions:
Answers:

On unix, you can use the ps tool to monitor it:

$ ps u -p 1347 | awk '{sum=sum+$6}; END {print sum/1024}'

where 1347 is some process id. Also, the result is in MB.

Questions:
Answers:

Heapy (and friends) may be what you’re looking for.

Also, caches typically have a fixed upper limit on their size to solve the sort of problem you’re talking about. For instance, check out this LRU cache decorator.

Questions:
Answers:

Using sh and os to get into python bayer’s answer.

float(sh.awk(sh.ps('u','-p',os.getpid()),'{sum=sum+$6}; END {print sum/1024}'))

Answer is in megabytes.

Questions:
Answers:

I like it, thank you for @bayer. I get a specific process count tool, now.

# Megabyte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'
87.9492 MB

# Byte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum " KB"}'
90064 KB

Attach my process list.

$ ps aux  | grep python
root       943  0.0  0.1  53252  9524 ?        Ss   Aug19  52:01 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root       950  0.6  0.4 299680 34220 ?        Sl   Aug19 568:52 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root      3803  0.2  0.4 315692 36576 ?        S    12:43   0:54 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
jonny    23325  0.0  0.1  47460  9076 pts/0    S+   17:40   0:00 python
jonny    24651  0.0  0.0  13076   924 pts/4    S+   18:06   0:00 grep python

Reference

Questions:
Answers:
han = win32api.OpenProcess(win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ, 0, os.getpid())
process_memory = int(win32process.GetProcessMemoryInfo(han)['WorkingSetSize'])