memory use for objects in python

March 18, 2008

Playing with various items to see how they impacted yum memory use today and we came up with a few interesting items:

Here’s the command we were running with more or less the same number of pkgs installed:

yum –disablerepo=’*’ –enablerepo=’development’ update

on an i686 box the memory use during the depsolve stage was 112M

on an x86_64 box the memory use during the depsolve stage was 409M

Now the number of pkgs available on the x86_64 box was more b/c x86_64 repos have about 25% more packages. But 25% does not account for a spike in memory use of 4x.

After a fair bit of messing around and eliminating variables I found this:

“Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t). Before it was 16 on all boxes, reflecting that Python couldn’t make use of allocations >= 2**32 bytes even on 64-bit boxes before 2.5.”

From here: http://svn.python.org/projects/peps/trunk/pep-0353.txt

From what I can tell, we end up with a 2x to 3x bump on x86_64.

This makes me sad.

Anyone know of a way to beat/mitigate this other than ‘use fewer python objects’?

3 Responses to “memory use for objects in python”

  1. Malcolm Parsons Says:

    “Anyone know of a way to beat/mitigate this other than ‘use fewer python objects’?”

    Don’t use python.

  2. loupgaroublond Says:

    112MB for i686? This makes yum practically unusable on a box that has anywhere less than 512MB of ram. It’s nice that you can confirm I’m not crazy when the hard drive would spin like mad on my sister’s laptop whenever I had firefox and yum running.

  3. Andy Wingo Says:

    You should specify what you mean by “memory use”. Resident, dirty, VMSIZE… it’s not clear what numbers you are talking about.

    I would think that a x3 increase would be expected on x86-64: x2 because each object takes up twice as much memory, on average, and some change because of having more packages.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: