Strange memory problem

Discussion in 'Apple' started by Oliver Jennrich, Jul 19, 2012.

  1. Hi all,

    I'm feeling rather stupid at the moment as the following problem looks
    very simple but I cannot find out what is going on.

    I have a command-line C-program that I which worked perfectly under
    10.5/6/7. Recently, we started to improve it and I'm now getting a weird
    behaviour - the allocation of seemingly small amounts of memory
    (e.g. 4MByte) fails, despite the fact that there is plenty of free
    memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the code
    that fails has nothing to do with the part that has been worked on.

    Now, as it has been a while that I compiled a new version, I cannot say
    with certainty that this behaviour started with the upgrade from Snow
    Leopard to Lion, but this is the only major change to the system.

    The strange thing is that a version compiled under SL still
    works. Compiling the old version under Lion shwos the same problem.

    Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.

    Has anybody an idea what the problem might be?
    --
    Space - The final frontier
    Oliver Jennrich, Jul 19, 2012
    #1
    1. Advertising

  2. In article <-berlin.de>,
    Oliver Jennrich <> wrote:

    > Hi all,
    >
    > I'm feeling rather stupid at the moment as the following problem looks
    > very simple but I cannot find out what is going on.
    >
    > I have a command-line C-program that I which worked perfectly under
    > 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
    > behaviour - the allocation of seemingly small amounts of memory
    > (e.g. 4MByte) fails, despite the fact that there is plenty of free
    > memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
    > code
    > that fails has nothing to do with the part that has been worked on.
    >
    > Now, as it has been a while that I compiled a new version, I cannot say
    > with certainty that this behaviour started with the upgrade from Snow
    > Leopard to Lion, but this is the only major change to the system.
    >
    > The strange thing is that a version compiled under SL still
    > works. Compiling the old version under Lion shwos the same problem.
    >
    > Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
    >
    > Has anybody an idea what the problem might be?


    99-44/100% of the time you are somewhere in your program writing past
    the bounds of the memory you allocated. I believe the most commonly used
    tool to diagnose memory corruption is valgrind, but since I've never
    personally used it myself I couldn't say for sure.
    Mark Storkamp, Jul 19, 2012
    #2
    1. Advertising

  3. In article <-berlin.de>,
    Oliver Jennrich <> wrote:

    > Hi all,
    >
    > I'm feeling rather stupid at the moment as the following problem looks
    > very simple but I cannot find out what is going on.
    >
    > I have a command-line C-program that I which worked perfectly under
    > 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
    > behaviour - the allocation of seemingly small amounts of memory
    > (e.g. 4MByte) fails, despite the fact that there is plenty of free
    > memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
    > code
    > that fails has nothing to do with the part that has been worked on.
    >
    > Now, as it has been a while that I compiled a new version, I cannot say
    > with certainty that this behaviour started with the upgrade from Snow
    > Leopard to Lion, but this is the only major change to the system.
    >
    > The strange thing is that a version compiled under SL still
    > works. Compiling the old version under Lion shwos the same problem.
    >
    > Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
    >
    > Has anybody an idea what the problem might be?


    There can be multiple memory managers lying around that attempt to
    improve performance by buffering small allocations. The compiler may
    inject one, there may be one in a standard library, a few statically
    linked in third party libraries, and then there's the unbuffered OS
    memory manager. I suspect that you have small blocks of memory
    allocated and freed in different memory managers now. It usually
    happens when one library resizes or frees memory that was allocated in
    another library.
    --
    I will not see posts from Google because I must filter them as spam
    Kevin McMurtrie, Jul 19, 2012
    #3
  4. Oliver Jennrich

    Bob Harris Guest

    In article <-berlin.de>,
    Oliver Jennrich <> wrote:

    > Hi all,
    >
    > I'm feeling rather stupid at the moment as the following problem looks
    > very simple but I cannot find out what is going on.
    >
    > I have a command-line C-program that I which worked perfectly under
    > 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
    > behaviour - the allocation of seemingly small amounts of memory
    > (e.g. 4MByte) fails, despite the fact that there is plenty of free
    > memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
    > code
    > that fails has nothing to do with the part that has been worked on.
    >
    > Now, as it has been a while that I compiled a new version, I cannot say
    > with certainty that this behaviour started with the upgrade from Snow
    > Leopard to Lion, but this is the only major change to the system.
    >
    > The strange thing is that a version compiled under SL still
    > works. Compiling the old version under Lion shwos the same problem.
    >
    > Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
    >
    > Has anybody an idea what the problem might be?


    So did you compile your code for 64-bits?

    gcc -m64 prog.c -o prog

    If you do NOT explicitly compile for 64 bits you can be limited to
    a maximum of 4GB (if that) for your program.

    If you are doing a lot of realloc() which each realloc() going
    from smaller to larger, you can fragment your heap so that you
    have lots of little holes that cannot be consolidated into a
    larger contiguous allocation.

    Bob Harris
    Bob Harris, Jul 20, 2012
    #4
  5. Oliver Jennrich

    Wes Groleau Guest

    On 07-19-2012 20:44, Bob Harris wrote:
    > If you are doing a lot of realloc() which each realloc() going
    > from smaller to larger, you can fragment your heap so that you
    > have lots of little holes that cannot be consolidated into a
    > larger contiguous allocation.


    I find this hard to believe. I worked on versions of Unix thirty years
    ago that continuously defragmented their free list.

    --
    Wes Groleau

    Nobody believes a theoretical analysis — except the guy who did it.
    Everybody believes an experimental analysis — except the guy who did it.
    — Unknown
    Wes Groleau, Jul 20, 2012
    #5
  6. Oliver Jennrich

    David Ritz Guest

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    On Thursday, 19 July 2012 13:52 +0200,
    in article <-berlin.de>,
    Oliver Jennrich <> wrote:

    > Hi all,


    > I'm feeling rather stupid at the moment as the following problem
    > looks very simple but I cannot find out what is going on.


    > I have a command-line C-program that I which worked perfectly under
    > 10.5/6/7. Recently, we started to improve it [...].


    > Now, as it has been a while that I compiled a new version, I cannot
    > say with certainty that this behaviour started with the upgrade from
    > Snow Leopard to Lion, but this is the only major change to the
    > system.


    > The strange thing is that a version compiled under SL still works.
    > Compiling the old version under Lion shwos the same problem.


    > Both gcc4.2 and cc show the problem, so I assuem it is not a
    > compiler bug.


    > Has anybody an idea what the problem might be?


    You might want to consider upgrading your Command Line Tool, under
    Xcode Version 4.3.3. After opening the Preferences in Xcode, go to
    Downloads. There you'll find a Command Line Tools package, which you
    should download and install.

    After it's installed, you'll find several modifications to /usr.

    $ ls -alF /usr/bin/*gcc*
    lrwxr-xr-x 1 root wheel 12 Jun 12 17:31 /usr/bin/gcc@ -> llvm-gcc-4.2
    lrwxr-xr-x 1 root wheel 52 Jun 12 17:32 /usr/bin/i686-apple-darwin11-llvm-gcc-4.2@ -> ../llvm-gcc-4.2/bin/i686-apple-darwin11-llvm-gcc-4.2
    lrwxr-xr-x 1 root wheel 32 Jun 12 17:32 /usr/bin/llvm-gcc@ -> ../llvm-gcc-4.2/bin/llvm-gcc-4.2
    lrwxr-xr-x 1 root wheel 32 Jun 12 17:32 /usr/bin/llvm-gcc-4.2@ -> ../llvm-gcc-4.2/bin/llvm-gcc-4.2

    $ ls -alF /usr/llvm-gcc-4.2/
    total 0
    drwxrwxr-x 7 root admin 238 Nov 3 2011 ./
    drwxr-xr-x 16 root wheel 544 May 31 10:33 ../
    drwxrwxr-x 9 root admin 306 Jun 12 17:34 bin/
    drwxrwxr-x 3 root admin 102 Nov 3 2011 include/
    drwxrwxr-x 3 root admin 102 Nov 3 2011 lib/
    drwxrwxr-x 3 root admin 102 Nov 3 2011 libexec/
    drwxrwxr-x 3 root admin 102 Nov 3 2011 share/

    $ ls -alF /usr/llvm-gcc-4.2/bin/
    total 732
    drwxrwxr-x 9 root admin 306 Jun 12 17:34 ./
    drwxrwxr-x 7 root admin 238 Nov 3 2011 ../
    -rwxr-xr-x 1 root wheel 127840 Jun 12 17:32 gcov-4.2*
    -rwxr-xr-x 1 root wheel 544448 Jun 12 17:32 i686-apple-darwin11-llvm-g++-4.2*
    -rwxr-xr-x 1 root wheel 544448 Jun 12 17:32 i686-apple-darwin11-llvm-gcc-4.2*
    -rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-c++-4.2*
    -rwxr-xr-x 1 root wheel 257600 Jun 12 17:32 llvm-cpp-4.2*
    -rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-g++-4.2*
    -rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-gcc-4.2*

    Perhaps using the compilers which are designed for use under Lion, or
    more exactly, Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT
    2012; root:xnu-1699.26.8~1/RELEASE_X86_64, will help you get things up
    and running.

    HTH.

    - --
    David Ritz <>
    Be kind to animals; kiss a shark.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.19 (Darwin)
    Comment: Public Keys: <http://dritz.home.mindspring.com/keys.txt>

    iEYEARECAAYFAlAIzsgACgkQUrwpmRoS3uvVcgCfZ8r0f20k4Mv2dy2qnjMbSs20
    uJoAoM0s1CAmWiHIuohv+Eeqk1eeBCoc
    =ioo9
    -----END PGP SIGNATURE-----
    David Ritz, Jul 20, 2012
    #6
  7. Oliver Jennrich

    Bob Harris Guest

    In article <juacij$5o2$>,
    Wes Groleau <> wrote:

    > On 07-19-2012 20:44, Bob Harris wrote:
    > > If you are doing a lot of realloc() which each realloc() going
    > > from smaller to larger, you can fragment your heap so that you
    > > have lots of little holes that cannot be consolidated into a
    > > larger contiguous allocation.

    >
    > I find this hard to believe. I worked on versions of Unix thirty years
    > ago that continuously defragmented their free list.


    for(n=0; ; n++) {
    array[n] = malloc(64);
    buf = realloc(buf, (n+1)*4096);
    }

    While this is arbitrary, I have seen code that effectively used
    realloc() with larger and larger buffers get hosed because between
    realloc() calls, there were smaller malloc() calls that "Blocked"
    the ability to consolidate the heap into larger contiguous areas.
    Eventually realloc() could no longer get a new buf, but there was
    tons of unused memory, just none of it large enough to satisfy the
    request.

    And I've been programming since the 70's, and seen code run into
    this more than once.
    Bob Harris, Jul 21, 2012
    #7
  8. Oliver Jennrich

    Wes Groleau Guest

    On 07-20-2012 19:09, Bob Harris wrote:
    > Wes Groleau <> wrote:
    >> On 07-19-2012 20:44, Bob Harris wrote:
    >>> If you are doing a lot of realloc() which each realloc() going
    >>> from smaller to larger, you can fragment your heap so that you
    >>> have lots of little holes that cannot be consolidated into a
    >>> larger contiguous allocation.

    >>
    >> I find this hard to believe. I worked on versions of Unix thirty years
    >> ago that continuously defragmented their free list.

    >
    > for(n=0; ; n++) {
    > array[n] = malloc(64);
    > buf = realloc(buf, (n+1)*4096);
    > }
    >
    > While this is arbitrary, I have seen code that effectively used
    > realloc() with larger and larger buffers get hosed because between
    > realloc() calls, there were smaller malloc() calls that "Blocked"
    > the ability to consolidate the heap into larger contiguous areas.
    > Eventually realloc() could no longer get a new buf, but there was
    > tons of unused memory, just none of it large enough to satisfy the
    > request.
    >
    > And I've been programming since the 70's, and seen code run into
    > this more than once.


    I have seen it, too. One compiler I had to use had a run-time that
    tried to manage its own heap and the algorithm it used was downright
    stupid. We eliminated constant storage errors by a linker directive
    that bypassed the run-time to use the malloc in Solaris.

    What I find hard to believe is that Mac OS, three _decades_ later still
    has such a lame heap management.

    --
    Wes Groleau

    “In the field of language teaching, Method A is the logical
    contradiction of Method B: if the assumptions from which
    A claims to be derived are correct, then B cannot work,
    and vice versa. Yet one colleague is getting excellent
    results with A and another is getting comparable results
    with B. How is this possible?â€
    — Earl W. Stevick
    Wes Groleau, Jul 21, 2012
    #8
  9. Oliver Jennrich

    Bob Harris Guest

    In article <jucpgr$n0q$>,
    Wes Groleau <> wrote:

    > On 07-20-2012 19:09, Bob Harris wrote:
    > > Wes Groleau <> wrote:
    > >> On 07-19-2012 20:44, Bob Harris wrote:
    > >>> If you are doing a lot of realloc() which each realloc() going
    > >>> from smaller to larger, you can fragment your heap so that you
    > >>> have lots of little holes that cannot be consolidated into a
    > >>> larger contiguous allocation.
    > >>
    > >> I find this hard to believe. I worked on versions of Unix thirty years
    > >> ago that continuously defragmented their free list.

    > >
    > > for(n=0; ; n++) {
    > > array[n] = malloc(64);
    > > buf = realloc(buf, (n+1)*4096);
    > > }
    > >
    > > While this is arbitrary, I have seen code that effectively used
    > > realloc() with larger and larger buffers get hosed because between
    > > realloc() calls, there were smaller malloc() calls that "Blocked"
    > > the ability to consolidate the heap into larger contiguous areas.
    > > Eventually realloc() could no longer get a new buf, but there was
    > > tons of unused memory, just none of it large enough to satisfy the
    > > request.
    > >
    > > And I've been programming since the 70's, and seen code run into
    > > this more than once.

    >
    > I have seen it, too. One compiler I had to use had a run-time that
    > tried to manage its own heap and the algorithm it used was downright
    > stupid. We eliminated constant storage errors by a linker directive
    > that bypassed the run-time to use the malloc in Solaris.
    >
    > What I find hard to believe is that Mac OS, three _decades_ later still
    > has such a lame heap management.


    If there was code that behaved as I've shown, there is nothing the
    heap management can do about it. The smaller request has given
    out an address. If that address is at the end of a previous larger
    request, then that larger request is limited in how it can be used
    again, as it cannot take back the address of the smaller request.

    I will say that today, many heap management utilities manage
    requests smaller than a page size separate than requests that are
    a page size or larger. But that does not mean the smaller request
    couldn't be in page sized units while the ever larger request
    could be in even larger page sized units.

    Again, if an address has been given out, the heap cannot take it
    back just because it happens to block consolidating 2 larger
    chunks. At best the heap can give the RAM back to the OS by
    unmapping the unused pages, that does doesn't do anything about
    the fragmented address space, and that is what the OP is running
    out of.

    Not only have I been programming since the '70, I've been working
    inside Unix OS kernels since the mid-90's and before that I wrote
    virtual memory hardware diagnostics. I have spent a lot of time
    worrying about address space, both real and virtual.

    Having said all that. We still do not know why the OP is running
    out of address space. I only suggested the realloc() as a way.
    It could be that he is using a 32-bit process but trying to use
    more than 4GB of address space.
    Bob Harris, Jul 22, 2012
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sp
    Replies:
    3
    Views:
    290
  2. ps
    Replies:
    3
    Views:
    453
    Jon Wichman
    Nov 3, 2003
  3. heynow
    Replies:
    4
    Views:
    495
    Kelly Cotter
    Jul 3, 2004
  4. Harvey Twyman
    Replies:
    78
    Views:
    1,326
    Chris Hills
    Oct 30, 2003
  5. Bernie Lehmann
    Replies:
    0
    Views:
    383
    Bernie Lehmann
    Mar 27, 2006
Loading...

Share This Page