Motherboard Forums


Reply
Thread Tools Display Modes

Strange memory problem

 
 
Oliver Jennrich
Guest
Posts: n/a
 
      07-19-2012, 11:52 AM
Hi all,

I'm feeling rather stupid at the moment as the following problem looks
very simple but I cannot find out what is going on.

I have a command-line C-program that I which worked perfectly under
10.5/6/7. Recently, we started to improve it and I'm now getting a weird
behaviour - the allocation of seemingly small amounts of memory
(e.g. 4MByte) fails, despite the fact that there is plenty of free
memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the code
that fails has nothing to do with the part that has been worked on.

Now, as it has been a while that I compiled a new version, I cannot say
with certainty that this behaviour started with the upgrade from Snow
Leopard to Lion, but this is the only major change to the system.

The strange thing is that a version compiled under SL still
works. Compiling the old version under Lion shwos the same problem.

Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.

Has anybody an idea what the problem might be?
--
Space - The final frontier
 
Reply With Quote
 
 
 
 
Mark Storkamp
Guest
Posts: n/a
 
      07-19-2012, 01:42 PM
In article <(E-Mail Removed)-berlin.de>,
Oliver Jennrich <(E-Mail Removed)> wrote:

> Hi all,
>
> I'm feeling rather stupid at the moment as the following problem looks
> very simple but I cannot find out what is going on.
>
> I have a command-line C-program that I which worked perfectly under
> 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
> behaviour - the allocation of seemingly small amounts of memory
> (e.g. 4MByte) fails, despite the fact that there is plenty of free
> memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
> code
> that fails has nothing to do with the part that has been worked on.
>
> Now, as it has been a while that I compiled a new version, I cannot say
> with certainty that this behaviour started with the upgrade from Snow
> Leopard to Lion, but this is the only major change to the system.
>
> The strange thing is that a version compiled under SL still
> works. Compiling the old version under Lion shwos the same problem.
>
> Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
>
> Has anybody an idea what the problem might be?


99-44/100% of the time you are somewhere in your program writing past
the bounds of the memory you allocated. I believe the most commonly used
tool to diagnose memory corruption is valgrind, but since I've never
personally used it myself I couldn't say for sure.
 
Reply With Quote
 
 
 
 
Kevin McMurtrie
Guest
Posts: n/a
 
      07-19-2012, 05:41 PM
In article <(E-Mail Removed)-berlin.de>,
Oliver Jennrich <(E-Mail Removed)> wrote:

> Hi all,
>
> I'm feeling rather stupid at the moment as the following problem looks
> very simple but I cannot find out what is going on.
>
> I have a command-line C-program that I which worked perfectly under
> 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
> behaviour - the allocation of seemingly small amounts of memory
> (e.g. 4MByte) fails, despite the fact that there is plenty of free
> memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
> code
> that fails has nothing to do with the part that has been worked on.
>
> Now, as it has been a while that I compiled a new version, I cannot say
> with certainty that this behaviour started with the upgrade from Snow
> Leopard to Lion, but this is the only major change to the system.
>
> The strange thing is that a version compiled under SL still
> works. Compiling the old version under Lion shwos the same problem.
>
> Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
>
> Has anybody an idea what the problem might be?


There can be multiple memory managers lying around that attempt to
improve performance by buffering small allocations. The compiler may
inject one, there may be one in a standard library, a few statically
linked in third party libraries, and then there's the unbuffered OS
memory manager. I suspect that you have small blocks of memory
allocated and freed in different memory managers now. It usually
happens when one library resizes or frees memory that was allocated in
another library.
--
I will not see posts from Google because I must filter them as spam
 
Reply With Quote
 
Bob Harris
Guest
Posts: n/a
 
      07-20-2012, 12:44 AM
In article <(E-Mail Removed)-berlin.de>,
Oliver Jennrich <(E-Mail Removed)> wrote:

> Hi all,
>
> I'm feeling rather stupid at the moment as the following problem looks
> very simple but I cannot find out what is going on.
>
> I have a command-line C-program that I which worked perfectly under
> 10.5/6/7. Recently, we started to improve it and I'm now getting a weird
> behaviour - the allocation of seemingly small amounts of memory
> (e.g. 4MByte) fails, despite the fact that there is plenty of free
> memory shown by 'activity monitor' (8GB MBP on 10.7.4, ~3 GB free). NB - the
> code
> that fails has nothing to do with the part that has been worked on.
>
> Now, as it has been a while that I compiled a new version, I cannot say
> with certainty that this behaviour started with the upgrade from Snow
> Leopard to Lion, but this is the only major change to the system.
>
> The strange thing is that a version compiled under SL still
> works. Compiling the old version under Lion shwos the same problem.
>
> Both gcc4.2 and cc show the problem, so I assuem it is not a compiler bug.
>
> Has anybody an idea what the problem might be?


So did you compile your code for 64-bits?

gcc -m64 prog.c -o prog

If you do NOT explicitly compile for 64 bits you can be limited to
a maximum of 4GB (if that) for your program.

If you are doing a lot of realloc() which each realloc() going
from smaller to larger, you can fragment your heap so that you
have lots of little holes that cannot be consolidated into a
larger contiguous allocation.

Bob Harris
 
Reply With Quote
 
Wes Groleau
Guest
Posts: n/a
 
      07-20-2012, 01:34 AM
On 07-19-2012 20:44, Bob Harris wrote:
> If you are doing a lot of realloc() which each realloc() going
> from smaller to larger, you can fragment your heap so that you
> have lots of little holes that cannot be consolidated into a
> larger contiguous allocation.


I find this hard to believe. I worked on versions of Unix thirty years
ago that continuously defragmented their free list.

--
Wes Groleau

Nobody believes a theoretical analysis — except the guy who did it.
Everybody believes an experimental analysis — except the guy who did it.
— Unknown

 
Reply With Quote
 
David Ritz
Guest
Posts: n/a
 
      07-20-2012, 03:21 AM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday, 19 July 2012 13:52 +0200,
in article <(E-Mail Removed)-berlin.de>,
Oliver Jennrich <(E-Mail Removed)> wrote:

> Hi all,


> I'm feeling rather stupid at the moment as the following problem
> looks very simple but I cannot find out what is going on.


> I have a command-line C-program that I which worked perfectly under
> 10.5/6/7. Recently, we started to improve it [...].


> Now, as it has been a while that I compiled a new version, I cannot
> say with certainty that this behaviour started with the upgrade from
> Snow Leopard to Lion, but this is the only major change to the
> system.


> The strange thing is that a version compiled under SL still works.
> Compiling the old version under Lion shwos the same problem.


> Both gcc4.2 and cc show the problem, so I assuem it is not a
> compiler bug.


> Has anybody an idea what the problem might be?


You might want to consider upgrading your Command Line Tool, under
Xcode Version 4.3.3. After opening the Preferences in Xcode, go to
Downloads. There you'll find a Command Line Tools package, which you
should download and install.

After it's installed, you'll find several modifications to /usr.

$ ls -alF /usr/bin/*gcc*
lrwxr-xr-x 1 root wheel 12 Jun 12 17:31 /usr/bin/gcc@ -> llvm-gcc-4.2
lrwxr-xr-x 1 root wheel 52 Jun 12 17:32 /usr/bin/i686-apple-darwin11-llvm-gcc-4.2@ -> ../llvm-gcc-4.2/bin/i686-apple-darwin11-llvm-gcc-4.2
lrwxr-xr-x 1 root wheel 32 Jun 12 17:32 /usr/bin/llvm-gcc@ -> ../llvm-gcc-4.2/bin/llvm-gcc-4.2
lrwxr-xr-x 1 root wheel 32 Jun 12 17:32 /usr/bin/llvm-gcc-4.2@ -> ../llvm-gcc-4.2/bin/llvm-gcc-4.2

$ ls -alF /usr/llvm-gcc-4.2/
total 0
drwxrwxr-x 7 root admin 238 Nov 3 2011 ./
drwxr-xr-x 16 root wheel 544 May 31 10:33 ../
drwxrwxr-x 9 root admin 306 Jun 12 17:34 bin/
drwxrwxr-x 3 root admin 102 Nov 3 2011 include/
drwxrwxr-x 3 root admin 102 Nov 3 2011 lib/
drwxrwxr-x 3 root admin 102 Nov 3 2011 libexec/
drwxrwxr-x 3 root admin 102 Nov 3 2011 share/

$ ls -alF /usr/llvm-gcc-4.2/bin/
total 732
drwxrwxr-x 9 root admin 306 Jun 12 17:34 ./
drwxrwxr-x 7 root admin 238 Nov 3 2011 ../
-rwxr-xr-x 1 root wheel 127840 Jun 12 17:32 gcov-4.2*
-rwxr-xr-x 1 root wheel 544448 Jun 12 17:32 i686-apple-darwin11-llvm-g++-4.2*
-rwxr-xr-x 1 root wheel 544448 Jun 12 17:32 i686-apple-darwin11-llvm-gcc-4.2*
-rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-c++-4.2*
-rwxr-xr-x 1 root wheel 257600 Jun 12 17:32 llvm-cpp-4.2*
-rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-g++-4.2*
-rwxr-xr-x 1 root wheel 117152 Jun 12 17:32 llvm-gcc-4.2*

Perhaps using the compilers which are designed for use under Lion, or
more exactly, Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT
2012; root:xnu-1699.26.8~1/RELEASE_X86_64, will help you get things up
and running.

HTH.

- --
David Ritz <(E-Mail Removed)>
Be kind to animals; kiss a shark.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (Darwin)
Comment: Public Keys: <http://dritz.home.mindspring.com/keys.txt>

iEYEARECAAYFAlAIzsgACgkQUrwpmRoS3uvVcgCfZ8r0f20k4M v2dy2qnjMbSs20
uJoAoM0s1CAmWiHIuohv+Eeqk1eeBCoc
=ioo9
-----END PGP SIGNATURE-----
 
Reply With Quote
 
Bob Harris
Guest
Posts: n/a
 
      07-20-2012, 11:09 PM
In article <juacij$5o2$(E-Mail Removed)>,
Wes Groleau <(E-Mail Removed)> wrote:

> On 07-19-2012 20:44, Bob Harris wrote:
> > If you are doing a lot of realloc() which each realloc() going
> > from smaller to larger, you can fragment your heap so that you
> > have lots of little holes that cannot be consolidated into a
> > larger contiguous allocation.

>
> I find this hard to believe. I worked on versions of Unix thirty years
> ago that continuously defragmented their free list.


for(n=0; ; n++) {
array[n] = malloc(64);
buf = realloc(buf, (n+1)*4096);
}

While this is arbitrary, I have seen code that effectively used
realloc() with larger and larger buffers get hosed because between
realloc() calls, there were smaller malloc() calls that "Blocked"
the ability to consolidate the heap into larger contiguous areas.
Eventually realloc() could no longer get a new buf, but there was
tons of unused memory, just none of it large enough to satisfy the
request.

And I've been programming since the 70's, and seen code run into
this more than once.
 
Reply With Quote
 
Wes Groleau
Guest
Posts: n/a
 
      07-20-2012, 11:27 PM
On 07-20-2012 19:09, Bob Harris wrote:
> Wes Groleau <(E-Mail Removed)> wrote:
>> On 07-19-2012 20:44, Bob Harris wrote:
>>> If you are doing a lot of realloc() which each realloc() going
>>> from smaller to larger, you can fragment your heap so that you
>>> have lots of little holes that cannot be consolidated into a
>>> larger contiguous allocation.

>>
>> I find this hard to believe. I worked on versions of Unix thirty years
>> ago that continuously defragmented their free list.

>
> for(n=0; ; n++) {
> array[n] = malloc(64);
> buf = realloc(buf, (n+1)*4096);
> }
>
> While this is arbitrary, I have seen code that effectively used
> realloc() with larger and larger buffers get hosed because between
> realloc() calls, there were smaller malloc() calls that "Blocked"
> the ability to consolidate the heap into larger contiguous areas.
> Eventually realloc() could no longer get a new buf, but there was
> tons of unused memory, just none of it large enough to satisfy the
> request.
>
> And I've been programming since the 70's, and seen code run into
> this more than once.


I have seen it, too. One compiler I had to use had a run-time that
tried to manage its own heap and the algorithm it used was downright
stupid. We eliminated constant storage errors by a linker directive
that bypassed the run-time to use the malloc in Solaris.

What I find hard to believe is that Mac OS, three _decades_ later still
has such a lame heap management.

--
Wes Groleau

“In the field of language teaching, Method A is the logical
contradiction of Method B: if the assumptions from which
A claims to be derived are correct, then B cannot work,
and vice versa. Yet one colleague is getting excellent
results with A and another is getting comparable results
with B. How is this possible?”
— Earl W. Stevick

 
Reply With Quote
 
Bob Harris
Guest
Posts: n/a
 
      07-22-2012, 12:07 AM
In article <jucpgr$n0q$(E-Mail Removed)>,
Wes Groleau <(E-Mail Removed)> wrote:

> On 07-20-2012 19:09, Bob Harris wrote:
> > Wes Groleau <(E-Mail Removed)> wrote:
> >> On 07-19-2012 20:44, Bob Harris wrote:
> >>> If you are doing a lot of realloc() which each realloc() going
> >>> from smaller to larger, you can fragment your heap so that you
> >>> have lots of little holes that cannot be consolidated into a
> >>> larger contiguous allocation.
> >>
> >> I find this hard to believe. I worked on versions of Unix thirty years
> >> ago that continuously defragmented their free list.

> >
> > for(n=0; ; n++) {
> > array[n] = malloc(64);
> > buf = realloc(buf, (n+1)*4096);
> > }
> >
> > While this is arbitrary, I have seen code that effectively used
> > realloc() with larger and larger buffers get hosed because between
> > realloc() calls, there were smaller malloc() calls that "Blocked"
> > the ability to consolidate the heap into larger contiguous areas.
> > Eventually realloc() could no longer get a new buf, but there was
> > tons of unused memory, just none of it large enough to satisfy the
> > request.
> >
> > And I've been programming since the 70's, and seen code run into
> > this more than once.

>
> I have seen it, too. One compiler I had to use had a run-time that
> tried to manage its own heap and the algorithm it used was downright
> stupid. We eliminated constant storage errors by a linker directive
> that bypassed the run-time to use the malloc in Solaris.
>
> What I find hard to believe is that Mac OS, three _decades_ later still
> has such a lame heap management.


If there was code that behaved as I've shown, there is nothing the
heap management can do about it. The smaller request has given
out an address. If that address is at the end of a previous larger
request, then that larger request is limited in how it can be used
again, as it cannot take back the address of the smaller request.

I will say that today, many heap management utilities manage
requests smaller than a page size separate than requests that are
a page size or larger. But that does not mean the smaller request
couldn't be in page sized units while the ever larger request
could be in even larger page sized units.

Again, if an address has been given out, the heap cannot take it
back just because it happens to block consolidating 2 larger
chunks. At best the heap can give the RAM back to the OS by
unmapping the unused pages, that does doesn't do anything about
the fragmented address space, and that is what the OP is running
out of.

Not only have I been programming since the '70, I've been working
inside Unix OS kernels since the mid-90's and before that I wrote
virtual memory hardware diagnostics. I have spent a lot of time
worrying about address space, both real and virtual.

Having said all that. We still do not know why the OP is running
out of address space. I only suggested the realloc() as a way.
It could be that he is using a 32-bit process but trying to use
more than 4GB of address space.
 
Reply With Quote
 
 
 
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Strange memory problem with satellite pro 4600 Bernie Lehmann Laptops 0 03-27-2006 10:34 AM
strange memory problem on IS7-E sp Abit 3 10-16-2004 07:10 PM
G4 4400 - strange vid problem, strange error heynow Nvidia 4 07-03-2004 01:28 AM
Strange Memory Problem ps ECS 3 11-03-2003 01:31 AM
Question About Strange 'C' Code Syntax ( Well strange to me anyway ) Harvey Twyman Embedded 78 10-30-2003 08:40 PM


All times are GMT. The time now is 01:03 PM.


Welcome!
Welcome to Motherboard Point
 

Advertisment