Motherboard Forums


Reply
Thread Tools Display Modes

Allocating more than 2 Go of memory

 
 
Mats Weber
Guest
Posts: n/a
 
      09-07-2004, 09:37 AM
I need to allocate more than 2 Go of memory in a C++ program with no
user interface (no Carbon or Cocoa).

When I try to cross the 2 Go barrier, I get the following messages:

*** malloc: vm_allocate(size=1069056) failed (error code=3)
*** malloc[15141]: error: Can't allocate region
*** malloc: vm_allocate(size=1069056) failed (error code=3)
*** malloc[15141]: error: Can't allocate region

and the program stops.

This is on a dual G5 with 4 Go of RAM and enough hard disk space,
running Mac OS X 10.3.5. I checked ulimit, there is no limit.
 
Reply With Quote
 
 
 
 
Gregory Weston
Guest
Posts: n/a
 
      09-07-2004, 11:04 AM
In article <(E-Mail Removed)>,
Mats Weber <(E-Mail Removed)> wrote:

> I need to allocate more than 2 Go of memory in a C++ program with no
> user interface (no Carbon or Cocoa).
>
> When I try to cross the 2 Go barrier, I get the following messages:
>
> *** malloc: vm_allocate(size=1069056) failed (error code=3)
> *** malloc[15141]: error: Can't allocate region
> *** malloc: vm_allocate(size=1069056) failed (error code=3)
> *** malloc[15141]: error: Can't allocate region
>
> and the program stops.
>
> This is on a dual G5 with 4 Go of RAM and enough hard disk space,
> running Mac OS X 10.3.5. I checked ulimit, there is no limit.


That's a documented limitation in OS X. I can't remember if it's known
that this will change in 10.4.

--
Standard output is like your butt. Everyone has one. When using a bathroom,
they all default to going into a toilet. However, a person can redirect his
"standard output" to somewhere else, if he so chooses. - Jeremy Nixon
 
Reply With Quote
 
 
 
 
Eric Albert
Guest
Posts: n/a
 
      09-07-2004, 05:27 PM
In article <(E-Mail Removed)>,
Gregory Weston <(E-Mail Removed)> wrote:

> In article <(E-Mail Removed)>,
> Mats Weber <(E-Mail Removed)> wrote:
>
> > I need to allocate more than 2 Go of memory in a C++ program with no
> > user interface (no Carbon or Cocoa).
> >
> > When I try to cross the 2 Go barrier, I get the following messages:
> >
> > *** malloc: vm_allocate(size=1069056) failed (error code=3)
> > *** malloc[15141]: error: Can't allocate region
> > *** malloc: vm_allocate(size=1069056) failed (error code=3)
> > *** malloc[15141]: error: Can't allocate region
> >
> > and the program stops.
> >
> > This is on a dual G5 with 4 Go of RAM and enough hard disk space,
> > running Mac OS X 10.3.5. I checked ulimit, there is no limit.

>
> That's a documented limitation in OS X. I can't remember if it's known
> that this will change in 10.4.


I don't think it's strictly a 2 GB limit -- rather, it has to do with
how things are laid out in memory and a very carefully designed program
may be able to allocate a little bit more.

That said, you can access more memory on a G5 today with certain messy
techniques involving something like mapping and unmapping large blocks
of memory. Check the archives of Apple's darwin-development mailing
list or ask DTS for details.

-Eric

--
Eric Albert (E-Mail Removed)
http://rescomp.stanford.edu/~ejalbert/
 
Reply With Quote
 
Gregory Weston
Guest
Posts: n/a
 
      09-07-2004, 09:43 PM
In article <ejalbert-866509.10272407092004@localhost>,
Eric Albert <(E-Mail Removed)> wrote:

> In article <(E-Mail Removed)>,
> Gregory Weston <(E-Mail Removed)> wrote:
>
> > In article <(E-Mail Removed)>,
> > Mats Weber <(E-Mail Removed)> wrote:
> >
> > > I need to allocate more than 2 Go of memory in a C++ program with no
> > > user interface (no Carbon or Cocoa).
> > >
> > > When I try to cross the 2 Go barrier, I get the following messages:
> > >
> > > *** malloc: vm_allocate(size=1069056) failed (error code=3)
> > > *** malloc[15141]: error: Can't allocate region
> > > *** malloc: vm_allocate(size=1069056) failed (error code=3)
> > > *** malloc[15141]: error: Can't allocate region
> > >
> > > and the program stops.
> > >
> > > This is on a dual G5 with 4 Go of RAM and enough hard disk space,
> > > running Mac OS X 10.3.5. I checked ulimit, there is no limit.

> >
> > That's a documented limitation in OS X. I can't remember if it's known
> > that this will change in 10.4.

>
> I don't think it's strictly a 2 GB limit -- rather, it has to do with
> how things are laid out in memory and a very carefully designed program
> may be able to allocate a little bit more.
>
> That said, you can access more memory on a G5 today with certain messy
> techniques involving something like mapping and unmapping large blocks
> of memory. Check the archives of Apple's darwin-development mailing
> list or ask DTS for details.
>
> -Eric


Sorry. Wetware memory failure. OS X has a limit of _four_ GB per
process. I'm not sure where I lost a bit.

--
Standard output is like your butt. Everyone has one. When using a bathroom,
they all default to going into a toilet. However, a person can redirect his
"standard output" to somewhere else, if he so chooses. - Jeremy Nixon
 
Reply With Quote
 
Dave Seaman
Guest
Posts: n/a
 
      09-07-2004, 11:57 PM
On Tue, 07 Sep 2004 21:43:40 GMT, Gregory Weston wrote:
> In article <ejalbert-866509.10272407092004@localhost>,
> Eric Albert <(E-Mail Removed)> wrote:


>> In article <(E-Mail Removed)>,
>> Gregory Weston <(E-Mail Removed)> wrote:
>>
>> > In article <(E-Mail Removed)>,
>> > Mats Weber <(E-Mail Removed)> wrote:
>> >
>> > > I need to allocate more than 2 Go of memory in a C++ program with no
>> > > user interface (no Carbon or Cocoa).
>> > >
>> > > When I try to cross the 2 Go barrier, I get the following messages:
>> > >
>> > > *** malloc: vm_allocate(size=1069056) failed (error code=3)
>> > > *** malloc[15141]: error: Can't allocate region
>> > > *** malloc: vm_allocate(size=1069056) failed (error code=3)
>> > > *** malloc[15141]: error: Can't allocate region


I have successfully allocated over 2 GB using malloc, but the amount to be
allocated must be unsigned if it is to fit into a 32-bit address.

--
Dave Seaman
Judge Yohn's mistakes revealed in Mumia Abu-Jamal ruling.
<http://www.commoncouragepress.com/index.cfm?action=book&bookid=228>
 
Reply With Quote
 
Mats Weber
Guest
Posts: n/a
 
      09-08-2004, 04:31 PM
In article <ejalbert-866509.10272407092004@localhost>,
Eric Albert <(E-Mail Removed)> wrote:

>I don't think it's strictly a 2 GB limit -- rather, it has to do with
>how things are laid out in memory and a very carefully designed program
>may be able to allocate a little bit more.


You are right, I did some more testing and with a very simple program I
was able to allocate 3.5 Go, so there is no 2 Go barrier:

#include <iostream>

int main()
{
for (int i = 0; ; ++i)
{
char* p = new char[1024 * 1024];

// make sure each page actually gets mapped by writing to it.
for (int j = 0; j < 1024; ++j)
p[1024 * j] = 77;

if (i % 100 == 0)
{
std::cout << i << "\n";
sleep(1);
}
}
}

Now if I modify the program to allocate 1 Ko blocks (much more
realistic, at least for what I am doing), the maximum I can allocate is
only 1.7 Go !

How can the memory allocator be so bad (note that there is absolutely no
fragmentation here) ?

Is there anyway around this ?
 
Reply With Quote
 
Mats Weber
Guest
Posts: n/a
 
      09-08-2004, 05:29 PM
BTW I just tried this on Linux (kernel version 2.4.18) and I was able to
allocate 3 Go in both cases: 1 Ko blocks and 1 Mo blocks.
 
Reply With Quote
 
Thomas Jahns
Guest
Posts: n/a
 
      09-09-2004, 02:18 PM
Mats Weber <(E-Mail Removed)> writes:

[...]

> Now if I modify the program to allocate 1 Ko blocks (much more
> realistic, at least for what I am doing), the maximum I can allocate is
> only 1.7 Go !
>
> How can the memory allocator be so bad (note that there is absolutely no
> fragmentation here) ?
>
> Is there anyway around this ?


The problem to allocate a large total amount of memory in small blocks
exists because memory is segmented (though not in the old x86
sense). One place in memory is your code and constant statically
allocated data (text segment), that's in most cases of negligible size
compared to total address space. Then there is the bss segment
containing uninitilized statically allocated data and the data segment
containing initialized statically allocated datafrom the executable who
both go into the data segment. Next there is a place for both, the heap
(growing towards higher addresses) and the stack (growing in the
opposite direction).

If these were the only segments, it would be easy to allocate nearly 4gb
in a 32 bit system but there is still the following two players in the
memory competition:

there is space reserved for mmap allocated memory (which malloc also
uses for very large chunks and which is also used for shared memory) and
the kernel is also mapped into every process for communication
purposes. Shared libraries go into the mmap space BTW.

Thus for every binary there is a number of restrictions on the placement
of different parts of the programs in memory. On IRIX the sizes of
heap/stack and mmap segments can be adjusted (by enlarging one while
making the other smaller), but I don't know if that is possible for
MacOS X.

The placement of large chunks in the mmap space explains why your
program can allocate more memory with large blocks and you might
consider using a layer above malloc to split large blocks into the
smaller ones you need.

However complicated the above description might sound, things are really
even much more involved. You might consider reading
<(E-Mail Removed)>
by Randolph J. Herber for more enlightenment or confusion, depending on
the time you can afford, about the relationship of virtual and real
memory (IMO a quite complex subject).

Thomas Jahns
--
"Computers are good at following instructions,
but not at reading your mind."
D. E. Knuth, The TeXbook, Addison-Wesley 1984, 1986, 1996, p. 9
 
Reply With Quote
 
 
 
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
allocating disk space for RAID by using partion magic 8 mamtim Abit 0 05-14-2007 11:55 AM
Unstable with more than 512 meg of memory. Brian Asus 7 10-12-2004 02:34 AM
Unstable with more than 512 meg of memory. Brian Asus 14 10-11-2004 03:07 PM
MS-6378: Allocating RAM to Video does not word Peter Tögel MSI 4 01-28-2004 02:54 PM
Memory errors when using more than one bank Jürgen Kosz Soyo 9 11-07-2003 05:33 PM


All times are GMT. The time now is 02:42 PM.


Welcome!
Welcome to Motherboard Point
 

Advertisment