On Mon, Feb 07, 2005 at 07:04:53AM -0400, bulia byak wrote:

Hmm, if it does fix it, I think we should take it because the bug is extremely annoying and the fix appears simple. I wonder if you can now do a pre3 with that bug, including windows build, so that everyone could test it?

I've cut pre3. I can't do a "proper" windows exe installer until later tonight. If you want a tarball of the build, though, I can put that up.

The gdkdraw.c: line 1351 warning bug seems to be really fixed in pre4. Thanks Kees!

Now we have this new one:

http://sourceforge.net/tracker/index.php?func=detail&aid=1118350&gro...

I tried this and never could really freeze it, though the memory was being eaten fast. In the past, I had similar lock-ups when all of the memory including virtual was eaten. According to Mental, this is how the garbage collector works - it does not release memory immediately. And indeed, I pushed memory consumption to 50% by stamping several hundred copies, then deleted them all (memory stayed at 50%), then did stumping again - and memory did not grow this time, i.e. the old memory was reused.

So what I propose with this bug is:

- The original reporter (inkblotter) please test 0.40 and see if it behaves the same for you (I think it does, but let's check that).

- Assign this bug to Mental and ask his opinion on whether we can release with this bug or not.

- It may be, of course, that the reason for the freeze is not memory-related. So everyone please test pre4 against this bug and report your findings.

Quoting bulia byak <buliabyak@...400...>:

According to Mental, this is how the garbage collector works - it does not release memory immediately.

This is true, but I wonder if I've mislead you a bit.

As you know, many automatic garbage collectors (like libgc) only free and recycle memory periodically. This means you may have some extra slush that could be freed, but hasn't yet.

There are other forces at work, though...

Pretty much all allocators, whether automatic or not, whether the system malloc() or some custom allocator like libgc's, work the same way: they request large blocks of memory from the operating system, then divvy those blocks into smaller ones internally to satisfy application allocation requests.

When an application frees memory, that memory is usually recycled internally rather than returned to the OS immediately. The reason for this is that the large memory blocks acquired from the OS must be completely unused before they can actually be freed.

Let's say for example that an allocator acquires 16 8MB blocks from the OS in response to 32768 4k application allocations...

In a worst-case scenario, it's possible that the application could free 32752 of those 4k blocks but the remaining 16 4k just happen to be distributed across the 16 8MB blocks requested from the OS.

If that happens, from the application's point of view it may only have 64k allocated, but as far as the OS is concerned, it's still using 128MB!

Note that this applies to nearly all allocators in common use.

While it's unusual for things to get quite that bad, memory fragmentation is common enough that many popular allocators (for example Perl's) simply don't bother trying to return memory to the OS at all (the memory will still get forcibly reclaimed by the OS when the process exits).

[ FWIW, libgc's allocator is one of the ones that _does_ make an effort to release memory to the OS, but it is limited by fragmentation like any other ]

Also note that for various reasons, the statistics you get from the OS aren't going to directly reflect the amount of heap-allocated memory. Be careful drawing conclusions from only looking at e.g. the output of top(1)...

(the worst thing is that due to the modern practice of overcommitting memory, the OS may literally lie to an application about the amount of memory it is being given, hoping the application won't really try to use it all)

The best approach to evaluating memory usage is if you can ask the allocator for information on memory usage directly, as that matches the world from the point of view of the application.

I think libgc provides means to get statistics; I don't know about the glibc allocator though. I am very tempted to add a memory usage dialog like Batik has...

-mental

Quoting bulia byak <buliabyak@...400...>:

Which numbers were you looking at? Virtual size? RSS? Did you compare Shared? Be aware that all of those are significantly affected by things other than heap allocation.

The next thing I'll test is an Inkscape build from the time when we used the new renderer but didn't use boehm yet. This will show who (renderer or boehm) is guilty of this gobbleage.

That's not a useful comparison; some other substantial things have changed memory-wise as well (SPRepr is now larger, for example, and we use many more shared libraries whose code counts as part of our memory usage).

Since livarot doesn't use the collector yet, you'd be much better off directly comparing the amount of memory allocated using the system allocator versus the amount of memory allocated using libgc's (and their respective heap sizes).

For libgc:

#include <gc/gc.h> int gc_mem_total = GC_get_heap_size(); int gc_mem_unused = GC_get_free_bytes();

note: gc_mem_unused is a pessimistic estimate; compare these numbers immediately before and after a call to GC_gcollect()

For glibc:

#include <malloc.h> struct mallinfo info = mallinfo(); int sys_mem_total = info.arena + info.hblkhd; int sys_mem_unused = info.fordblks;

note: I haven't verified that this is correct (it might be too small)...

Note in both cases that this won't let you blame one allocator or the other -- it's not the fault of an allocator if it's being asked to allocate more memory -- but it will narrow down who has the hungrier clients.

If the standard allocator accounts for most of the memory usage, then we still need to investigate further whether that really means livarot (though I think it is).

Btw, if it looks like a lot is being allocated using the garbage-collected allocator, check how much of that is accounted for in in the variable GC_non_gc_bytes... if that's a large percentage of the gc heap size I might be inclined to blame the present implementation of GC::Anchored.

But any way, at the moment we need to decide: do we release with that bug or not? Mental, please review the original report, try to reproduce, and give us your opinion.

I'll have a look later tonight.

-mental

On Tue, 8 Feb 2005 14:00:48 -0800, Kees Cook <inkscape@...62...> wrote:

On Tue, Feb 08, 2005 at 04:35:47PM -0400, bulia byak wrote:

...

But anyway, at the moment we need to decide: do we release with that bug or not? Mental, please review the original report, try to reproduce, and give us your opinion.

I think we need to release with these problems. They're not show-stoppers, IMHO, just annoying. It's okay to have bugs, as long as most people can workaround the problems.

Sure, but in this case, the question is if this bug is new or not. If it's a regression compared to 0.40, that will push it much closer to the must-fix-before-0.41 status.

On Tue, 2005-02-08 at 15:35, bulia byak wrote:

This seems to be true. I just tested Sodipodi and it also did not decrease its memory usage after all 1000+ stamped shapes were deleted.

Well, another factor there is that for many major structs in Sodipodi, it has private memory pools implemented on top of the system allocator. Memory in those pools is never released even with free(), so Sodipodi cannot shrink, with or without fragmentation.

I tore all that stuff out after the fork.

It can be a legitimate optimization if the system malloc is really inefficient and memory allocation is a bottleneck, but I am not sure that was ever measured to be the case.

-mental