AUG
10
2009

KDE and resource usage - how to get it wrong in several simple steps

Do you want to write something about KDE's memory usage? Simple, just follow these steps:

  1. Launch KDE.
  2. Run some random tool for measuring memory usage, preferably top.
  3. Pick a column you think you know what it means. If you can't decide, just pick one, preferably something with big numbers (big numbers look better, remember).
  4. Complain that the numbers are way too big. For higher bonus, compare it to something else, preferably something that gets nowhere near KDE's usage of shared resources (=libraries, mostly, but not only).
  5. ???
  6. Profit!

Really, it's as simple as that. So many people do it, you can too. See, for example, this review of KDE4.3. I have nothing against the review itself, since, I admit, I mostly skimmed over it, only two things caught my attention. First one was getting it backwards who copied from who the filtering feature in Present Windows compositing effect, which slightly amused me, and second one was the part talking about resources, which didn't.

The first item there is about kioslaves supposedly staying running for way too long, and it's not interesting here (although I like how it says "I suspect", like if checking it for real and then creating a bugreport is way too much work). The second item is about KWrited taking, imagine it, 18M of memory, and the last one is about the nVidia libraries wasting a good amount of memory, again.

Checking the KWrited item (I guess I was curious, or maybe bored) showed me several things. First of all, I don't have any KWrited daemon - openSUSE has libutempter, which allows building KWrited just as a KDED module. So instead I took KAccess, which should get similar treatment and not run on every system when it's mostly useless. Looking at numbers in top showed resident memory usage (RSS) 40M and shared memory usage (SHR) 25M. That was interesting for several reasons, like 40M being quite a lot, even for shared usage (since resident memory usage includes also all shared stuff, and so it doesn't really mean that much), or the fact that unshared memory usage for a small daemon being 40M-25M=15M is just plain rubbish.

The 40M is actually easy to explain - it is loaded using kdeinit, so it includes all memory from the basic libraries that are preloaded by kdeinit and shared by everything it launches. That includes the ~10M unshared bonus from nVidia libGL libraries that kdeinit helps to share.

The seeming 15M of unshared memory was more interesting and strange. Checking /proc/PID/status confirmed that data of the process is nowhere near that and that there must be something wrong going on. Checking /proc/PID/smaps in details showed similar results, again nothing anywhere near 15M. It is also not nVidia, since the memory is marked as dirty, but shared. And then it suddenly was obvious. The kernel (since that is whose numbers top blindly repeats) doesn't consider dirty shared memory to be really shared. A major portion of the 15M is the position-fixing of the non-PIC nVidia libraries, and some of it is also memory taken by symbol-binding to libraries. If a shared memory is dirty, then it may be dirty, but it is still shared. It got dirty before kdeinit started forking off new processes for launching, so it must be shared. Checking with Exmap, the only memory tool I trust so far, confirms. It is still some memory taken (and still big enough for those who don't know which year it is), but it's much smaller than what it seemed like.

Now, no wonder pretty much nobody can get KDE's resource usage right, when both people don't understand it and tools report nonsense. As it is said, two wrongs don't make a right. Here, they usually just make only a bigger wrong.

So, John, I think I might know what you could add to KSysGuard, after all. You could try to make KSysGuard a memory measuring tool that, unlike the rest, doesn't suck and make people write nonsense about KDE's resource usage. The magic line, in bash terms, is:


echo 0 $(cat /proc/PID/smaps | grep TYPE | awk '{print $2}' | sed 's#^#+#' ) | bc

Where PID is pid and TYPE is:

  • Size - that, as http://ktown.kde.org/~seli/memory/analysis.html says, means next to nothing in practice, so if you use it, make sure to hide it well.
  • Rss - resident memory usage, all memory the process uses. While somewhat interesting as the grand total, not really that good number on its own, as a good portion of it is shared with many other process. Put it somewhere in the back.
  • Shared - (i.e. Shared_Dirty and Shared_Clean together), the total amount of memory shared, what would SHR in top be, if it wasn't broken. Not really that interesting, unless somebody wants to admire how good we are at reusing stuff.
  • Private - (again, both Private_Dirty and Private_Clean), the memory of the process itself, its own memory that is not shared with anything. It is probably worth the second memory column, after Pss below.
  • Swap - I guess it says something useful too, but with my machine's swap being pretty much useless, I can't quite say :) .
  • References - with documentation for smaps not being what it should be (that is, existing), I can't quite say what it is, but I think it is for measuring statistics about how much memory a process accesses over a time. There seems to be a way to reset the number. For KSysGuard, probably useless. However, when trying to find out in kernel sources about it, I found a nice little gem, the next entry.
  • Pss - this is the thing I love about Exmap. It is not as good as what Exmap can do, but it's good enough. It means, according to the sources, Proportional Set Size, and it is Rss adjusted for sharing. If a process has 1M private and 10M shared between 10 processes in total, Pss is 1+10/1=2M. In other words, this is the number that can take into account the fact that KDE applications massively share memory. This should be THE memory column in KSysguard.

I hope this can work out for KSysGuard. I am slightly worried that fetching all these numbers from kernel will take too much time, but that needs to be tried. But, if it will work, do we all now know what will be so special about KSysguard?

Comments

KSysGuard attempts to do something similar.

It shows RSS-SHR in one column ("Memory") and SHR in another column ("Shared").

You mention that top's SHR is broken - presumably this means that /proc/statm numbers are broken as well, which ksysguard relies on.

One interesting thing though - I'm getting bug reports that that RSS-SHR is sometimes negative. Any ideas how that can happen?


By John Tapsell at Mon, 08/10/2009 - 21:52

Maybe SHR still includes the memory that is swapped out. RSS should only be memory that is not swapped out. Given this negative numbers can occur.


By ponto at Tue, 08/11/2009 - 08:08

Btw, kwrited numbers for me:

Memory: 1.1MB (RSS-SHR)
Shared Mem: 6.0MB (SHR)


By John Tapsell at Mon, 08/10/2009 - 21:54

I'm a fan of exmap; there's another interesting tool - parsing smaps - you might not be aware of: smem.


By lucke at Mon, 08/10/2009 - 22:25

Very instructive post, thanks!

As an Awk lover, I just could not resist rewriting your shell line as one Awk invocation:

awk '$0 ~ "TYPE" {total+=$2} END { print total}' /proc/PID/smaps

:)


By aurélien gâteau at Tue, 08/11/2009 - 09:17

I asked a few other people to run this time command, and one person got:

real 0m0.465s

:-/


By John Tapsell at Tue, 08/11/2009 - 13:17

Somehow my original post got lost..

time cat /proc/*/smaps > /dev/null

Takes 0.1 seconds on my system. Updating once every other second would still take 5% of CPU. And on other systems it can take up 0.5 seconds.

In conclusion, I can't really use these results in ksysguard. But I do want to add a single-process page where this information can be added.


By John Tapsell at Tue, 08/11/2009 - 14:52

time cat /proc/*/smaps > /dev/null

real 0m0.052s
user 0m0.012s
sys 0m0.044s


By vlad88sv at Thu, 08/13/2009 - 15:41

Very rough, this measurement, but it show acceptable results for many needs

# kdm
# free
total used free shared buffers cached
Mem: 2046684 291964 1754720 0 30024 192960
-/+ buffers/cache: 68980 1977704
Swap: 0 0 0

# kde4.3 with konsole, kate, [email protected]
# free
total used free shared buffers cached
Mem: 2046684 768708 1277976 0 47408 457120
-/+ buffers/cache: 264180 1782504
Swap: 0 0 0

# kde4.3 with konsole, kate, [email protected] flush caches
# echo 3 > /proc/sys/vm/drop_caches
# free
total used free shared buffers cached
Mem: 2046684 408524 1638160 0 3484 154956
-/+ buffers/cache: 250084 1796600
Swap: 0 0 0


By vivo75 at Wed, 08/12/2009 - 09:45