KDE and resource usage - how to get it wrong in several simple steps

Do you want to write something about KDE's memory usage? Simple, just follow these steps:

1. Launch KDE.
2. Run some random tool for measuring memory usage, preferably top.
3. Pick a column you think you know what it means. If you can't decide, just pick one, preferably something with big numbers (big numbers look better, remember).
4. Complain that the numbers are way too big. For higher bonus, compare it to something else, preferably something that gets nowhere near KDE's usage of shared resources (=libraries, mostly, but not only).
5. ???
6. Profit!

Really, it's as simple as that. So many people do it, you can too. See, for example, this review of KDE4.3. I have nothing against the review itself, since, I admit, I mostly skimmed over it, only two things caught my attention. First one was getting it backwards who copied from who the filtering feature in Present Windows compositing effect, which slightly amused me, and second one was the part talking about resources, which didn't.

The first item there is about kioslaves supposedly staying running for way too long, and it's not interesting here (although I like how it says "I suspect", like if checking it for real and then creating a bugreport is way too much work). The second item is about KWrited taking, imagine it, 18M of memory, and the last one is about the nVidia libraries wasting a good amount of memory, again.

Checking the KWrited item (I guess I was curious, or maybe bored) showed me several things. First of all, I don't have any KWrited daemon - openSUSE has libutempter, which allows building KWrited just as a KDED module. So instead I took KAccess, which should get similar treatment and not run on every system when it's mostly useless. Looking at numbers in top showed resident memory usage (RSS) 40M and shared memory usage (SHR) 25M. That was interesting for several reasons, like 40M being quite a lot, even for shared usage (since resident memory usage includes also all shared stuff, and so it doesn't really mean that much), or the fact that unshared memory usage for a small daemon being 40M-25M=15M is just plain rubbish.

The 40M is actually easy to explain - it is loaded using kdeinit, so it includes all memory from the basic libraries that are preloaded by kdeinit and shared by everything it launches. That includes the ~10M unshared bonus from nVidia libGL libraries that kdeinit helps to share.

The seeming 15M of unshared memory was more interesting and strange. Checking /proc/PID/status confirmed that data of the process is nowhere near that and that there must be something wrong going on. Checking /proc/PID/smaps in details showed similar results, again nothing anywhere near 15M. It is also not nVidia, since the memory is marked as dirty, but shared. And then it suddenly was obvious. The kernel (since that is whose numbers top blindly repeats) doesn't consider dirty shared memory to be really shared. A major portion of the 15M is the position-fixing of the non-PIC nVidia libraries, and some of it is also memory taken by symbol-binding to libraries. If a shared memory is dirty, then it may be dirty, but it is still shared. It got dirty before kdeinit started forking off new processes for launching, so it must be shared. Checking with Exmap, the only memory tool I trust so far, confirms. It is still some memory taken (and still big enough for those who don't know which year it is), but it's much smaller than what it seemed like.

Now, no wonder pretty much nobody can get KDE's resource usage right, when both people don't understand it and tools report nonsense. As it is said, two wrongs don't make a right. Here, they usually just make only a bigger wrong.

So, John, I think I might know what you could add to KSysGuard, after all. You could try to make KSysGuard a memory measuring tool that, unlike the rest, doesn't suck and make people write nonsense about KDE's resource usage. The magic line, in bash terms, is:

Where PID is pid and TYPE is:

• Size - that, as http://ktown.kde.org/~seli/memory/analysis.html says, means next to nothing in practice, so if you use it, make sure to hide it well.
• Rss - resident memory usage, all memory the process uses. While somewhat interesting as the grand total, not really that good number on its own, as a good portion of it is shared with many other process. Put it somewhere in the back.
• Shared - (i.e. Shared_Dirty and Shared_Clean together), the total amount of memory shared, what would SHR in top be, if it wasn't broken. Not really that interesting, unless somebody wants to admire how good we are at reusing stuff.
• Private - (again, both Private_Dirty and Private_Clean), the memory of the process itself, its own memory that is not shared with anything. It is probably worth the second memory column, after Pss below.
• Swap - I guess it says something useful too, but with my machine's swap being pretty much useless, I can't quite say :) .
• References - with documentation for smaps not being what it should be (that is, existing), I can't quite say what it is, but I think it is for measuring statistics about how much memory a process accesses over a time. There seems to be a way to reset the number. For KSysGuard, probably useless. However, when trying to find out in kernel sources about it, I found a nice little gem, the next entry.
• Pss - this is the thing I love about Exmap. It is not as good as what Exmap can do, but it's good enough. It means, according to the sources, Proportional Set Size, and it is Rss adjusted for sharing. If a process has 1M private and 10M shared between 10 processes in total, Pss is 1+10/1=2M. In other words, this is the number that can take into account the fact that KDE applications massively share memory. This should be THE memory column in KSysguard.

I hope this can work out for KSysGuard. I am slightly worried that fetching all these numbers from kernel will take too much time, but that needs to be tried. But, if it will work, do we all now know what will be so special about KSysguard?