Update: The fontconfig packages linked from this post seem to cause KDE bug #116176 with KDE3.5 packages. Well, I warned you.
Biggest at least in terms of slowness and memory usage. Yes, I'm talking about fontconfig checking all fonts and building an in-memory representation at application startup, as already mentioned in the KDE performance tips page. This fontconfig problem has had a noticeable impact on memory usage and startup time of KDE applications. But that has come to an end now it seems.
Thanks to Patrick Lam the latest development version of fontconfig now uses mmap-ed caches, in a similar way like KDE has been using ksycoca since ages. I tested it yesterday, and the bad news is, to my big disappointment it performed really poorly, even worse than the last stable version. But worry not, the good news is that it was just a minor problem and I have already a patch that fixes the issue and I'll send it to the fontconfig developers. In other words it now seems to work great.
For the brave ones I've made SUSE packages that you can try (fontconfig and fontconfig-devel). Remember to run 'fc-cache -f -v' after installing in order to create the new cache files. Of course, unsupported, use at your own risk, blah blah blah. And if you find a problem, you know who to tell (yes, right, the fontconfig developers).
Update: I've just found another bug. Don't run fc-cache while you have any application running. It seems fontconfig rewrites the old file with new contents instead of unlinking the old file first and writing a new file. That results in fontconfig crashes in the already running applications.