OCT
15
2014

Ubuntu's Linux Scheduler or Why Baloo Might be Slowing Your System in 14.04

Submitted by Jonathan Riddell

Last month I posted about packaging and why it takes time. I commented that the Stable Release Update process could not be rushed because a regression is worse than a known bug. Then last week I was pointed to a problem where Baloo was causing a user's system to run slow. Baloo is the new indexer from KDE and will store all your files in a way you can easily search for them and was a faster replacement for Nepomuk. Baloo has been written to be as lightweight as these things can be using IONice, a feature of Linux which allows processes to say "this isn't very important let everyone else go first".

Except IONice wasn't working. Turns out Ubuntu changed the default Linux scheduler from CFQ to Deadline which doesn't support IONice. Kubuntu devs who had been looking at this for some time had already worked out how to change it back to the upstream defaults in our development version Utopic and in the backports packages we put on Launchpad. Last week we uploaded it as a proposed Stable Release Update and as expected the SRU team was sceptical. We should have been faster with the SRU which is our fault. They're there to be sceptical but the only change here is to go back to using upstream defaults. After much wondering why it was changed in the first place it seems that Unity was having problems with the CFQ scheduler and so it was changed, now we have suggestions that Baloo should be changed to adapt to that which is crazy. Nobody seems to have considered fixing Unity or that making the change in the scheduler in the first place would affect software outside of Unity. We tried taking the issue to the Ubuntu Technical Board but their meeting didn't happen this week.

So alas no fix in the immediate future, if it bothers you best use Kubuntu Backports. When someone on the SRU team is brave enough to approve it into -proposed we'll put out a call for testers and it'll get into -updates eventually. It's what happens when you have a large project like Ubuntu with many competing demands, but it would be nice if the expectation was on Unity to get fixed rather than on Kubuntu to deal with the bureaucracy to workaround their workarounds.

Comments

Hi,
for the record, there are another two options for people affected by this:

1. Change the scheduler at runtime
echo "cfq" > /sys/block/sda/queue/scheduler

2. Change the default scheduler on the kernel cmdline from grub:
Add elevator=cfq to the kernel command line.

Rolf

By Rolf at Wed, 10/15/2014 - 13:15

Hi Jonathan,
this is indeed annoying. But can't affected users just change the scheduler manually like described here, at least as a temporary solution?
http://www.linuxhowtos.org/System/iosched.htm
Best,
Michael

By Mike at Wed, 10/15/2014 - 13:16

It's not a Unity issue. Deadline scheduler is better for server loads, and also greatly improves responsiveness when desktop users copy large files, or use VM software.

Turning it back to CFQ will improve Baloo, at the expense of Kubuntu users who copy large files or use virtual machines on their desktop.

By mdeslaur at Wed, 10/15/2014 - 16:10

Actually, with SSD popularity CFQ scheduler will be even less used in the feature. You should never impose stuff like your IO scheduler of choice on the user. That's pretty much means - you should thank Ubuntu guys to pointing you a bug in Baloo ;)

By DeadLock at Thu, 10/16/2014 - 07:30

I'm running on ZFS, using a custom-built (Kubuntu) kernel with the ConKolivas patches and the CFQ scheduler (though I really ought to try the noop one). How does that work out for baloo?
More to the point: can baloo be configured to index only email, like nepomuk could?

By RJVB at Thu, 10/16/2014 - 09:40

Why is the suggestion that Baloo should be patched to gracefully handle non-default schedulers crazy?

I have nothing to do with either camp involved here, but "nobody considered fixing their software in a way that works in my use case" followed by complaints "they had some crazy idea about fixing my software in a way that works for their use case" seems hypocritical in the extreme.

There are several blogs/G+ posts on this issue (which is itself odd), is there some underlying feud, or is this really about technical purists thinking their way is superior?

By Huw at Thu, 10/16/2014 - 12:23

I don't think it's fair to immediately assume that Unity is the problem here. Why does Baloo not work with any Linux scheduler? Or is this something that should be fixed in the deadline scheduler?

Until these questions are answered in a deeper technical analysis, I don't think it's reasonable to make a snap judgement like this.

Just because the deadline scheduler is not the default does not immediately make it an incorrect choice if some other software has a problem using it.

There have been statements flying around the mailing lists that "no other distro uses it", and these statements have been refuted by others. I have yet to see a proper analysis or report about this.

It's what happens when you have a large project like Ubuntu with many competing demands, but it would be nice if the expectation was on Unity to get fixed rather than on Kubuntu to deal with the bureaucracy to workaround their workarounds.

It would be nice if the expectation was for everybody to work together, get to the bottom of the problem, and fix it in a way that works well for all stakeholders, rather than focus on where to put the blame.

By KDE User at Thu, 10/16/2014 - 14:10

A desktop with good search functionality will have to read the content of all files. This requires a lot of CPU time. Writing very fast indexing code helps to cull the total amount of CPU required. Making sure that the indexer does not interfere with real-time tasks is unfortunately harder. On would hope to rely on the linux scheduling.

If that does not work as expected, fixing the issue upstream (linux kernel) is a good approach but a slow one. Until the kernel is fixed people will still complain that the indexer slows down the user experience.

The CPU and IO schedulers are not the only issue. The indexer constantly reads files and this will invalidate the cache. A file indexer should use POSIX_FADV_DONTNEED to avoid doing that.

The CPU and IO scheduling may be faulty but the indexer could keep it's own accounting. That is ugly and not trivial. Simply pausing the indexer for some percentage of the time would generally fix the problem. It would take longer for the indexing to finish, but it would improve user experience.

By Jos van den Oever at Sun, 10/19/2014 - 00:02

Am somewhat a Linux intermediate but been troubleshooting for decades. Is this a logical point or am I missing the mark?

Why not silo the whole indexing and its caching process? Most, if not, all hardware is multi-CPU. Also, there is frequently enough RAM and/or disk space to hold a separate cache. So, resources are beginning to be less of a bottleneck but indexing is still somewhat ancient due to it still being resource heavy. Since it hasn't changed that much giving it its own set of resources independently shouldn't be much of a problem. Create a separate partition for indexing cache that can be updated independently; assign one dedicated core for indexing leaving the others for productivity/system processes; and assign a bank of RAM. Then make the scheduling configuration really intuitive for users based on the amount of resources they are using, and also learning about when the best times for indexing are. In other words, during regular periods of just surfing the web, not many resources are in use compared to times of graphics rendering, etc. So, create a little artificial intelligence (really monitoring) to keep track of things and start/stop indexing, ie, smart indexing scheduler.

Also, since most applications will dedicate to using one CPU or core anyway the performance hit for productivity will be reduced instead of interference. It's like the old adage, water will seek its own level but if you keep building a bigger hole water will continue to find the edges. This means that indexing seeks to capture and use as many resources as is available without much throttling. So, the more resources available, the more indexing will seek to use.

By Bruce Wolfe at Mon, 10/20/2014 - 18:33

Does the mentioned performance problem only present on a HDD or would a SSD also be affected? The deadline scheduler is said to be better for SSDs.

By KDE User at Tue, 10/21/2014 - 03:28

Pages