Redesigning Lokalize's Translation Memory Tab - GSoC Week 9

After working on the UI of TM Tab in Lokalize during the first phase of GSoC, I got back to work on the backend. Currently, only a single TM can be searched in TM Tab. My changes enable querying across multiple selected TMs as per translators' feature request.

A little about threads and mutexes before I share week-wise updates. The GUI runs on the main thread, while TM operations (querying, opening/closing databases, removing files) run as jobs on a separate worker thread pool, TM::threadPool(). This pool is explicitly capped to a single worker thread at startup- setMaxThreadCount(1). This means TM jobs never actually run concurrently with each other, they queue and execute strictly one at a time (this can be changed given my new changes but will have to be careful).

QSqlDatabase connections can only be used on the thread that created them, which is why each job looks up or clones a connection specific to the current thread before querying.

A queued connection is what lets a signal emitted on the worker thread safely invoke a slot on the GUI thread: rather than running the slot's code immediately on the emitting thread.

A mutex protects shared or concurrently-accessed data.

Week 7

I spent week 7 exploring other approaches for quering multiple DBs and merging the results. QSqlQueryModel is only useful to get query results from a single db. It just enabled std::move(*job->query) and there was no way to append results from other TMs.

I felt that my proposed approach, to fire N ExecQueryJobs, merge them in an in-memory SQLite db, and then pass it on to the view, was making things unnecessarily complex and somewhat redundant. I'm sharing the discarded ideas as well in case it helps someone in the future.

• Use ATTACH to add other db(s) and perform UNION on the search query. There were some concerns over the complexity of ATTACH and also Lokalize supports remote TMs (PostgreSQL) which doesn't have the provision of ATTACH.
• Then, I looked into QConcatenateTablesProxyModel which some KDE apps already use, but there is some bug with its working with QSortFilterProxyModel. I could have had "N TMDBModel instances" (which doesn't seem like a good idea) for each TM and later join them.

Week 8

I decided upon QAbstractTableModel. Unlike QSqlQueryModel (which is a subclass of it), it's more flexible and I had to refactor some of the code. This also meant lazy fetch couldn't be preserved and the results would be stored in QSqlRecords (to preserve all the existing usage on results).

Manually tested against a single TM. Rewrote the TMJob Test too. I find it strange that we always write a test for it to pass. Also from one of the articles I read: "Use test driven development. When you write the test just before the production code, you would never write a monster test, would you?"

Week 9

setFilter() is modified such that ExecQueryJob is fired for each TM. slotQueryExecuted() accumulates each job's rows into the model as they arrive, and only fires resultsFetched() once every selected TM's job has reported back. Row count is also reported directly at the end.

Well, contrary to what I wrote in the last blog, I did end up with an approach quite similar to what TMView does. Are mentors always right?

The part on which I spent a lot of time was what happens if someone retypes their query, or checks/unchecks a TM, while the previous search's jobs are still mid-flight. Stale results shouldn't be allowed to land on top of a newer search's rows. The fix is a generation counter to keep track.

After a lot of coding and debugging, seeing the resuts from multiple TMs made the entire hard work finally pay off.

While testing, the bug mentioned in the previous blog kept pestering me so I fixed it too. I was so fixated on empty/short string that I did not think that the place it was being used could be a problem. It's nice to see the Art of Debugging materialize out of pages of Roger S. Pressman's textbook.

I'll begin to work on TM View once these changes are reviewed and merged.