kmail hangs with 100% CPU

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

kmail hangs with 100% CPU

Achim Gratz

Since the KDEPIM update in August akonadi would at random times start
using up CPU and had some errors in the mysql error log about deadlocks.
So I moved the ~/.local/share/akonadi directory aside and let it create
a new one.  That worked and it also seemed to have indexed some stuff
that for whatever reason it hadn't before.  However, kmail now
completely freaks out when encountering large folders (they're local
maildirs) and just hangs with 100% CPU and no further reaction.  It
doesn't appear to try and communicate with akonadi or anything else, it
just hangs.  Sometimes it tells it has either threaded or processed some
large percentage of the entries (however the total number doesn't match
what's in the folder), but there is absolutely no reaction to whatever
attempts at leaving the folder or anything else.

The largest folder that still works has 4091 entries, the next size up
that fails is 5506 mails (about 200 of that are in a sub-folder,
though).

So I've moved the old directory back and I can use these folders again.
The wierd background activity from mysqld (CPU + I/O, both can be
stopped by doing an "akonadictl restart", but will return after some
activity in kmail) is back as are these errors in mysql.err:

2017-09-10 18:52:16 7f5328a0fb00 InnoDB: Error: Fetch of persistent statistics requested for table "akonadi"."tagtypetable" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead.
2017-09-10 18:53:32 7f530abff700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch).

Does anybody have an idea what is going on and how to fix it properly?


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Martin Herkt
On 2017 M09 10, Sun 19:03:18 CEST Achim Gratz wrote:
> Does anybody have an idea what is going on and how to fix it properly?

No idea what’s going on. Been having other trouble with the MySQL backend
myself, more than half a year ago. It wouldn’t index properly and sometimes
fail to move items to different directories (INBOX to Trash in particular).
Restarting Akonadi or rebuilding the local database would sometimes mitigate
the issue, but it wasn’t a long-term solution.

I’ve since switched to the PostgreSQL backend, which seems to work perfectly
with no issues (plus it’s faster).

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Achim Gratz
Martin Herkt writes:
> I’ve since switched to the PostgreSQL backend, which seems to work perfectly
> with no issues (plus it’s faster).

So how do I do that?  Is there some tutorial or step-by-step someplace?


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Martin Herkt
On 2017 M09 11, Mon 19:52:50 CEST Achim Gratz wrote:
> Martin Herkt writes:
> > I’ve since switched to the PostgreSQL backend, which seems to work
> > perfectly with no issues (plus it’s faster).
>
> So how do I do that?  Is there some tutorial or step-by-step someplace?

There used to be a configuration thing for Akonadi in systemsettings which
allowed you to change the storage backend, but that seems to be gone now :(

Basically make sure libQt5Sql5-postgresql and postgresql-server are installed,
and change Driver=QMYSQL to Driver=QPSQL in ~/.config/akonadi/akonadiserverrc.

Then restart Akonadi (akonadictl restart). Should work.

There’s an SQLite backend for Akonadi as well (requires a separate package),
but I don’t know how mature it is. Last time I tried that, it was quite slow
and unreliable, but that was years ago and things might have changed.

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Luca Beltrame
Il giorno Mon, 11 Sep 2017 23:44:51 +0200
Martin Herkt <[hidden email]> ha scritto:

> Then restart Akonadi (akonadictl restart). Should work.

Note that *no* migration of the data is performed. This is why the
option was removed from akonadiconsole, to prevent people from
accidentally destroying their data.

--
Luca Beltrame - KDE Forums team
GPG key ID: A29D259B

attachment0 (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Wolfgang Bauer
Am Dienstag, 12. September 2017, 07:25:05 CEST schrieb Luca Beltrame:
> Il giorno Mon, 11 Sep 2017 23:44:51 +0200
>
> Martin Herkt <[hidden email]> ha scritto:
> > Then restart Akonadi (akonadictl restart). Should work.
>
> Note that *no* migration of the data is performed. This is why the
> option was removed from akonadiconsole, to prevent people from
> accidentally destroying their data.

It's no different to wiping out the database though (i.e. deleting/moving away
~/.local/share/akonadi/), as the OP already did... ;-)

I don't see the described problem here btw on Leap 42.3 and kdepim 17.08.1
(17.08.0 worked fine too), with the mysql backend.
But there is an upstream bug report that sounds similar:
https://bugs.kde.org/show_bug.cgi?id=384359

Kind Regards,
Wolfgang

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Achim Gratz
Wolfgang Bauer writes:
> It's no different to wiping out the database though (i.e. deleting/moving away
> ~/.local/share/akonadi/), as the OP already did... ;-)

Well, I moved the old one back for now.  Anyway, I'm curious to know
what data I'd supposedly lose by doing that.

> I don't see the described problem here btw on Leap 42.3 and kdepim 17.08.1
> (17.08.0 worked fine too), with the mysql backend.
> But there is an upstream bug report that sounds similar:
> https://bugs.kde.org/show_bug.cgi?id=384359

I don't think this describes the same situation.  In that bugreport,
mysqld and akonadi eat CPU and do disk I/O.  In my case it's kmail alone
(and no I/O I can see).  Is there a way to figure out what kmail tries
to do?  Based on the suspicious relation to the folder size there might
be something (relatively) easy to fix going on.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Achim Gratz
In reply to this post by Martin Herkt
Martin Herkt writes:

> On 2017 M09 11, Mon 19:52:50 CEST Achim Gratz wrote:
>> Martin Herkt writes:
>> > I’ve since switched to the PostgreSQL backend, which seems to work
>> > perfectly with no issues (plus it’s faster).
>>
>> So how do I do that?  Is there some tutorial or step-by-step someplace?
>
> There used to be a configuration thing for Akonadi in systemsettings which
> allowed you to change the storage backend, but that seems to be gone now :(
>
> Basically make sure libQt5Sql5-postgresql and postgresql-server are installed,
> and change Driver=QMYSQL to Driver=QPSQL in ~/.config/akonadi/akonadiserverrc.
>
> Then restart Akonadi (akonadictl restart). Should work.

Oh OK.  Based on some descriptions from Debian I was under the
impression I'd need some akonadi-*-postgresql package for this to work.

> There’s an SQLite backend for Akonadi as well (requires a separate package),
> but I don’t know how mature it is. Last time I tried that, it was quite slow
> and unreliable, but that was years ago and things might have changed.

I don't want to read too much into those forum discussions, but if what
I read is close enough to the truth the expected performance would not
be great.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Luca Beltrame
In reply to this post by Achim Gratz
Il giorno Tue, 12 Sep 2017 20:22:56 +0200
Achim Gratz <[hidden email]> ha scritto:

> alone (and no I/O I can see).  Is there a way to figure out what
> kmail tries to do?  Based on the suspicious relation to the folder
> size there might be something (relatively) easy to fix going on.

If you can install the relevant debug symbols, attach gdb and provide a
backtrace of kmail in that state it would be very useful in figuring
out what's going on.

--
Luca Beltrame - KDE Forums team
GPG key ID: A29D259B

attachment0 (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Wolfgang Bauer
In reply to this post by Achim Gratz
Am Dienstag, 12. September 2017, 20:22:56 CEST schrieb Achim Gratz:
> Well, I moved the old one back for now.  Anyway, I'm curious to know
> what data I'd supposedly lose by doing that.

I answered a similar question here recently:
https://lists.opensuse.org/opensuse-factory/2017-08/msg00623.html

There may be other things stored in the database though, in general I would
expect this to be the case for all "special" things that are not supported by
the "conventional" storage backends (maildir, IMAP, ...), e.g. certain mail
flags that are not supported by your IMAP server, custom tags, things like
that.
If that could be a problem for you depends on your usage I suppose.

> I don't think this describes the same situation.  In that bugreport,
> mysqld and akonadi eat CPU and do disk I/O.  In my case it's kmail alone
> (and no I/O I can see).

Well, you did mention high CPU usage and disk I/O by Akonadi and mysql too in
your original mail...
But ok, I read it completely again, and those problems apparently seem to be
fixed by removing the database.

If its actually kmail that uses up the CPU now, I highly doubt that switching
the Akonadi database backend would have any effect though.
Kmail only talks to Akonadi, not the database backend.

As a side-note:
The mysql errors mentioned do sound to me like the database got corrupted
somehow... (the mysql internal part, not Akonadi's tables)

> Is there a way to figure out what kmail tries
> to do?

Getting a backtrace with gdb, as Luca wrote.

According to your problem description, it might be somehow related to message
threading/grouping though, so try to turn that off maybe and see if it helps.
I.e. try to switch to one of the "flat" options in
 View->Message List->Aggregation.

Another thing that would probably be worth a try is to rename
~/.local/share/kmail2/ and/or the settings file
(~/.config/kmail2rc).
I think I heard about problems with corrupted autosaved mails in the past, and
I think saved searches can (or could in the past) also cause such problems,
but I'm not sure at the moment where those are saved as I never used them
myself (maybe ~/.local/share/kmail2/ as well).
Although, IIUIC the problem only occurs with certain mail folders, so that
probably won't help then.

Kind Regards,
Wolfgang

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Achim Gratz
In reply to this post by Luca Beltrame
Luca Beltrame writes:
> If you can install the relevant debug symbols, attach gdb and provide a
> backtrace of kmail in that state it would be very useful in figuring
> out what's going on.

I was hoping for something less intrusive, like snooping the dbus
messages.  Not sure if I find the time to install all that stuff and
then run things in the debugger.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Achim Gratz
In reply to this post by Wolfgang Bauer
Wolfgang Bauer writes:
> Am Dienstag, 12. September 2017, 20:22:56 CEST schrieb Achim Gratz:
>> Well, I moved the old one back for now.  Anyway, I'm curious to know
>> what data I'd supposedly lose by doing that.
>
> I answered a similar question here recently:
> https://lists.opensuse.org/opensuse-factory/2017-08/msg00623.html

Hmm. I guess I won't have a problem then.  As I said, all mail is
already local and any flags that may have been stored somewhere were
lost anyway during the botched kmail2 transition since I had to manually
import all mails at that time.

> Well, you did mention high CPU usage and disk I/O by Akonadi and mysql too in
> your original mail...

Yes, that's back with the old (somewhat damaged) database back in use at
the moment.  It mostly happens when I send a new mail and a restart of
akonadi gets rid of it, so I can cope for now.

> But ok, I read it completely again, and those problems apparently seem to be
> fixed by removing the database.

Yes.  Just that I get another problem I don't have a workaround for.

> If its actually kmail that uses up the CPU now, I highly doubt that switching
> the Akonadi database backend would have any effect though.

Well, I don't really know what happens and it really seems to be
triggered by moving above the 4096 barrier, so maybe if the database
backends serve up the data in smaller or larger chunks this might make
a difference.  There is no discernible delay when opening a slightly
smaller folder, so I doubt some superlinear algorithm triggers this.

> Kmail only talks to Akonadi, not the database backend.

I _think_ that the problem is precisely that Akonadi does deliver all
the data.  Currently these large folders have a notceable delay when
entering them before any entries are shown, but eventually they will be
(and kmail remains responsive in the meantime, so I can switch to a
different folder for instance).

> According to your problem description, it might be somehow related to message
> threading/grouping though, so try to turn that off maybe and see if it helps.
> I.e. try to switch to one of the "flat" options in
>  View->Message List->Aggregation.

Hmm. Could be although I got at least once a status message from kmial
that it had threaded something close to the number of messages in the
folder before it stopped responding.

> Another thing that would probably be worth a try is to rename
> ~/.local/share/kmail2/ and/or the settings file
> (~/.config/kmail2rc).

I did that once while trying to fixup things while trsnsitioning from
kmail2 and I don't want to do it again if I can avoid it.

> I think I heard about problems with corrupted autosaved mails in the past, and
> I think saved searches can (or could in the past) also cause such problems,
> but I'm not sure at the moment where those are saved as I never used them
> myself (maybe ~/.local/share/kmail2/ as well).

I don't think that's a problem or at least I wouldn't know where these
would come from.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kmail hangs with 100% CPU

Knurpht-openSUSE
In reply to this post by Martin Herkt
Op maandag 11 september 2017 14:45:37 CEST schreef Martin Herkt:

> On 2017 M09 10, Sun 19:03:18 CEST Achim Gratz wrote:
> > Does anybody have an idea what is going on and how to fix it properly?
>
> No idea what’s going on. Been having other trouble with the MySQL backend
> myself, more than half a year ago. It wouldn’t index properly and sometimes
> fail to move items to different directories (INBOX to Trash in particular).
> Restarting Akonadi or rebuilding the local database would sometimes mitigate
> the issue, but it wasn’t a long-term solution.
>
> I’ve since switched to the PostgreSQL backend, which seems to work perfectly
> with no issues (plus it’s faster).

Thanks for this. Got 9 accounts, ~ 120,000 emails ( kept everything from 2001
on, except spam ) and faster? Oh yeah, much faster. I never had any real
trouble with akonadi/mysql until today. Akonadi indexer is still busy, but the
overall behaviour of kmail re. opening huge folders is much snappier.

--
Gertjan Lettink, a.k.a. Knurpht

openSUSE Board Member
openSUSE Forums Team
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]