not restarting the docker service on running zypper update?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

not restarting the docker service on running zypper update?

Jordi Massaguer Pla-2
Hi packagers,

I am packaging docker and we have this problem that if a user runs an
unattended update, docker gets updated and so restarted. However, this
is causing downtime issues on their services which are inside containers.

I know there is the "--skip-interactive" flag for zypper ...

So my question is. Should we mark docker as "interactive" and if so, how
do we do that?

Should we not restart docker when installing? I think you should, but am
I wrong?

Are there other packages with similar problems? mariadb?

thanks in advance

jordi


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Dominique Leuenberger / DimStar
On Fri, 2017-01-20 at 12:02 +0100, Jordi Massaguer Pla wrote:

> Hi packagers,
>
> I am packaging docker and we have this problem that if a user runs
> an 
> unattended update, docker gets updated and so restarted. However,
> this 
> is causing downtime issues on their services which are inside
> containers.
>
> I know there is the "--skip-interactive" flag for zypper ...
>
> So my question is. Should we mark docker as "interactive" and if so,
> how 
> do we do that?
That's not the right approach

> Should we not restart docker when installing? I think you should, but
> am 
> I wrong?

There is a config flag in /etc/susconfig that gives users control over
if they normally want their services restarted on update (which is the
default and we should keep doing for all services that do not 'break
the update sequence')

See /etc/sysconfig/services:
DISABLE_RESTART_ON_UPDATE= and DISABLE_STOP_ON_REMOVAL=

Packages that KNOW that they case very negative effects on restart
(e.g. dbus, the X server, PackageKit) can export this variable in their
post script just before executing the service handling macros. But for
docker, where the issue is 'only' downtime, I'd argue it is up to the
admin to decide if he wants to skip the restart (by setting the
corresponding variables in his system) - and thus take care of actually
enabling the new services, or not (follow the default, restart the
service up update)

> Are there other packages with similar problems? mariadb?

The 'rule' is basically: if the service can be restarted without
impacting the update session, it should be done. Known exceptions so
far are:
* PackageKit: users updating with any PK-related tool would result in
an interrupted system update - hence PK must not be restarted
* xdm: a restart of the display-manager terminates running X-sessions
of a user. This clearly impacts the running update process if not done
in a screen/tmux session (or on a text console)
* dbus-1: just seen last week: restarting dbus makes pretty much all
running dbus-sessions disapear. with X running on systemd-logind
nowadays, this is a bad idea.

Cheers,
Dominique

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Ralf Lang-3
> The 'rule' is basically: if the service can be restarted without
> impacting the update session, it should be done. Known exceptions so
> far are:
> * PackageKit: users updating with any PK-related tool would result in
> an interrupted system update - hence PK must not be restarted
> * xdm: a restart of the display-manager terminates running X-sessions
> of a user. This clearly impacts the running update process if not done
> in a screen/tmux session (or on a text console)
> * dbus-1: just seen last week: restarting dbus makes pretty much all
> running dbus-sessions disapear. with X running on systemd-logind
> nowadays, this is a bad idea.

Running live updates on virtualisation hosts (xen, kvm, docker) without
clever cherrypicking can potentially give you a lot more issues,
especially if you cannot simply reboot the host on short notice.

That shouldn't be an issue with docker on laptops - what could possibly
run there? - but with servers acting as docker hosts, you should
carefully select your live updates and postpone everything else until
you can failover your services to another host (or switch them off for a
while)


--
Ralf Lang
Linux Consultant / Developer
Tel.: +49-170-6381563
Mail: [hidden email]
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Simon Lees-3
In reply to this post by Jordi Massaguer Pla-2


On 01/20/2017 09:32 PM, Jordi Massaguer Pla wrote:

> Hi packagers,
>
> I am packaging docker and we have this problem that if a user runs an
> unattended update, docker gets updated and so restarted. However, this
> is causing downtime issues on their services which are inside containers.
>
> I know there is the "--skip-interactive" flag for zypper ...
>
> So my question is. Should we mark docker as "interactive" and if so, how
> do we do that?
>
> Should we not restart docker when installing? I think you should, but am
> I wrong?
>
> Are there other packages with similar problems? mariadb?
>
> thanks in advance
>
> jordi
>
>
Sorry, I only just noticed this today, we already added handling for
this in the docker spec file [1] late last year. Changing the
"last_migration_version" will cause "DISABLE_RESTART_ON_UPDATE=yes" to
be exported in the postun script which will cause the service not to be
restarted see boo#980555. This means the docker service will only be
restarted if a migration won't be caused.

For dbus we have done it slightly differently [2] by modifying the file
in /etc/sysconfig it allowed us to block the update even though
DISABLE_RESTART_ON_UPDATE is not exported in the older version's postun
section, we also now always export DISABLE_RESTART_ON_UPDATE=yes for
dbus as we never want it to restart during an update so in a few months
time we will probably remove the code messing with config files as
hopefully most of tumbleweed will have migrated away.

Appologies again that it took this long for me to miss and then see the
email, I was off at a conference.

1.
https://build.opensuse.org/package/view_file/Virtualization:containers/docker/docker.spec?expand=1
2.
https://build.opensuse.org/package/view_file/Base:System/dbus-1/dbus-1.spec?expand=1

--

Simon Lees (Simotek)                            http://simotek.net

Emergency Update Team                           keybase.io/simotek
SUSE Linux                           Adelaide Australia, UTC+10:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Simon Lees-3
In reply to this post by Dominique Leuenberger / DimStar


On 01/23/2017 08:47 PM, Dominique Leuenberger / DimStar wrote:

> On Fri, 2017-01-20 at 12:02 +0100, Jordi Massaguer Pla wrote:
>> Hi packagers,
>>
>> I am packaging docker and we have this problem that if a user runs
>> an
>> unattended update, docker gets updated and so restarted. However,
>> this
>> is causing downtime issues on their services which are inside
>> containers.
>>
>
>> Should we not restart docker when installing? I think you should, but
>> am
>> I wrong?
>
>
> Packages that KNOW that they case very negative effects on restart
> (e.g. dbus, the X server, PackageKit) can export this variable in their
> post script just before executing the service handling macros. But for
> docker, where the issue is 'only' downtime, I'd argue it is up to the
> admin to decide if he wants to skip the restart (by setting the
> corresponding variables in his system) - and thus take care of actually
> enabling the new services, or not (follow the default, restart the
> service up update)
>
In http://bugzilla.suse.com/show_bug.cgi?id=980555 it was decided that
as migrations can take many hours in this case the service shouldn't be
automatically restarted. (When docker isn't doing a migration it should be).

--

Simon Lees (Simotek)                            http://simotek.net

Emergency Update Team                           keybase.io/simotek
SUSE Linux                           Adelaide Australia, UTC+10:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Dominique Leuenberger / DimStar
In reply to this post by Ralf Lang-3
On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:

> > The 'rule' is basically: if the service can be restarted without
> > impacting the update session, it should be done. Known exceptions
> > so
> > far are:
> > * PackageKit: users updating with any PK-related tool would result
> > in
> > an interrupted system update - hence PK must not be restarted
> > * xdm: a restart of the display-manager terminates running X-
> > sessions
> > of a user. This clearly impacts the running update process if not
> > done
> > in a screen/tmux session (or on a text console)
> > * dbus-1: just seen last week: restarting dbus makes pretty much
> > all
> > running dbus-sessions disapear. with X running on systemd-logind
> > nowadays, this is a bad idea.
>
> Running live updates on virtualisation hosts (xen, kvm, docker)
> without
> clever cherrypicking can potentially give you a lot more issues,
> especially if you cannot simply reboot the host on short notice.
>
> That shouldn't be an issue with docker on laptops - what could
> possibly
> run there? - but with servers acting as docker hosts, you should
> carefully select your live updates and postpone everything else until
> you can failover your services to another host (or switch them off
> for a
> while)
Sorry, but on such a business critical system I expect the admin to
have disabled automatic restarting of all services in /etc/sysconfig
anyway.

Cheers,
Dominiqe

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Simon Lees-3


On 01/31/2017 06:56 PM, Dominique Leuenberger / DimStar wrote:

> On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:
>>> The 'rule' is basically: if the service can be restarted without
>>> impacting the update session, it should be done. Known exceptions
>>> so
>>> far are:
>>> * PackageKit: users updating with any PK-related tool would result
>>> in
>>> an interrupted system update - hence PK must not be restarted
>>> * xdm: a restart of the display-manager terminates running X-
>>> sessions
>>> of a user. This clearly impacts the running update process if not
>>> done
>>> in a screen/tmux session (or on a text console)
>>> * dbus-1: just seen last week: restarting dbus makes pretty much
>>> all
>>> running dbus-sessions disapear. with X running on systemd-logind
>>> nowadays, this is a bad idea.
>>
>> Running live updates on virtualisation hosts (xen, kvm, docker)
>> without
>> clever cherrypicking can potentially give you a lot more issues,
>> especially if you cannot simply reboot the host on short notice.
>>
>> That shouldn't be an issue with docker on laptops - what could
>> possibly
>> run there? - but with servers acting as docker hosts, you should
>> carefully select your live updates and postpone everything else until
>> you can failover your services to another host (or switch them off
>> for a
>> while)
>
> Sorry, but on such a business critical system I expect the admin to
> have disabled automatic restarting of all services in /etc/sysconfig
> anyway.
>
> Cheers,
> Dominiqe
>
I'm just a messenger, I don't disagree here, but what we expect doesn't
seem to always happen.

--

Simon Lees (Simotek)                            http://simotek.net

Emergency Update Team                           keybase.io/simotek
SUSE Linux                           Adelaide Australia, UTC+10:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Ralf Lang-3
In reply to this post by Dominique Leuenberger / DimStar
On 31.01.2017 09:26, Dominique Leuenberger / DimStar wrote:

> On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:
>>> The 'rule' is basically: if the service can be restarted without
>>> impacting the update session, it should be done. Known exceptions
>>> so
>>> far are:
>>> * PackageKit: users updating with any PK-related tool would result
>>> in
>>> an interrupted system update - hence PK must not be restarted
>>> * xdm: a restart of the display-manager terminates running X-
>>> sessions
>>> of a user. This clearly impacts the running update process if not
>>> done
>>> in a screen/tmux session (or on a text console)
>>> * dbus-1: just seen last week: restarting dbus makes pretty much
>>> all
>>> running dbus-sessions disapear. with X running on systemd-logind
>>> nowadays, this is a bad idea.
>>
>> Running live updates on virtualisation hosts (xen, kvm, docker)
>> without
>> clever cherrypicking can potentially give you a lot more issues,
>> especially if you cannot simply reboot the host on short notice.
>>
>> That shouldn't be an issue with docker on laptops - what could
>> possibly
>> run there? - but with servers acting as docker hosts, you should
>> carefully select your live updates and postpone everything else until
>> you can failover your services to another host (or switch them off
>> for a
>> while)
>
> Sorry, but on such a business critical system I expect the admin to
> have disabled automatic restarting of all services in /etc/sysconfig
> anyway.
I just wanted to advise against admins running random live updates on
virtualisation hosts. Packaging cannot anticipate use cases and admins
should not assume it does. I don't disagree with you.


--
Ralf Lang
Linux Consultant / Developer
Tel.: +49-170-6381563
Mail: [hidden email]
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Lars Vogdt
Am 2. Februar 2017 19:48:37 MEZ, schrieb Ralf Lang <[hidden email]>:
>> Sorry, but on such a business critical system I expect the admin to
>> have disabled automatic restarting of all services in /etc/sysconfig
>> anyway.
>
>I just wanted to advise against admins running random live updates on
>virtualisation hosts. Packaging cannot anticipate use cases and admins
>should not assume it does. I don't disagree with you.

From my personal point as sysadmin maintaining hundreds of machines over years: I trust the packagers that they know best about their services. I never touched the variable you mentioned on any of my systems since I use Linux.

If a services has to be up for the magic 99,999%, you need a HA setup anyway.

In all other cases, I expect:
* security updates might require a restart to have the patched service running instead of the vulnerable one
* a restart might take some time, but I expect the service to be up as before without loss of configured features
* as even productive machines might have a hardware problem, see a power outage or simply need a reboot after kernel/ systemd update, the system as such has to come up with all needed services up and running after a reboot

So from packagers I would except:
* never ever change (runtime) configurations
* if you restart a service, do it right (I saw untested conditions in init scripts that resulted in a failure during restart)
* if needed (and only then) allow a sysadmin to disable an automated restart - but this requires a lot of documentation as this is nothing an admin like me expects as a default setting => and add a big warning sign that disabling an automated restart might have big security implications!

Lars



--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: not restarting the docker service on running zypper update?

Jordi Massaguer Pla-2


On 02/03/2017 07:59 AM, Lars Vogdt wrote:

> Am 2. Februar 2017 19:48:37 MEZ, schrieb Ralf Lang <[hidden email]>:
>>> Sorry, but on such a business critical system I expect the admin to
>>> have disabled automatic restarting of all services in /etc/sysconfig
>>> anyway.
>> I just wanted to advise against admins running random live updates on
>> virtualisation hosts. Packaging cannot anticipate use cases and admins
>> should not assume it does. I don't disagree with you.
>  From my personal point as sysadmin maintaining hundreds of machines over years: I trust the packagers that they know best about their services. I never touched the variable you mentioned on any of my systems since I use Linux.
>
> If a services has to be up for the magic 99,999%, you need a HA setup anyway.
>
> In all other cases, I expect:
> * security updates might require a restart to have the patched service running instead of the vulnerable one
> * a restart might take some time, but I expect the service to be up as before without loss of configured features
> * as even productive machines might have a hardware problem, see a power outage or simply need a reboot after kernel/ systemd update, the system as such has to come up with all needed services up and running after a reboot
>
> So from packagers I would except:
> * never ever change (runtime) configurations
> * if you restart a service, do it right (I saw untested conditions in init scripts that resulted in a failure during restart)

The problem with docker is that restarting docker stops your containers
and they will only come back if, and only if, you had started them with
the option "restart".

https://docs.docker.com/engine/reference/run/#/restart-policies---restart

However, I have the feeling most users do not expect this behavior, they
expect to have the same containers running after an update.

> * if needed (and only then) allow a sysadmin to disable an automated restart - but this requires a lot of documentation as this is nothing an admin like me expects as a default setting => and add a big warning sign that disabling an automated restart might have big security implications!
>
> Lars
>
>
>

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]