"Operation not Permited" starting docker images after rollback

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

"Operation not Permited" starting docker images after rollback

Michael Aquilina
Hello again,

After a recent rollback I've noticed my docker containers dont want to
start. From what I can tell this is related to btrfs since all the
messages are all along the lines of:

ERROR: for dev-redis  Cannot start service dev-redis: mkdir
/var/lib/docker/btrfs/subvolumes/54d50db5d99cb194405ae0f57a2f2c5b10b1589c8baef38ced2f4ba2725e08d4/data:
operation not permitted

During the last rollback, I had experienced the exact same issue. In
the end I had resorted to deleting everything in /var/lib/docker and
starting over from scratch. I would like to try solve this problem
correclty this time round though.

Has anyone else experienced this issue or have an idea what could be causing it?

--
Michael Aquilina
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Aleksa Sarai
> After a recent rollback I've noticed my docker containers dont want to
> start. From what I can tell this is related to btrfs since all the
> messages are all along the lines of:

If this is a snapper rollback this might be caused by some weird btrfs
issue. I believe we have a subvolume for /var/lib/docker, but if we
don't then a rollback with snapper might have screwed up the layer store.

Worst case scenario you can do something like:

% btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/*
% rm -rf /var/lib/docker

Which will delete everything that Docker had stored and you can start
"fresh".

--
Aleksa Sarai
Software Engineer (Containers)
SUSE Linux GmbH
https://www.cyphar.com/
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Michael Aquilina
Not sure how I can check this, but running this command:

sudo ls /var/lib/docker/btrfs/subvolumes

lists various subvolumes.

I'm pretty sure the docker folder is running on the btrfs partition
rather than one of the other various partitions.

The worst case approach you just described is the solution I had taken
the last time after I had given up. I woud prefer not to nuke my
docker installation everytime I rollback as a lot of my docker images
require a long bootstrap process to setup at work :/

Is there an inherent problem with the way docker has been setup on
btrfs that is causing this issue? Is this something we can fix?

Thanks for the reply!
Mike

On Mon, Sep 18, 2017 at 3:29 AM, Aleksa Sarai <[hidden email]> wrote:

>> After a recent rollback I've noticed my docker containers dont want to
>> start. From what I can tell this is related to btrfs since all the
>> messages are all along the lines of:
>
>
> If this is a snapper rollback this might be caused by some weird btrfs
> issue. I believe we have a subvolume for /var/lib/docker, but if we don't
> then a rollback with snapper might have screwed up the layer store.
>
> Worst case scenario you can do something like:
>
> % btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/*
> % rm -rf /var/lib/docker
>
> Which will delete everything that Docker had stored and you can start
> "fresh".
>
> --
> Aleksa Sarai
> Software Engineer (Containers)
> SUSE Linux GmbH
> https://www.cyphar.com/



--
Michael Aquilina
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Thorsten Kukuk
On Mon, Sep 18, Michael Aquilina wrote:

> Not sure how I can check this, but running this command:
>
> sudo ls /var/lib/docker/btrfs/subvolumes
>
> lists various subvolumes.
>
> I'm pretty sure the docker folder is running on the btrfs partition
> rather than one of the other various partitions.

The default of openSUSE is that /var/lib/docker is on the root filesystem
and not an own subvolume (which would solve your rollback problems)
or partition (which would additional solve performance problems, deadlocks,
and a lot of more problems).

Please put /var/lib/docker on an own partition. Everything else doesn't
make any sense.

  Thorsten

--
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Aleksa Sarai
In reply to this post by Michael Aquilina
> sudo ls /var/lib/docker/btrfs/subvolumes
You'd want to do btrfs subvolume list (or something like that). You
could try btrfs subvolume create /@/var/lib/docker, and then set up the
auto-mounting in fstab, to solve this problem for your install.

> I'm pretty sure the docker folder is running on the btrfs partition
> rather than one of the other various partitions.

Yes, but if it's not in a subvolume (as opposed to living in one of the
snapshot subvolumes) then this problem might happen.

> The worst case approach you just described is the solution I had taken
> the last time after I had given up. I woud prefer not to nuke my
> docker installation everytime I rollback as a lot of my docker images
> require a long bootstrap process to setup at work :/

Yes, this is something that shouldn't be necessary -- I agree. But if
you want to save your images for the migration you could just use
docker-load and docker-save for all of your images. You could even use
this technique to migrate to a different storage driver.

> Is there an inherent problem with the way docker has been setup on
> btrfs that is causing this issue? Is this something we can fix?

On paper, creating a subvolume for /var/lib/docker should solve the
problem. We have this for Kubic but I'm not sure whether we have this
for the main distribution.

Unfortunately it's non-trivial to migrate to this sort of setup (it
might be possible to do that with btrfs-send and btrfs-receive, but I
haven't personally done that sort of migration). You could try with
docker-save and docker-load though.

I've submitted patches to Docker upstream for them to use a subvolume
for /var/lib/docker, but those patches have been rejected on the basis
of "the distribution/administrator should do the right thing here".
Unfortunately due to how our distribution's subvolume policy works (it
needs to be set at install time in yast-installation) changing this is
non-trivial in my experience.

--
Aleksa Sarai
Software Engineer (Containers)
SUSE Linux GmbH
https://www.cyphar.com/
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Thorsten Kukuk
On Mon, Sep 18, Aleksa Sarai wrote:

> On paper, creating a subvolume for /var/lib/docker should solve the problem.
> We have this for Kubic but I'm not sure whether we have this for the main
> distribution.

/var/lib/docker as subvolume is only a solution for being safe for
rollback. But for CaaSP and Kubic we have a seperate partition, a
subvolume alone is not enough to solve all problems.

  Thorsten

--
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: "Operation not Permited" starting docker images after rollback

Johannes Meixner
In reply to this post by Michael Aquilina

Hello,

only a side-note FYI:

On Sep 18 10:18 Michael Aquilina wrote (excerpt):
> Not sure how I can check this, but running this command:
> sudo ls /var/lib/docker/btrfs/subvolumes
> lists various subvolumes.

At least on first glance this looks similar to
https://github.com/rear/rear/issues/1496

Simply put:
When applications "just create" btrfs subvolumes
on a more complicated btrfs subvolume structure
(in particular from within a mounted btrfs subvolume)
then - usually - things go wrong, see also
https://lists.opensuse.org/opensuse-factory/2016-10/msg00397.html


Kind Regards
Johannes Meixner
--
SUSE LINUX GmbH - GF: Felix Imendoerffer, Jane Smithard,
Graham Norton - HRB 21284 (AG Nuernberg)

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]