archiving a btrfs root

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

archiving a btrfs root

jdd@dodin.org
Hello,

I want a copy of my /

I don't really plan to restore the hole thing with the backup, but I
need to find again the various config files I may have spread in / to
use them in the future, in case of disaster.

rsync seems *not* to be able to do this, at least with the usual -av
options.

as far as I can tel, rsync recurses int he .snapshot folder and fills
the target.

I don't really understand what is said there:

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step

"Initial Bootstrapping"

I don't really need incremental backups, I only care of the last config

the source is / (excluding other file system used by /home or data), the
target any other disk, for example usb HDD

what may I do?

thanks
--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Michael Hamilton
On Thu, 18 Jan 2018, [hidden email] wrote:

> Hello,
>
> I want a copy of my /
>
> I don't really plan to restore the hole thing with the backup, but I
> need to find again the various config files I may have spread in / to
> use them in the future, in case of disaster.
>
> rsync seems *not* to be able to do this, at least with the usual -av
> options.
>
> as far as I can tel, rsync recurses int he .snapshot folder and fills
> the target.
>
> I don't really understand what is said there:
>
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step
>
> "Initial Bootstrapping"
>
> I don't really need incremental backups, I only care of the last config
>
> the source is / (excluding other file system used by /home or data), the
> target any other disk, for example usb HDD
>
> what may I do?
>
> thanks
> --
> http://dodin.org
>

You could use the any of the rsync --exclude or --include options as appropriate.

Depending on what you want to presever you may wish to include some of
the sub-volumes.  Some details of the structure can be found here:

  https://forums.opensuse.org/showthread.php/521277-LEAP-42-2-btrfs-root-filesystem-subvolume-structure

The replies in the discusion thread expand/correct some of the details.

I'm using rsync to preserve a copy of / for the same reasons as yourself.
These are the options I use:

   rsync -ax -HAX --delete  --devices --sparse

But I'm using ext2 so things are a bit simpler for me.  Because it's ext2 I
can usually fully recover from the rsync copy.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 17 January 2018 at 20:04, Michael Hamilton <[hidden email]> wrote:

> On Thu, 18 Jan 2018, [hidden email] wrote:
>> Hello,
>>
>> I want a copy of my /
>>
>> I don't really plan to restore the hole thing with the backup, but I
>> need to find again the various config files I may have spread in / to
>> use them in the future, in case of disaster.
>>
>> rsync seems *not* to be able to do this, at least with the usual -av
>> options.
>>
>> as far as I can tel, rsync recurses int he .snapshot folder and fills
>> the target.
>>
>> I don't really understand what is said there:
>>
>> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step
>>
>> "Initial Bootstrapping"
>>
>> I don't really need incremental backups, I only care of the last config
>>
>> the source is / (excluding other file system used by /home or data), the
>> target any other disk, for example usb HDD
>>
>> what may I do?
>>
>> thanks
>> --
>> http://dodin.org
>>

Why not just use the below?

btrfs send / | btrfs receive /mnt/backup

That will backup the root subvolume (but none of the other subvolumes)
to a different btrfs filesystem or subvolume located at /mnt/backup

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

jdd@dodin.org
Le 18/01/2018 à 10:39, Richard Brown a écrit :

> Why not just use the below?
>
> btrfs send / | btrfs receive /mnt/backup
>
> That will backup the root subvolume (but none of the other subvolumes)
> to a different btrfs filesystem or subvolume located at /mnt/backup
>
right now my backup hdd is ext4, is btrfs on target necessary? (I can
manage this)

thanks

jdd



--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 18 January 2018 at 10:44, [hidden email] <[hidden email]> wrote:

> Le 18/01/2018 à 10:39, Richard Brown a écrit :
>
>> Why not just use the below?
>>
>> btrfs send / | btrfs receive /mnt/backup
>>
>> That will backup the root subvolume (but none of the other subvolumes)
>> to a different btrfs filesystem or subvolume located at /mnt/backup
>>
> right now my backup hdd is ext4, is btrfs on target necessary? (I can manage
> this)
>
> thanks
>
> jdd

For btrfs send/receive to work (which would be easiest), you do, yes.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Andrei Borzenkov
In reply to this post by Richard Brown
18.01.2018 12:39, Richard Brown пишет:

> On 17 January 2018 at 20:04, Michael Hamilton <[hidden email]> wrote:
>> On Thu, 18 Jan 2018, [hidden email] wrote:
>>> Hello,
>>>
>>> I want a copy of my /
>>>
>>> I don't really plan to restore the hole thing with the backup, but I
>>> need to find again the various config files I may have spread in / to
>>> use them in the future, in case of disaster.
>>>
>>> rsync seems *not* to be able to do this, at least with the usual -av
>>> options.
>>>
>>> as far as I can tel, rsync recurses int he .snapshot folder and fills
>>> the target.
>>>
>>> I don't really understand what is said there:
>>>
>>> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step
>>>
>>> "Initial Bootstrapping"
>>>
>>> I don't really need incremental backups, I only care of the last config
>>>
>>> the source is / (excluding other file system used by /home or data), the
>>> target any other disk, for example usb HDD
>>>
>>> what may I do?
>>>
>>> thanks
>>> --
>>> http://dodin.org
>>>
>
> Why not just use the below?
>
> btrfs send / | btrfs receive /mnt/backup
>
> That will backup the root subvolume (but none of the other subvolumes)
> to a different btrfs filesystem or subvolume located at /mnt/backup
>

I'm missing instructions how to recreate bootable openSUSE installation
from /mnt/backup ...

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Carlos E. R.-2
On 01/18/2018 08:14 PM, Andrei Borzenkov wrote:
> 18.01.2018 12:39, Richard Brown пишет:
>> On 17 January 2018 at 20:04, Michael Hamilton <[hidden email]> wrote:
>>> On Thu, 18 Jan 2018, [hidden email] wrote:
>>>> Hello,
>>>>
>>>> I want a copy of my /
>>>>

...

>>
>> Why not just use the below?
>>
>> btrfs send / | btrfs receive /mnt/backup
>>
>> That will backup the root subvolume (but none of the other subvolumes)
>> to a different btrfs filesystem or subvolume located at /mnt/backup
>>
>
> I'm missing instructions how to recreate bootable openSUSE installation
> from /mnt/backup ...

Me too.

I know two possibilities.

One is install fresh from YaST, then overwrite it from backup. I have
never tried and I have my doubts. Maybe overwrite using that
send-receive trick. The assumption is that YaST will create the proper
volumes. Oh, wait, send-receive only copies the root subvolume.

The other is to recreate from dd image. Maybe clonezilla, but it is not
a true imager, as it doesn't image the boot sectors (it reinstalls grub
instead).

--
Cheers / Saludos,

                Carlos E. R.

   (from openSUSE Malachite, Leap 42.3 x86_64 (ssd))

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 18 January 2018 at 20:24, Carlos E. R. <[hidden email]> wrote:

> I know two possibilities.
>
> One is install fresh from YaST, then overwrite it from backup. I have never
> tried and I have my doubts. Maybe overwrite using that send-receive trick.
> The assumption is that YaST will create the proper volumes. Oh, wait,
> send-receive only copies the root subvolume.
>
> The other is to recreate from dd image. Maybe clonezilla, but it is not a
> true imager, as it doesn't image the boot sectors (it reinstalls grub
> instead).
>

or maybe you should consider the 3rd option..you know..the one
extensively documented on the openSUSE wiki

https://en.opensuse.org/SDB:Disaster_Recovery

> On 01/18/2018 08:14 PM, Andrei Borzenkov wrote:
>> I'm missing instructions how to recreate bootable openSUSE installation
>> from /mnt/backup ...
>

jdd's original post clearly stated that he was NOT interested in fully
restoring a system, just wanting to backup the rootfs to have a
reference for things like config files

It was a fair request, simple, and yet unique enough that there is no
clear documentation for that scenario

Andrei your contribution to the thread is unwelcome, unhelpful to the
original poster, and pointless to the wider world in the light of the
facts that full system recovery is not the context of jdd's support
request and that full system recovery options are extensively
documented.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

jdd@dodin.org
Le 18/01/2018 à 21:23, Richard Brown a écrit :

> jdd's original post clearly stated that he was NOT interested in fully
> restoring a system, just wanting to backup the rootfs to have a
> reference for things like config files

exactly. Two things:

* when I have a disaster, I like better rebuild a system from scratch
with the most up to date openSUSE version. During my usual work, I very
often try applications and don't like them or don't like them anymore,
so starting from scratch is a good way of cleaning :-)

* recovering from disaster as documented need very cautious preliminary
work that is aimed to system that have to recover fast. Not my case (I
have more than one computer I can use :-)
>
> It was a fair request, simple, and yet unique enough that there is no
> clear documentation for that scenario

with ext4 it's pretty simple, just rsync the / with -x

but rsync do not cope well with .snapshots (I just discovered this
folder :-(), but I could get a pretty good result copying / with
dolphin, but excluding a lot of things, liken /sys, /proc/ /dev/
/.snapshot, /run, /windows...

thanks
jdd
--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

jdd@dodin.org
In reply to this post by Carlos E. R.-2
Le 18/01/2018 à 20:24, Carlos E. R. a écrit :

>> I'm missing instructions how to recreate bootable openSUSE installation
>> from /mnt/backup ...
>
> Me too.
>
> I know two possibilities.

always star a backup system from the recovery possible. dd (piped by
gzip) is one of the only way I know, provided I have an exact or very
similar hardware, which is never the case...

unuseful for a desktop. Do backup/archive the DATA, not the system. If
config files where really centralized, there would be no problem.

but web sites, databases, config files can be nearly anywhere...

jdd
--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Carlos E. R.-2
On 2018-01-18 22:12, [hidden email] wrote:

> Le 18/01/2018 à 20:24, Carlos E. R. a écrit :
>
>>> I'm missing instructions how to recreate bootable openSUSE installation
>>> from /mnt/backup ...
>>
>> Me too.
>>
>> I know two possibilities.
>
> always star a backup system from the recovery possible. dd (piped by
> gzip) is one of the only way I know, provided I have an exact or very
> similar hardware, which is never the case...
>
> unuseful for a desktop. Do backup/archive the DATA, not the system. If
> config files where really centralized, there would be no problem.
>
> but web sites, databases, config files can be nearly anywhere...
Yes, which is why we need a method to create an empty btrfs filesystem,
create the volumes and subvolumes, and rsync the files back. Lastly,
install the booter.

I can recover from scratch a system only if it does not use btrfs.

--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Carlos E. R.-2
In reply to this post by Richard Brown
On 2018-01-18 21:23, Richard Brown wrote:

> On 18 January 2018 at 20:24, Carlos E. R. <[hidden email]> wrote:
>> I know two possibilities.
>>
>> One is install fresh from YaST, then overwrite it from backup. I have never
>> tried and I have my doubts. Maybe overwrite using that send-receive trick.
>> The assumption is that YaST will create the proper volumes. Oh, wait,
>> send-receive only copies the root subvolume.
>>
>> The other is to recreate from dd image. Maybe clonezilla, but it is not a
>> true imager, as it doesn't image the boot sectors (it reinstalls grub
>> instead).
>>
>
> or maybe you should consider the 3rd option..you know..the one
> extensively documented on the openSUSE wiki
>
> https://en.opensuse.org/SDB:Disaster_Recovery
Oh my...

Sorry, but this is largely hard to digest. It is not one method, it is
several. Maybe if it were broken in several articles, one per method :-?

It talks about rear - it is the first time I see this, so I must
investigate more about it.

It mentions rear-SUSE and autoyast. It also says that they are difficult
tools to use. Some of these many be only for SLES, it is not clear on
the article.


There is a script, actually three or four that I don't know if they are
the same script, they are too big and difficult to analyze, and written
for a particular version of SLES.


The basic information we asked is missing:

How to create on a new empty btrfs partition the exact volume structure
required in each openSUSE release, because they are different on each
release.



>> On 01/18/2018 08:14 PM, Andrei Borzenkov wrote:
>>> I'm missing instructions how to recreate bootable openSUSE installation
>>> from /mnt/backup ...
>>
>
> jdd's original post clearly stated that he was NOT interested in fully
> restoring a system, just wanting to backup the rootfs to have a
> reference for things like config files
>
> It was a fair request, simple, and yet unique enough that there is no
> clear documentation for that scenario
>
> Andrei your contribution to the thread is unwelcome, unhelpful to the
> original poster, and pointless to the wider world in the light of the
> facts that full system recovery is not the context of jdd's support
> request and that full system recovery options are extensively
> documented.
Andrei comments are very welcome by others, though. Threads have a life
of their own, people add more questions to them, and in the end we all
learn, including of course the OP.

Sorry, but we have asked many times for that information.

--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 19 January 2018 at 01:55, Carlos E. R. <[hidden email]> wrote:

>> Andrei your contribution to the thread is unwelcome, unhelpful to the
>> original poster, and pointless to the wider world in the light of the
>> facts that full system recovery is not the context of jdd's support
>> request and that full system recovery options are extensively
>> documented.
>
> Andrei comments are very welcome by others, though. Threads have a life
> of their own, people add more questions to them, and in the end we all
> learn, including of course the OP.
>
> Sorry, but we have asked many times for that information.

Fine, I wrote the following just for you

https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/

Enjoy,

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Michael Hamilton
On Sat, 20 Jan 2018, Richard Brown wrote:

> On 19 January 2018 at 01:55, Carlos E. R. <[hidden email]> wrote:
>
> >> Andrei your contribution to the thread is unwelcome, unhelpful to the
> >> original poster, and pointless to the wider world in the light of the
> >> facts that full system recovery is not the context of jdd's support
> >> request and that full system recovery options are extensively
> >> documented.
> >
> > Andrei comments are very welcome by others, though. Threads have a life
> > of their own, people add more questions to them, and in the end we all
> > learn, including of course the OP.
> >
> > Sorry, but we have asked many times for that information.
>
> Fine, I wrote the following just for you
>
> https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/
>
> Enjoy,
>

Thanks for that, it confirms some past research I'd done.  But I'm still
uncertain about your approach.  When I last looked at this (see previous
email in thread from me) I also saw many other subvolumes:

    @/var/lib/machines
    @/var/lib/mailman
    @/var/lib/mariadb
    @/var/lib/mysql
    @/var/lib/named
    @/var/lib/pgsql
    @/var/log
    @/var/spool

Is creating the above unnecessary?

Are some subvolumes created by software installs?  

I had thought this might be the case and that I would need to use some
commands to discover the list of actual subvolumes?

I thought of creating a script that could replicate the structure of an existing
root, but number of steps and amount of error checking put me off.

Cheers,
Michael


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 20 January 2018 at 04:45, Michael Hamilton <[hidden email]> wrote:
> On Sat, 20 Jan 2018, Richard Brown wrote:
>> On 19 January 2018 at 01:55, Carlos E. R. <[hidden email]> wrote:

>> Fine, I wrote the following just for you
>>
>> https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/
>>
>> Enjoy,
>>
>
> Thanks for that, it confirms some past research I'd done.  But I'm still
> uncertain about your approach.  When I last looked at this (see previous
> email in thread from me) I also saw many other subvolumes:
>
>     @/var/lib/machines
>     @/var/lib/mailman
>     @/var/lib/mariadb
>     @/var/lib/mysql
>     @/var/lib/named
>     @/var/lib/pgsql
>     @/var/log
>     @/var/spool
>
> Is creating the above unnecessary?

Like the article says, my guide is suitable only for Tumbleweed
20180117 (ie the latest snapshot), Leap 15, and SLE 15 or later

These versions contain the new structure which has a single /var subvolume

This is the culmination of months of work changing many packages.
Older SUSE versions had important root filesystem data in the /var
filetree, specifically the rpm database in /var/lib/rpmdb (put there
by rpm) and fillup-templates in /var/adm/fillup-templates (put there
by 207 packages).

These have been progressively relocated in Tumbleweed over the last
months (rpmdb is now in /usr/lib/sysimage/rpm for example) enabling us
to introduce the new /var-as-a-subvolume structure this week

The old structure will still be supported for those people upgrading,
we won't be automatically migrating users data from the old structure
to the new structure,  but I did not want to document my guide for the
old structure given it was a pain in the arse to work with compared to
the shiny new future we have.

That's also why I put the relevant versions in bold in the article -
if anyone follows my guide on earlier versions of our distributions,
they will end up with a broken system when rolling back, because
things like the rpm database will not be snapshotted due to the rpm
package in those versions still using it's old location

> Are some subvolumes created by software installs?

Yes, /var/lib/machines is created by systemd/machined - at time of
writing it is the only example I know of software creating it's own
subvolume, and I just recently installed every-single-package in
Tumbleweed in the hunt for others.

> I had thought this might be the case and that I would need to use some
> commands to discover the list of actual subvolumes?

`btrfs subvolume list /` will show all of the subvolumes in the `/` partition

the `btrfs` command is really useful, I'd recommend reading up on it's
capabilities if you have any similar questions regarding subvolumes or
other btrfs functionality.

> I thought of creating a script that could replicate the structure of an existing
> root, but number of steps and amount of error checking put me off.

Well that's what my guide in essence is doing, for the new default
subvolume layout (and as /var/lib/machines is a subvolume within a
subvolume managed by systemd we don't need to care about it in this
case)

The challenge with any analysis of an existing system would be
identifying whether or not a subvolume was created by the system (ie.
YaST) and therefore certainly needs to be replicated, or whether the
subvolume was created dynamically by the needs of the software (eg.
machined) in which case whether or not it needs to be replicated is
more ambiguous.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Anton Aylward-2
In reply to this post by Michael Hamilton
On 19/01/18 10:45 PM, Michael Hamilton wrote:
>
> Is creating the above unnecessary?

It depends.
BtrFS has somewhat of a 'One file system to rule them all' attitude, but then
again, many people also created those as subdirectories when they only had a
single partition or a  simple 'system + /home'.

myself, i've always been obsessive to a flaw about having mountable file system.
I'm a great beeliver in functional partitioning.

As in "Code is code and data is data".
Taken to some .... well extremes.
oh, i'm rigours about having, for example, ~/bin and ~/lib and /usr/local/bin,
/ur/local/lib.

But I'm also obsessive about putting all the Apache data and functionality under
/srv.  I use RubyOnRails and that, too, is pretty good about code/data divisions.

So, yes, for me /var is a separate mountable file system.
You need all those "parts".  What goes in them is, well 'data', but it is
'variable data' such as the databases -- and of course the spool and the logs :-)

Whether you let BtrFS implement them as subdivisions -- subvolumes -- of the
single volume or implement them as a separate file system is a decision on your
part.  As they say "YMMV".  You really need to make this decision at install
time; trying to 'fix it up' afterwards is a !PAIN!.  BTDT.

--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Richard Brown
On 20 January 2018 at 12:00, Anton Aylward <[hidden email]> wrote:

> So, yes, for me /var is a separate mountable file system.
> You need all those "parts".  What goes in them is, well 'data', but it is
> 'variable data' such as the databases -- and of course the spool and the logs :-)
>
> Whether you let BtrFS implement them as subdivisions -- subvolumes -- of the
> single volume or implement them as a separate file system is a decision on your
> part.  As they say "YMMV".  You really need to make this decision at install
> time; trying to 'fix it up' afterwards is a !PAIN!.  BTDT.

On any openSUSE version before Tumbleweed 20180117 or Leap 15 then
your above advice is not sensible.

Treating /var as a separate mountable filesystem, be it as a
partition, subvolume, or separate disk, will result in a broken system
in the event of using snapper to rollback your system

This is primarily because the rpm database used to record which
packages are installed on your system is located in /var/lib/rpmdb

Therefore rolling back to a point where you had a different set of
packages installed will lead to the rpm database still thinking it had
your pre-rollback packages installed

That's a broken system.

There are also other examples, such as the fillup-templates used to
popular /etc/sysconfig, which are also located in
/var/adm/fillup-templates until recently.

So, sure, NOW, in a freshly installed Tumbleweed and SOON in Leap
users will be able to treat /var as a separate mountable filesystem
without system-breaking consequences, but for the majority of
openSUSE's current users it's essential for the integrity of their
system that they do NOT treat /var as a separate mountable filesystem
when using btrfs, snapshots, and rollback.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Peter Suetterlin-2
In reply to this post by Richard Brown
Richard Brown wrote:

> Like the article says, my guide is suitable only for Tumbleweed
> 20180117 (ie the latest snapshot), Leap 15, and SLE 15 or later
>
> These versions contain the new structure which has a single /var subvolume
>
> This is the culmination of months of work changing many packages.
> Older SUSE versions had important root filesystem data in the /var
> filetree, specifically the rpm database in /var/lib/rpmdb (put there
> by rpm) and fillup-templates in /var/adm/fillup-templates (put there
> by 207 packages).

Which immediately asks for a similar follow-up article describing how to
properly migrate to the new structure, removing unneeded subvolumes in an
(older) system

:D

(You know users, if you give them a finger they want the whole hand/arm...)

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: archiving a btrfs root

Michael Hamilton
In reply to this post by Richard Brown
On Sat, 20 Jan 2018, Richard Brown wrote:

> On 20 January 2018 at 04:45, Michael Hamilton <[hidden email]> wrote:
> > On Sat, 20 Jan 2018, Richard Brown wrote:
> >> On 19 January 2018 at 01:55, Carlos E. R. <[hidden email]> wrote:
>
> >> Fine, I wrote the following just for you
> >>
> >> https://rootco.de/2018-01-19-opensuse-btrfs-subvolumes/
> >>
> >> Enjoy,
> >>
> >
> > Thanks for that, it confirms some past research I'd done.  But I'm still
> > uncertain about your approach.  When I last looked at this (see previous
> > email in thread from me) I also saw many other subvolumes:
> >
> >     @/var/lib/machines
> >     @/var/lib/mailman
> >     @/var/lib/mariadb
> >     @/var/lib/mysql
> >     @/var/lib/named
> >     @/var/lib/pgsql
> >     @/var/log
> >     @/var/spool
> >
> > Is creating the above unnecessary?
>
> Like the article says, my guide is suitable only for Tumbleweed
> 20180117 (ie the latest snapshot), Leap 15, and SLE 15 or later
>
> These versions contain the new structure which has a single /var subvolume
>
> This is the culmination of months of work changing many packages.
> Older SUSE versions had important root filesystem data in the /var
> filetree, specifically the rpm database in /var/lib/rpmdb (put there
> by rpm) and fillup-templates in /var/adm/fillup-templates (put there
> by 207 packages).
>

That's great news.  The messy var structure was one of the
reasons I backed away from btrfs, it's nice that this been given
some attention.

> These have been progressively relocated in Tumbleweed over the last
> months (rpmdb is now in /usr/lib/sysimage/rpm for example) enabling us
> to introduce the new /var-as-a-subvolume structure this week
>
> The old structure will still be supported for those people upgrading,
> we won't be automatically migrating users data from the old structure
> to the new structure,  but I did not want to document my guide for the
> old structure given it was a pain in the arse to work with compared to
> the shiny new future we have.
>
> That's also why I put the relevant versions in bold in the article -
> if anyone follows my guide on earlier versions of our distributions,
> they will end up with a broken system when rolling back, because
> things like the rpm database will not be snapshotted due to the rpm
> package in those versions still using it's old location
>
> > Are some subvolumes created by software installs?
>
> Yes, /var/lib/machines is created by systemd/machined - at time of
> writing it is the only example I know of software creating it's own
> subvolume, and I just recently installed every-single-package in
> Tumbleweed in the hunt for others.
>
> > I had thought this might be the case and that I would need to use some
> > commands to discover the list of actual subvolumes?
>
> `btrfs subvolume list /` will show all of the subvolumes in the `/` partition
>
> the `btrfs` command is really useful, I'd recommend reading up on it's
> capabilities if you have any similar questions regarding subvolumes or
> other btrfs functionality.

Yes I had read up extensively and carried out many experiments in a
VM before creating my own notes which parallel your own.  But mine
were written over a year ago for 42.2 (see link in first message I posted
in this thread).  

I still have concerns that some important details required for recreating the
root structure are not reported by the btrfs command, such as the need to
find all the directories where copy-on-write has been disabled, something
like:

(find `btrfs subvolume list / |  awk '$NF !~ "[@]/[.]snapshots" { sub("@/","/", $NF); print $NF}'` -xdev -type d  | xargs lsattr -d | awk '$1 ~ /C/ {print $NF}') 2> /dev/null

It would be great if the the btrfs command could report this directly.

>
> > I thought of creating a script that could replicate the structure of an existing
> > root, but number of steps and amount of error checking put me off.
>
> Well that's what my guide in essence is doing, for the new default
> subvolume layout (and as /var/lib/machines is a subvolume within a
> subvolume managed by systemd we don't need to care about it in this
> case)

I was looking to create a script that could be trusted to do the job
automatically like the installer is doing.  But I wasn't sure whether
I knew enough to cover all the different options for root and all the
ways these commands might error.   I often manage to screw up
if there are more than 10 steps to read and follow, so I prefer to
automate.

>
> The challenge with any analysis of an existing system would be
> identifying whether or not a subvolume was created by the system (ie.
> YaST) and therefore certainly needs to be replicated, or whether the
> subvolume was created dynamically by the needs of the software (eg.
> machined) in which case whether or not it needs to be replicated is
> more ambiguous.
>

It would be good if a subvolume could somehow be tagged or labelled
with info such as what created it and any other special requirements.

Anyway, it sounds like there are many improvements coming through
that might yet win me over to a btrfs root.   Thanks for making the effort
to provide your notes and for providing some insight into the recent
tumbleweed changes to var.

Cheers,
Michael

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]