Q: Burning backup CD/DVD incrementally

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Q: Burning backup CD/DVD incrementally

Anton Aylward-2
As regular readers might recall, I have much of my system arranged as 5G
partitions of that I can back up each partition onto a DVD.  I use K3B.

I'm looking at doing selected, perhaps "incremental', backups.
It is easy enough to select , for example, changed files using FIND.
FIND produces a list, but K3B and the tools for making ISO images want files in
a directory.

The best I can think of is to have a dummy directory that has symlinks to the
files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
about the 'tree'.

Can the readership think of an alternative?  Is there a .iso builder that takes
a file list or input stream of file names?




--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Vojtěch Zeisek-2
Dne čtvrtek 11. ledna 2018 16:05:51 CET, Anton Aylward napsal(a):

> As regular readers might recall, I have much of my system arranged as 5G
> partitions of that I can back up each partition onto a DVD.  I use K3B.
>
> I'm looking at doing selected, perhaps "incremental', backups.
> It is easy enough to select , for example, changed files using FIND.
> FIND produces a list, but K3B and the tools for making ISO images want files
> in a directory.
>
> The best I can think of is to have a dummy directory that has symlinks to
> the files generated by FIND.  It seems a bit of a kludge.  I'm certainly
> not happy about the 'tree'.
>
> Can the readership think of an alternative?  Is there a .iso builder that
> takes a file list or input stream of file names?
Two quick solutions came to my mind:
1) If You are running Btrfs, You can burn the snapshots (perhaps packed by tar
to that ~4 GB blocks).
2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) - it
makes incremental backups (tar, zip, gpg) - You can set size of volumes, You
can compress the volumes, if require also encrypt.
In both cases You'll have one directory of files of given size and You can
burn them.

--
Vojtěch Zeisek
https://trapa.cz/

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Patrick Shanahan-2
In reply to this post by Anton Aylward-2
* Anton Aylward <[hidden email]> [01-11-18 10:06]:

> As regular readers might recall, I have much of my system arranged as 5G
> partitions of that I can back up each partition onto a DVD.  I use K3B.
>
> I'm looking at doing selected, perhaps "incremental', backups.
> It is easy enough to select , for example, changed files using FIND.
> FIND produces a list, but K3B and the tools for making ISO images want files in
> a directory.
>
> The best I can think of is to have a dummy directory that has symlinks to the
> files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
> about the 'tree'.
>
> Can the readership think of an alternative?  Is there a .iso builder that takes
> a file list or input stream of file names?

not an iso builder but a space saving backup that provides the directory
structure where one could easily create an iso.  Look at backintime, is in
openSUSE OSS builds.
--
(paka)Patrick Shanahan       Plainfield, Indiana, USA          @ptilopteri
http://en.opensuse.org    openSUSE Community Member    facebook/ptilopteri
Registered Linux User #207535                    @ http://linuxcounter.net
Photos: http://wahoo.no-ip.org/piwigo                    paka @ IRCnet freenode

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

David Haller-4
In reply to this post by Anton Aylward-2
Hello,

On Thu, 11 Jan 2018, Anton Aylward wrote:
>Can the readership think of an alternative? Is there a .iso builder
>that takes a file list or input stream of file names?

The one that is used by basically all apps anyway behind the scenes:
mkisofs.

SYNOPSIS
       mkisofs [ options ] [ -o filename ] pathspec [pathspec ...]
       mkisofs [ options ] [ -o filename ] -find [find expression]

See the manpage for more (esp. 'mkisofs -find -help').

Once you have the iso, burning that can be as simple as

    cdrecord dev=/dev/sr0 foo.iso

HTH,
-dnh

--
printk("IOP: oh my god, they killed the ISM IOP!\n");
        linux-2.6.19/arch/m68k/mac/iop.c

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by Vojtěch Zeisek-2
On 11/01/18 10:12 AM, Vojtěch Zeisek wrote:
>> Can the readership think of an alternative?  Is there a .iso builder that
>> takes a file list or input stream of file names?

> Two quick solutions came to my mind:
> 1) If You are running Btrfs, You can burn the snapshots (perhaps packed by tar
> to that ~4 GB blocks).

I don't use BtrFS.
Let's not have a discussion about that.


> 2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) - it
> makes incremental backups (tar, zip, gpg) - You can set size of volumes, You
> can compress the volumes, if require also encrypt.
> In both cases You'll have one directory of files of given size and You can
> burn them.

Well, I installed duplicity and deja-dup.
I started deja-dup and it stated it was doing an initial scan for the first backup.
Then it told me it was out of space.
There's not much in the way of command line options and I never got the point
where the GUI offered options.
I tried it on a less populated FS, one of my ones under /var that has only 15%
occupancy.  After a short while the machine froze.  I can't think of anything
else that was going on to account for that, I was just watching the scan list;
it stopped as the machine froze.  Neither ctl-c nor any three fingered salute
worked, I had to power cycle.

I also looked at the man page for duplicity as was very put off.  Much to much
to worry about.


For the record:
My full backups of the file systems on the 5G partitions using k3B has the
advantage that they are not TAR'd.  I can mount the DVD and it is just another
file system.  I can read or copy any file without having to any special program.
The idea of doing incremental to CDs, or incremental of a number of the 5G's to
a DVD with the same kind of access is too attractive.

Running FIND to get a list by whatever criteria, date of change is a good enough
one, but  FIND allows for other criteria as well, remember, then generating
symlinks in a dummy directory and pointing K3B at that directory is ..... well
it works, but it takes a bit of hacking to get sensible pathnames under the
dummy directory, and k3b isn't the most reliable tool for following symlinks.
Sometimes that has an unexpected consequence.

For example: in my ~/Document there is a symlink to ~/Downloads
I would rather that was not followed.


--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Patrick Shanahan-2
* Anton Aylward <[hidden email]> [01-11-18 14:55]:
 [...]

> For the record:
> My full backups of the file systems on the 5G partitions using k3B has the
> advantage that they are not TAR'd.  I can mount the DVD and it is just another
> file system.  I can read or copy any file without having to any special program.
> The idea of doing incremental to CDs, or incremental of a number of the 5G's to
> a DVD with the same kind of access is too attractive.
>
> Running FIND to get a list by whatever criteria, date of change is a good enough
> one, but  FIND allows for other criteria as well, remember, then generating
> symlinks in a dummy directory and pointing K3B at that directory is ..... well
> it works, but it takes a bit of hacking to get sensible pathnames under the
> dummy directory, and k3b isn't the most reliable tool for following symlinks.
> Sometimes that has an unexpected consequence.
>
> For example: in my ~/Document there is a symlink to ~/Downloads
> I would rather that was not followed.

you *should* consider backintime

--
(paka)Patrick Shanahan       Plainfield, Indiana, USA          @ptilopteri
http://en.opensuse.org    openSUSE Community Member    facebook/ptilopteri
Registered Linux User #207535                    @ http://linuxcounter.net
Photos: http://wahoo.no-ip.org/piwigo                    paka @ IRCnet freenode

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Carlos E. R.-2
In reply to this post by Anton Aylward-2
On 2018-01-11 16:05, Anton Aylward wrote:

> As regular readers might recall, I have much of my system arranged as 5G
> partitions of that I can back up each partition onto a DVD.  I use K3B.
>
> I'm looking at doing selected, perhaps "incremental', backups.
> It is easy enough to select , for example, changed files using FIND.
> FIND produces a list, but K3B and the tools for making ISO images want files in
> a directory.
>
> The best I can think of is to have a dummy directory that has symlinks to the
> files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
> about the 'tree'.
Rather hardlinks, or the software would have to "follow" the links.


> Can the readership think of an alternative?  Is there a .iso builder that takes
> a file list or input stream of file names?

I asked about that the other day. I want to do backups to Bluray DVDs
(can be 100 GB each). Same as you, not a tar (not reliable, and no
gain), but files directly saved on the ISO. Needs a database that tells
which DVD contains which file and version of it. And of course, should
handle splitting of source across as many output DVDs as needed. And
should be encrypted.


--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Vojtěch Zeisek-2
In reply to this post by Anton Aylward-2
Dne čtvrtek 11. ledna 2018 20:53:52 CET, Anton Aylward napsal(a):

> On 11/01/18 10:12 AM, Vojtěch Zeisek wrote:
> > 2) Use duplicity (or some GUI using it like Deja-Dup, perhaps kbackup) -
> > it
> > makes incremental backups (tar, zip, gpg) - You can set size of volumes,
> > You can compress the volumes, if require also encrypt.
> > In both cases You'll have one directory of files of given size and You can
> > burn them.
>
> Well, I installed duplicity and deja-dup.
> I started deja-dup and it stated it was doing an initial scan for the first
> backup. Then it told me it was out of space.
I never used any GUI over duplicity...

> There's not much in the way of command line options and I never got the
> point where the GUI offered options.
> I tried it on a less populated FS, one of my ones under /var that has only
> 15% occupancy.  After a short while the machine froze.  I can't think of
> anything else that was going on to account for that, I was just watching
> the scan list; it stopped as the machine froze.  Neither ctl-c nor any
> three fingered salute worked, I had to power cycle.

It uses plenty of CPU when preparing the archive volume, but I never
experienced such a freeze. Might be watching htop in separate terminal or so
could show more. It is not expected behavior, although its true, that initial
backup can take long time (later incremental backups are much faster).

> I also looked at the man page for duplicity as was very put off.  Much to
> much to worry about.
--
Vojtěch Zeisek
https://trapa.cz/

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by David Haller-4
On 11/01/18 12:09 PM, David Haller wrote:
> Hello,
>
> On Thu, 11 Jan 2018, Anton Aylward wrote:
>> Can the readership think of an alternative? Is there a .iso builder
>> that takes a file list or input stream of file names?
>
> The one that is used by basically all apps anyway behind the scenes:
> mkisofs.

Yes, I'm aware of this.
I've drilled down on that and other aspects of making ISOs.
Frankly, the option list is downright scary1


--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by Carlos E. R.-2
On 11/01/18 06:30 PM, Carlos E. R. wrote:
>>
>> The best I can think of is to have a dummy directory that has symlinks to the
>> files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
>> about the 'tree'.

> Rather hardlinks, or the software would have to "follow" the links.

Since the dummy directory is on new (temporary, 'throw-away') partition and the
sources are from a variety of partitions, no, it has to be symlinks.


--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

David T-G-2
Anton --

...and then Anton Aylward said...
%
% On 11/01/18 06:30 PM, Carlos E. R. wrote:
% >>
% >> The best I can think of is to have a dummy directory that has symlinks to the
% >> files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
% >> about the 'tree'.
%
% > Rather hardlinks, or the software would have to "follow" the links.
%
% Since the dummy directory is on new (temporary, 'throw-away') partition and the
% sources are from a variety of partitions, no, it has to be symlinks.

Wait ...  Why would the scratch area have to be on a separate partition?
Hard links take [almost] no space, so unless the volume is crazy full
there should be room to create a working dir and throw in a zillion
links.


HTH & HAND

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
On 12/01/18 11:25 AM, David T-G wrote:

> Anton --
>
> ...and then Anton Aylward said...
> %
> % On 11/01/18 06:30 PM, Carlos E. R. wrote:
> % >>
> % >> The best I can think of is to have a dummy directory that has symlinks to the
> % >> files generated by FIND.  It seems a bit of a kludge.  I'm certainly not happy
> % >> about the 'tree'.
> %
> % > Rather hardlinks, or the software would have to "follow" the links.
> %
> % Since the dummy directory is on new (temporary, 'throw-away') partition and the
> % sources are from a variety of partitions, no, it has to be symlinks.
>
> Wait ...  Why would the scratch area have to be on a separate partition?
> Hard links take [almost] no space, so unless the volume is crazy full
> there should be room to create a working dir and throw in a zillion
> links.

Go back and read what I said as 'specification'

In absolute terms, maybe you're right in that it doesn't have to be on a
throwaway partition.  Maybe it could be a directory under /tmp for example, that
was deleted afterwards.

I'm well aware that hard links are just an entry in the directory with an inode
number of an existing file.  But really!  A symlink is a small file with the of
the linked file.  it's really not that much more.  Unless you have FS with
bigblocks.  And, even so, many late model FS use tricks to deal with files of
only a few bytes, perhaps storing the data in the inode space.  Check you MAN pages.

But if you go back you'll also see mention of scatter-gather.
Hard links have to be on the same file system' symlinks can cross file systems.

Yes, I could have a scratch hardlink directory for backup on each FS and then
point K3B at *ALL* of them, but that gets to be a nightmare from the admin &
list generation POV.



--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by Patrick Shanahan-2
On 11/01/18 03:04 PM, Patrick Shanahan wrote:
> you *should* consider backintime

Reading the docco pages it seems that it requires hardlinks ....
See my notes on using find to generate the list across a number of file systems
and why I consider the use of symlinks.

example:

I have a tree of mounted FS under ~/Photography/ByYear
Actually there are side threes for special projects and experimentation.

So I want to backup *all* the sidecar files that have changed in the last month
under that tree.  All in one swoop.
I use FIND.
There are currently 9 such sub-branches on 5 file systems, each in a LVM LE.
(read: 'Partition')



--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

David T-G-2
In reply to this post by Anton Aylward-2
Anton, et al --

...and then Anton Aylward said...
%
% On 12/01/18 11:25 AM, David T-G wrote:
% >
% > ...and then Anton Aylward said...
% > %
% > % Since the dummy directory is on new (temporary, 'throw-away') partition and the
% > % sources are from a variety of partitions, no, it has to be symlinks.
% >
% > Wait ...  Why would the scratch area have to be on a separate partition?
...
%
% Go back and read what I said as 'specification'

I see only four other messages from you in this thread, and 'specification'
is not in any of them, so I don't think I've missed a direct quote.  In
your original email you proposed a dummy dir (tree?) of symlinks but also
said that you weren't too happy with that idea.

I don't mean to challenge you; it's your question and you can ask anything
you wish :-)  But I'm honestly confused, because I guess I've overlooked
something.


%
...
% But if you go back you'll also see mention of scatter-gather.
% Hard links have to be on the same file system' symlinks can cross file systems.

Yeah, but aren't you the guy who has all 5G volumes so that each fits on
a disc, and so doesn't it make sense to write your script tackle a volume
at a time with the local "scratch backup" dir right there?  That makes
the script more general anyway.


%
% Yes, I could have a scratch hardlink directory for backup on each FS and then
% point K3B at *ALL* of them, but that gets to be a nightmare from the admin &
% list generation POV.

OK, and that's fair enough.  And, yes, traversing symlinks will get you
the actual files, so that works.

One nice thing about taking every volume individually is that when one
contains so many changes that it would overflow the incremental backup
past a single DVD you could then fire off a full backup of just that
volume and thus exclude it from the incrementals.  Eventually you'd
probably settle down to a fluid mix of full backups and a catch-all
incremental and just not have to worry about the specifics of what's
going on.  [I'd be sure to set some timeout per volume, too, so that
you are guaranteed a fresh full every so often.]

%
%
%
% --
%          A: Yes.
%      >   Q: Are you sure?
%      >>  A: Because it reverses the logical flow of conversation.
%      >>> Q: Why is top posting frowned upon?
%
%
% --
% To unsubscribe, e-mail: [hidden email]
% To contact the owner, e-mail: [hidden email]
%


:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Carlos E. R.-2
In reply to this post by Anton Aylward-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Friday, 2018-01-12 at 11:57 -0500, Anton Aylward wrote:

...

> But if you go back you'll also see mention of scatter-gather.
> Hard links have to be on the same file system' symlinks can cross file systems.
>
> Yes, I could have a scratch hardlink directory for backup on each FS and then
> point K3B at *ALL* of them, but that gets to be a nightmare from the admin &
> list generation POV.


But most tools by default copy the symlink, not the file pointed at. If
you tell the tool to copy the file pointed at, ie, to follow the symlinks,
it will do so with all symlinks, destroying the symlink structures you may
have on the source.


Years ago I wrote a backup script that would first create a new directory
tree without files, replicating the directory tree that I was going to
backup. Then the files were copied to that tree, but using "mkzftree",
which compressed the files (I'm reading from the script, I tried several
variants). Finally I would create an ISO image using "mkisofs -z ...", ie,
resulting in a compressed DVD, which is transparently readable on Linux.

AFAIK k3b does not support this. No way I know of to create such an ISO in
one go, nor to estimate the size before starting.

- From the command line it is possible to create an ISO taking a list of
source paths, thus scriptable.

Nothing of the above handles incremental backups. It could be done by
doing rsyncs to another hard disk (single huge partition), using a new
directory for each "increment". Each of those directories might be then
copied to separate DVDs - but you need to keep the hard disk.

- --
Cheers,
        Carlos E. R.
        (from openSUSE 42.2 x86_64 "Malachite" at Telcontar)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlpZG3wACgkQtTMYHG2NR9WdrwCfb/ZBSvA9+hdU5cXGrkDgQzmc
NqoAn38K9pv4Agit3fWRBRhUglgs708F
=oJI4
-----END PGP SIGNATURE-----

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Carlos E. R.-2
In reply to this post by Anton Aylward-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Friday, 2018-01-12 at 11:18 -0500, Anton Aylward wrote:

> On 11/01/18 12:09 PM, David Haller wrote:
>> Hello,
>>
>> On Thu, 11 Jan 2018, Anton Aylward wrote:
>>> Can the readership think of an alternative? Is there a .iso builder
>>> that takes a file list or input stream of file names?
>>
>> The one that is used by basically all apps anyway behind the scenes:
>> mkisofs.
>
> Yes, I'm aware of this.
> I've drilled down on that and other aspects of making ISOs.
> Frankly, the option list is downright scary1

I have created ISOs on the CLI years ago, it is not that difficult. I can
help with that part. The dificulty is adjusting the size. I have no idea
how to split and span to several DVDs.


What I envison is going through this list one by one:

<https://en.wikipedia.org/wiki/List_of_backup_software>

Some are proprietary but work on Linux.

- --
Cheers,
        Carlos E. R.
        (from openSUSE 42.2 x86_64 "Malachite" at Telcontar)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlpZH4EACgkQtTMYHG2NR9VbUgCfVzVI8TzDKGV6vV8lmDRg13lB
ZC4AnRl5tES3msrVXj9MxjJ39gIMAGto
=xjSj
-----END PGP SIGNATURE-----

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by Patrick Shanahan-2
On 11/01/18 03:04 PM, Patrick Shanahan wrote:
> you *should* consider backintime

Well I spent some time studying it and it is definitely not what I wantfor this
role.
I can see other backup use cases, in particular for a situation that i'm
discussing and advising a salesman of my acquaintance on.
But not me, here, now.

--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by Carlos E. R.-2
On 12/01/18 03:50 PM, Carlos E. R. wrote:
> I have created ISOs on the CLI years ago, it is not that difficult. I can
> help with that part. The dificulty is adjusting the size. I have no idea
> how to split and span to several DVDs.

As far as I can make out there seems to be a few different approaches to all this.

On the one hand there are the 'snapshot management' types such as dev-dup and
backintime that need to maintain a database and really are a
dis-to-disk(-to-offline) so that the snapshots are easily recoverable

Then there are the ones that don't keep the database, or at best rely one the
label on the DVD, written or electronic, and are probably event drive, such as a
completion of a particular project and hence archiving all its working papers
and reports.

There *may* be some way to logically organise this archiving, for example
'monthly' as the project progresses.  That takes the pressure off the 'spanning
multiple DVDs' issue.

They don't need the database of the individual files in the same way.


Once you do get into spanning multiple DVDs then you get into two issues.
The decision of where to split goes hand-in-hand with the decision of what files
to put on each DVD most efficiently (or optimally) -- the 'packing problem'
which is at the very least NP-hard and quite probably NP-complete
https://en.wikipedia.org/wiki/Bin_packing_problem
https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem
https://en.wikipedia.org/wiki/Packing_problems

The second issue is that you need a global index of what is on each DVD.
Either you have to pre-compute the "bin packing" and ahead of time and record
the index on each DVD as a  'header' that says 'it's here or its on the other
one' or the index has to be external.

I can't say I like any of this.

What we need is a K3B that is super-super smart, since I really don't fancy
trying to all this with shell script.


> What I envison is going through this list one by one:
>
> <https://en.wikipedia.org/wiki/List_of_backup_software>

Well, yes, there is that.
Let me know ....

--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Anton Aylward-2
In reply to this post by David T-G-2
On 12/01/18 12:15 PM, David T-G wrote:

> Anton, et al --
>
> ...and then Anton Aylward said...
> %
> % On 12/01/18 11:25 AM, David T-G wrote:
> % >
> % > ...and then Anton Aylward said...
> % > %
> % > % Since the dummy directory is on new (temporary, 'throw-away') partition and the
> % > % sources are from a variety of partitions, no, it has to be symlinks.
> % >
> % > Wait ...  Why would the scratch area have to be on a separate partition?
> ...
> %
> % Go back and read what I said as 'specification'
>
> I see only four other messages from you in this thread, and 'specification'

Take that as 'description'


> is not in any of them, so I don't think I've missed a direct quote.  In
> your original email you proposed a dummy dir (tree?) of symlinks but also
> said that you weren't too happy with that idea.

You missed out a bit.
You missed out
  -- generating the list according to 'whatever' criteria using FIND
  -- using K3B pointed to the dummy dir with the symlinks
        and telling it to follow the symlinks
                which might have odd side effects if the symlinks cascade
       



> %
> ...
> % But if you go back you'll also see mention of scatter-gather.
> % Hard links have to be on the same file system' symlinks can cross file systems.
>
> Yeah, but aren't you the guy who has all 5G volumes so that each fits on
> a disc, and so doesn't it make sense to write your script tackle a volume
> at a time with the local "scratch backup" dir right there?  That makes
> the script more general anyway.

Are you proposing a system whereby each of the "5G" partitions has its own dummy
dir with symlinks that pertain ONLY to the FS on that partition, generate them
all and point k3B at *all* of those dummy dirs:



        for each mounted file system in $LIST
                create $FSROOT/dummy_dir
                find $FSROOT --criteria | \
                      build sylinks list in $FSROOT/dummy_dir
        done
        k3b $( for each mounted file system in $LIST
                   echo $FSROOT/dummy_dir
               done
             )

That assumes the whole will fit on a single DVD.

But it still leaves me with the issue of ....
Oh, wait, they don't have to by symlinks any more!
They are all links on the same FS :-)



> One nice thing about taking every volume individually is that when one
> contains so many changes that it would overflow the incremental backup
> past a single DVD you could then fire off a full backup of just that
> volume and thus exclude it from the incrementals.  Eventually you'd
> probably settle down to a fluid mix of full backups and a catch-all
> incremental and just not have to worry about the specifics of what's
> going on.  [I'd be sure to set some timeout per volume, too, so that
> you are guaranteed a fresh full every so often.]

The reason for the 5G partitions was so that I could do a full backup of the
whole of the FS on that partition to a single DVD.

If that was all I wanted to do then yes, I could back up the whole thing.

But, as an example case, I have the "ByYear" photograph for more than a decade.
in one month I may alter a scattered set across that decade.
I don't want to have to make 10 full DVD backups if there is just one file
changed.   If this was a BtrFS rather than a ReiserFS or JFS  then I could have
snapper see the single change and there would be the single entry in the
.snapper tree.

The whole point of this exercise if to backup ONLY the changes.
That's where FIND came into it.

Well, OK, with FIND I can use other criteria.
- I could back up only certain sidebar files
- I could archive only the changed or newly produced JPGs

With a bit of sophistication, I suppose, I could backup based on an EXIF field
of the photographs.  Tat will take some experimentation :-P



--
         A: Yes.
     >   Q: Are you sure?
     >>  A: Because it reverses the logical flow of conversation.
     >>> Q: Why is top posting frowned upon?


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Q: Burning backup CD/DVD incrementally

Carlos E. R.-2
In reply to this post by Anton Aylward-2
On 2018-01-13 02:27, Anton Aylward wrote:

> On 12/01/18 03:50 PM, Carlos E. R. wrote:
>> I have created ISOs on the CLI years ago, it is not that difficult. I can
>> help with that part. The dificulty is adjusting the size. I have no idea
>> how to split and span to several DVDs.
>
> As far as I can make out there seems to be a few different approaches to all this.
>
> On the one hand there are the 'snapshot management' types such as dev-dup and
> backintime that need to maintain a database and really are a
> dis-to-disk(-to-offline) so that the snapshots are easily recoverable
Need a big disk for destination. Can't use DVDs.

>
> Then there are the ones that don't keep the database, or at best rely one the
> label on the DVD, written or electronic, and are probably event drive, such as a
> completion of a particular project and hence archiving all its working papers
> and reports.
>
> There *may* be some way to logically organise this archiving, for example
> 'monthly' as the project progresses.  That takes the pressure off the 'spanning
> multiple DVDs' issue.
>
> They don't need the database of the individual files in the same way.
>
>
> Once you do get into spanning multiple DVDs then you get into two issues.
> The decision of where to split goes hand-in-hand with the decision of what files
> to put on each DVD most efficiently (or optimally) -- the 'packing problem'
> which is at the very least NP-hard and quite probably NP-complete
> https://en.wikipedia.org/wiki/Bin_packing_problem
> https://en.wikipedia.org/wiki/Knapsack_problem#Multiple_knapsack_problem
> https://en.wikipedia.org/wiki/Packing_problems
>
> The second issue is that you need a global index of what is on each DVD.
> Either you have to pre-compute the "bin packing" and ahead of time and record
> the index on each DVD as a  'header' that says 'it's here or its on the other
> one' or the index has to be external.
>
> I can't say I like any of this.
>
> What we need is a K3B that is super-super smart, since I really don't fancy
> trying to all this with shell script.
I'm sleepy.

Let me say that the best backup program I have ever used was PCtools
Backup, before the 90's. For MsDOS.

I used it with floppies. It was so fast that I barely had time to write
the label on the floppies. It formatted them on the go. It could verify
them. It wrote the index of files on the last floppy, and on a file on
the hard disk. A file could span more than one floppy. It did
compression on the fly, and used forward error recovery (it could
recover bad sectors).

It did full backup or incremental writing changed files automatically,
taking note of deleted files too. I could define a list of directories
and files to include or exclude.

I miss it. I don't understand how thirty years later there is nothing
similar for DVDs on Linux. Or any other media.

>> What I envision is going through this list one by one:
>>
>> <https://en.wikipedia.org/wiki/List_of_backup_software>
>
> Well, yes, there is that.
> Let me know ....

In time. It is large. I have done nothing tonight.

I can say that the list includes dead projects.

--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
12