difference between two disks

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Dave Howorth-3
On Tue, 23 Jan 2018 22:57:18 +0100
Bernhard Voelker <[hidden email]> wrote:

> On 01/23/2018 09:00 PM, [hidden email] wrote:
> > so even non existent and not openned files can take place... any
> > way to reclame the space?  
>
> AFAIK no.  The only workaround I know is to create a new
> directory "d1.new", move all remaining entries (if any) from "d1"
> to "d1.new", then rmdir "d1" and rename "d1.tmp" to "d1".

I take it you meant to say 'copy' (cp) rather than 'move' (mv) since
move will just change the directory pointers.

> > and same with --apparent-size?  
>
> yes, this is /apparent/ size. ;-)
>
> > so is there a way to really compare two mirrored disks to see if the
> > copy is good (ext4)?  
>
> well, you can check with rsync again, telling it to compare the
> content as well (file size and time stamps is not good enough!):
>
>    rsync -HAXaxi --checksum --dry-run SRC/. DST/.
>
> If nothing shows up there, then I'd say the backup looks good.
>
> Have a nice day,
> Berny
>


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Dave Howorth-3
In reply to this post by Carlos E. R.-2
On Tue, 23 Jan 2018 23:13:20 +0100
"Carlos E. R." <[hidden email]> wrote:

> On 2018-01-23 20:56, [hidden email] wrote:
> > Le 23/01/2018 à 20:29, Andrei Borzenkov a écrit :  
> >> 23.01.2018 22:27, [hidden email] пишет:  
> >>> Le 23/01/2018 à 15:26, Andrei Borzenkov a écrit :
> >>>
> >>>  
> >>>> One obvious example - file had non-zero block (and so it
> >>>> consumed real storage) which was later overwritten by zeroes. It
> >>>> will continue to consume real block on source, but on
> >>>> destination it is replaced by hole (no storage consumption).
> >>>>  
> >>> even with checksum control??
> >>>  
> >>
> >> checksums will see zeros in both cases.
> >>  
> > really curious  
>
> Think of it as compression, the files are the same, but compressed.

Not a terribly good analogy, since a compressed file will have a
different checksum.

Where a run of zeros is important, of course people use something more
sophisticated than a simple checksum to guarantee integrity.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Carlos E. R.-2
On 2018-01-24 01:29, Dave Howorth wrote:

> On Tue, 23 Jan 2018 23:13:20 +0100 "Carlos E. R." <> wrote:
>
>> On 2018-01-23 20:56, [hidden email] wrote:
>>> Le 23/01/2018 à 20:29, Andrei Borzenkov a écrit :  
>>>> 23.01.2018 22:27, [hidden email] пишет:  
>>>>> Le 23/01/2018 à 15:26, Andrei Borzenkov a écrit :
>>>>>
>>>>>  
>>>>>> One obvious example - file had non-zero block (and so it
>>>>>> consumed real storage) which was later overwritten by zeroes. It
>>>>>> will continue to consume real block on source, but on
>>>>>> destination it is replaced by hole (no storage consumption).
>>>>>>  
>>>>> even with checksum control??
>>>>>  
>>>>
>>>> checksums will see zeros in both cases.
>>>>  
>>> really curious  
>>
>> Think of it as compression, the files are the same, but compressed.
>
> Not a terribly good analogy, since a compressed file will have a
> different checksum.
Not if it is a compressed filesytem: all the applications see are the
(apparently) expanded files, exactly equal to the originals :-)

--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Andrei Borzenkov
In reply to this post by Dave Howorth-3
24.01.2018 03:25, Dave Howorth пишет:

> On Tue, 23 Jan 2018 22:57:18 +0100
> Bernhard Voelker <[hidden email]> wrote:
>
>> On 01/23/2018 09:00 PM, [hidden email] wrote:
>>> so even non existent and not openned files can take place... any
>>> way to reclame the space?  
>>
>> AFAIK no.  The only workaround I know is to create a new
>> directory "d1.new", move all remaining entries (if any) from "d1"
>> to "d1.new", then rmdir "d1" and rename "d1.tmp" to "d1".
>
> I take it you meant to say 'copy' (cp) rather than 'move' (mv) since
> move will just change the directory pointers.
>

Which is exactly what is necessary here.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: dir size after emptying (was "Re: [opensuse] difference between two disks")

David T-G-2
In reply to this post by Dave Howorth-3
Dave, et al --

...and then Dave Howorth said...
%
% On Tue, 23 Jan 2018 22:57:18 +0100
% Bernhard Voelker <[hidden email]> wrote:
%
% > On 01/23/2018 09:00 PM, [hidden email] wrote:
% > > so even non existent and not openned files can take place... any
% > > way to reclame the space?  
% >
% > AFAIK no.  The only workaround I know is to create a new
% > directory "d1.new", move all remaining entries (if any) from "d1"
% > to "d1.new", then rmdir "d1" and rename "d1.tmp" to "d1".
%
% I take it you meant to say 'copy' (cp) rather than 'move' (mv) since
% move will just change the directory pointers.
[snip]

No, I think he meant move.  The problem is that the original "d1" has
slots for thousands of files but only a few actually in use, and the fix
is simply to create a new directory into which to swing (not copy, same
inodes & extents, etcetc) those files before removing the emptied
original directory.


HTH & HANN

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diffing and inodes (was "Re: [opensuse] difference between two disks")

David T-G-2
In reply to this post by jdd@dodin.org
jdd --

...and then [hidden email] said...
%
% Le 23/01/2018 à 15:56, Carlos E. R. a écrit :
%
...
% >find . -ls
% >
% linux-owxt:/run/media/jdd/intenso5to2 # find . -ls > ../find5.txt
% linux-owxt:/run/media/jdd/intenso5to2 # cd ../intenso4to/
% linux-owxt:/run/media/jdd/intenso4to # find . -ls > ../find4.txt
%
% the size of the to files is not identical, but the size of the file
% (more than 300Mb)
[snip]

This won't work because -ls includes the inode number, which will not be
the same for a given file on two disks.  Here's another case where you
could use that cool printf option to pull out only the pieces you need to
compare :-)


HTH & bonne nuit

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

David T-G-2
In reply to this post by jdd@dodin.org
jdd --

...and then [hidden email] said...
%
...
% so is there a way to really compare two mirrored disks to see if the
% copy is good (ext4)?

I forget if we've discussed this before...  What about good ol'

  diff -r $SRC $DST

to recurse through both trees?  Do you *need* to care if any sparse files
have been expanded upon copy (ie are there a lot of them waiting to blow
up your target disk space?)?


HTH & bonne nuit

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diffing and inodes (was "Re: [opensuse] difference between two disks")

Carlos E. R.-2
In reply to this post by David T-G-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Tuesday, 2018-01-23 at 23:08 -0500, David T-G wrote:

> ...and then [hidden email] said...
> %
> % Le 23/01/2018 à 15:56, Carlos E. R. a écrit :
> %
> ...
> % >find . -ls
> % >
> % linux-owxt:/run/media/jdd/intenso5to2 # find . -ls > ../find5.txt
> % linux-owxt:/run/media/jdd/intenso5to2 # cd ../intenso4to/
> % linux-owxt:/run/media/jdd/intenso4to # find . -ls > ../find4.txt
> %
> % the size of the to files is not identical, but the size of the file
> % (more than 300Mb)
> [snip]
>
> This won't work because -ls includes the inode number, which will not be
> the same for a given file on two disks.  Here's another case where you
> could use that cool printf option to pull out only the pieces you need to
> compare :-)
Inode number is not the inode count, as in ls -l command?

OH, you mean the first number. Crumbs.

cer@Telcontar:/home_aux/cer/Documents/benchmarking> find . -ls
158907909    4 drwxr-xr-x   2 cer      users        4096 Jan 11 11:20 .
158907918    8 -rw-r--r--   1 root     root         4178 Jan 10 20:12 ./4.4.103-18.41-default.log
158907947   12 -rw-r--r--   1 cer      users        9145 Jan 11 10:59 ./Isengard-sql-bench_20180110-antes.txt
158907948   12 -rw-r--r--   1 cer      users        8602 Jan 11 10:59 ./Telcontar-sql-bench_20180110-antes.txt
158907949   20 -rw-r--r--   1 cer      users       19516 Jan 10 23:41 ./Telcontar-iozone_20180110-antes.txt
158907946    4 -rw-r--r--   1 cer      users        2258 Jan 10 19:52 ./Telcontar-interbench_20180110-antes_y_malo.txt
158907950   20 -rw-r--r--   1 cer      users       19506 Jan 11 00:09 ./Isengard-iozone_20180110-antes.txt


Ok, then, adapting another example in the info find page, this:

find -printf "%3n %-8u %-8g %8s %T+ %p\n->%l\0" | cat


(which fails on the '.' entry, by the way)


- --
Cheers,
        Carlos E. R.
        (from openSUSE 42.2 x86_64 "Malachite" at Telcontar)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlpobwcACgkQtTMYHG2NR9V75QCgmCJUQVjGsPUTnxn4A4M+Fe/C
hSMAniJZzurg/WW6+FX/ebZbJpAHyk8P
=gRWe
-----END PGP SIGNATURE-----
Reply | Threaded
Open this post in threaded view
|

Re: dir size after emptying (was "Re: [opensuse] difference between two disks")

Dave Howorth-3
In reply to this post by David T-G-2
On Tue, 23 Jan 2018 23:05:12 -0500
David T-G <[hidden email]> wrote:

> Dave, et al --
>
> ...and then Dave Howorth said...
> %
> % On Tue, 23 Jan 2018 22:57:18 +0100
> % Bernhard Voelker <[hidden email]> wrote:
> %
> % > On 01/23/2018 09:00 PM, [hidden email] wrote:
> % > > so even non existent and not openned files can take place... any
> % > > way to reclame the space?  
> % >
> % > AFAIK no.  The only workaround I know is to create a new
> % > directory "d1.new", move all remaining entries (if any) from "d1"
> % > to "d1.new", then rmdir "d1" and rename "d1.tmp" to "d1".
> %
> % I take it you meant to say 'copy' (cp) rather than 'move' (mv) since
> % move will just change the directory pointers.
> [snip]
>
> No, I think he meant move.  The problem is that the original "d1" has
> slots for thousands of files but only a few actually in use, and the
> fix is simply to create a new directory into which to swing (not
> copy, same inodes & extents, etcetc) those files before removing the
> emptied original directory.

Ah, sorry. I didn't realize the discussion was just about the
directories rather than the files.

> HTH & HANN
>
> :-D


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diffing and inodes

David T-G-2
In reply to this post by Carlos E. R.-2
Carlos & jdd, et al --

...and then Carlos E. R. said...
%
% On Tuesday, 2018-01-23 at 23:08 -0500, David T-G wrote:
%
% >...and then [hidden email] said...
% >%
% >% Le 23/01/2018 à 15:56, Carlos E. R. a écrit :
% >%
% >...
% >% >find . -ls
...
% >This won't work because -ls includes the inode number, which will not be
...
%
% Inode number is not the inode count, as in ls -l command?

Correct; it is not that.


%
% OH, you mean the first number. Crumbs.

Yep :-)


%
% cer@Telcontar:/home_aux/cer/Documents/benchmarking> find . -ls
% 158907909    4 drwxr-xr-x   2 cer      users        4096 Jan 11 11:20 .
% 158907918    8 -rw-r--r--   1 root     root         4178 Jan 10 20:12 ./4.4.103-18.41-default.log
...
%
% Ok, then, adapting another example in the info find page, this:
%
% find -printf "%3n %-8u %-8g %8s %T+ %p\n->%l\0" | cat

Interesting...  Why bother to null-terminate if you aren't sending to
xargs with -0?  Although I do see a newline \n in there, so I really
should try it ... later.


%
% (which fails on the '.' entry, by the way)

Try

  find * .??* -printf ...

to avoid it.  [Hmmm...  That might break on weird characters again.]
Maybe

  find ./ -printf ...

but I doubt it.  Of course, you could always just jump past the first
line, too, but that's cheating :-)


%
%
% - -- Cheers,
%        Carlos E. R.


HTH & HAND

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diffing and inodes

Carlos E. R.-2
On 2018-01-24 12:44, David T-G wrote:
> Carlos & jdd, et al --
>
> ...and then Carlos E. R. said...
> %
> % On Tuesday, 2018-01-23 at 23:08 -0500, David T-G wrote:


> ...
> %
> % Ok, then, adapting another example in the info find page, this:
> %
> % find -printf "%3n %-8u %-8g %8s %T+ %p\n->%l\0" | cat
>
> Interesting...  Why bother to null-terminate if you aren't sending to
> xargs with -0?  Although I do see a newline \n in there, so I really
> should try it ... later.

No idea, I just took an example from the documentation; it should be
correct ™

Sigh.

> %
> % (which fails on the '.' entry, by the way)
>
> Try
>
>   find * .??* -printf ...
>
> to avoid it.  [Hmmm...  That might break on weird characters again.]
> Maybe
>
>   find ./ -printf ...
>
> but I doubt it.  Of course, you could always just jump past the first
> line, too, but that's cheating :-)
>
--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

jdd@dodin.org
In reply to this post by David T-G-2
Le 24/01/2018 à 05:11, David T-G a écrit :
> jdd --

> % so is there a way to really compare two mirrored disks to see if the
> % copy is good (ext4)?
>
> I forget if we've discussed this before...  What about good ol'
>
>    diff -r $SRC $DST
>
> to recurse through both trees?  Do you *need* to care if any sparse files
> have been expanded upon copy (ie are there a lot of them waiting to blow
> up your target disk space?)?
>

the to disks are simply archives made mostly trough Dolphin (copy/paste)
then mirrored from the first disk (5to2) to the second (4To)

diff run after around 24h with 18% processor (there are around 3.5To
data) with little differences, but ones I don't understand.

# rsync -ai --delete /run/media/jdd/intenso5to2/ /run/media/jdd/intenso4to/

to make sure nothing was changed since last mirror (nothing copied this
time)

try to remove empty dirs:

# find /run/media/jdd/intenso5to2/ -type d -empty -delete
# find /run/media/jdd/intenso4to/ -type d -empty -delete

but... this is both not completely harmless, because some empty dirs may
be relevant, anyway this do not remove hidden dirs and trigger a diff error:

diff -r /run/media/jdd/intenso5to2/ /run/media/jdd/intenso4to/ 2> diff-r.txt

gives a non empty result:

255370 25 janv. 00:34 diff-r.txt

but most if not all are about symbolic links that do not point to
anything (due to the copy)

https://www.cjoint.com/doc/18_01/HAzgipL8PRh_Screenshot-20180125-070751.png

I guess diff tried to follow the links?

but also (no such file):

diff:
/run/media/jdd/intenso5to2/archives-avant-2016/archives-2009-180go/sauf-multimedia/web/data/srv/www/htdocs/cubahoy2009:
Aucun fic
hier ou dossier de ce type
diff:
/run/media/jdd/intenso4to/archives-avant-2016/archives-2009-180go/sauf-multimedia/web/data/srv/www/htdocs/cubahoy2009:
Aucun fichier ou dossier de ce type

and there is no file of this name on any disk, not even link, so from
where come the diff message?

so no meaningful difference

thanks
jdd
--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

Carlos E. R.-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Thursday, 2018-01-25 at 07:27 +0100, [hidden email] wrote:

> Le 24/01/2018 à 05:11, David T-G a écrit :
>
>> %  so is there a way to really compare two mirrored disks to see if the
>> %  copy is good (ext4)?
>>
>>  I forget if we've discussed this before...  What about good ol'
>>
>>     diff -r $SRC $DST
>>
>>  to recurse through both trees?  Do you *need* to care if any sparse files
>>  have been expanded upon copy (ie are there a lot of them waiting to blow
>>  up your target disk space?)?
diff run on the disks themselves? I didn't know that was possible.


> the to disks are simply archives made mostly trough Dolphin (copy/paste) then
> mirrored from the first disk (5to2) to the second (4To)
>
> diff run after around 24h with 18% processor (there are around 3.5To data)
> with little differences, but ones I don't understand.

Why not post those differences here?

>
> # rsync -ai --delete /run/media/jdd/intenso5to2/ /run/media/jdd/intenso4to/
>
> to make sure nothing was changed since last mirror (nothing copied this time)
>
> try to remove empty dirs:

I would not do that.

- --
Cheers,
        Carlos E. R.
        (from openSUSE 42.2 x86_64 "Malachite" at Telcontar)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlprIXoACgkQtTMYHG2NR9VIIwCdE5mq6RttBO2brB9q6aCuj+q4
suYAoJLVnKY4RqR+KH7IEjzEHVAqFUyu
=BW0Z
-----END PGP SIGNATURE-----
Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

jdd@dodin.org
Le 26/01/2018 à 13:39, Carlos E. R. a écrit :

> diff run on the disks themselves? I didn't know that was possible.

I didn't either :-), but it works. However, for 3.5Tb data it's very
long (around 12h, not thinked to time it)

> Why not post those differences here?

too large. I posted an abstract on an other post

most (if not all) where broken symbolic links, looks like diff try to
follow the links and, of course, links on a copy are often broken

jdd


--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

Bernhard Voelker
On 01/26/2018 03:23 PM, [hidden email] wrote:
> most (if not all) where broken symbolic links, looks like diff try to follow the links and, of course, links on a copy are often broken

diff --no-dereference ...

Have a nice day,
Berny

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: diff -r (was "Re: [opensuse] difference between two disks")

jdd@dodin.org
Le 26/01/2018 à 17:22, Bernhard Voelker a écrit :
> On 01/26/2018 03:23 PM, [hidden email] wrote:
>> most (if not all) where broken symbolic links, looks like diff try to follow the links and, of course, links on a copy are often broken
>
> diff --no-dereference ...
>
> Have a nice day,
> Berny
>
thanks, I will try an other time, too long to be run :-(

jdd

--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

123