difference between two disks

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

difference between two disks

jdd@dodin.org
I have two disks that should be identical. But Dolphin say there is a
200Gb difference in file size, the "to2" (source) having 1.3 To free
when the target have 1.5To free

I just run as root:

rsync -ai --delete-before /run/media/jdd/intenso5to2/
/run/media/jdd/intenso5to/

with no message (no file transfer)

The following text is better read from bottom to top:

http://susepaste.org/98227966

fdisk shows the two disk are the same. In fact bough at the same time,
two 5Tb usb disks for archiving

the du -h show the disk should be the same, and roughly are, but du -d
says the contrary

summary:

3242684936      intenso5to
3242680980      intenso5to2

any idea what happen, and what I can do to find where are the
differences between the two disks?

thanks

jdd



--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Richard Brown
On 18 January 2018 at 17:45, [hidden email] <[hidden email]> wrote:

> I have two disks that should be identical. But Dolphin say there is a 200Gb
> difference in file size, the "to2" (source) having 1.3 To free when the
> target have 1.5To free
>
> I just run as root:
>
> rsync -ai --delete-before /run/media/jdd/intenso5to2/
> /run/media/jdd/intenso5to/
>
> with no message (no file transfer)
>
> The following text is better read from bottom to top:
>
> http://susepaste.org/98227966
>
> fdisk shows the two disk are the same. In fact bough at the same time, two
> 5Tb usb disks for archiving
>
> the du -h show the disk should be the same, and roughly are, but du -d says
> the contrary
>
> summary:
>
> 3242684936      intenso5to
> 3242680980      intenso5to2
>
> any idea what happen, and what I can do to find where are the differences
> between the two disks?
>
> thanks
>
> jdd



which filesystem(s) are on each disk?

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Wol's lists
On 18/01/18 16:57, Richard Brown wrote:

> On 18 January 2018 at 17:45, [hidden email] <[hidden email]> wrote:
>> I have two disks that should be identical. But Dolphin say there is a 200Gb
>> difference in file size, the "to2" (source) having 1.3 To free when the
>> target have 1.5To free
>>
>> I just run as root:
>>
>> rsync -ai --delete-before /run/media/jdd/intenso5to2/
>> /run/media/jdd/intenso5to/
>>
>> with no message (no file transfer)
>>
>> The following text is better read from bottom to top:
>>
>> http://susepaste.org/98227966
>>
>> fdisk shows the two disk are the same. In fact bough at the same time, two
>> 5Tb usb disks for archiving
>>
>> the du -h show the disk should be the same, and roughly are, but du -d says
>> the contrary
>>
>> summary:
>>
>> 3242684936      intenso5to
>> 3242680980      intenso5to2
>>
>> any idea what happen, and what I can do to find where are the differences
>> between the two disks?
>>
>> thanks
>>
>> jdd
>
>
>
> which filesystem(s) are on each disk?
>
The other obvious thing is, are there any hard links on the source disk?
If so, you'll probably have two copies on the target ...

Cheers,
Wol

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
In reply to this post by Richard Brown
Le 18/01/2018 à 17:57, Richard Brown a écrit :

> which filesystem(s) are on each disk?
>
the two of them are ext4

jdd

--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
In reply to this post by Wol's lists
Le 18/01/2018 à 18:21, Wols Lists a écrit :

> The other obvious thing is, are there any hard links on the source disk?
> If so, you'll probably have two copies on the target ...
>

not that I'm aware of.

find . -type f -links +1 -printf '%i %n %p\n'

don't find any

:-(
jdd


--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Nick LeRoy-3
On Thu, Jan 18, 2018 at 12:33 PM, [hidden email] <[hidden email]> wrote:

> Le 18/01/2018 à 18:21, Wols Lists a écrit :
>
>> The other obvious thing is, are there any hard links on the source disk?
>> If so, you'll probably have two copies on the target ...
>>
>
> not that I'm aware of.
>
> find . -type f -links +1 -printf '%i %n %p\n'
>
> don't find any

I'd suggest that there are probably sparse files on the original
filesystem.  By default, rsync does not detect sparse files or handle
them specially.  To change this behavior, specify '-S' or '--sparse'.

-Nick

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Felix Miata-3
In reply to this post by jdd@dodin.org
[hidden email] composed on 2018-01-19 13:23 (UTC+0100):

> the two of them are ext4

Compare outputs of each using tune2fs -l. Do the attributes that are not
supposed to be unique match?
--
"Wisdom is supreme; therefore get wisdom. Whatever else you
get, get wisdom." Proverbs 4:7 (New Living Translation)

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Carlos E. R.-2
In reply to this post by jdd@dodin.org
On 01/18/2018 05:45 PM, [hidden email] wrote:
> I have two disks that should be identical. But Dolphin say there is a
> 200Gb difference in file size, the "to2" (source) having 1.3 To free
> when the target have 1.5To free
>
> I just run as root:
>
> rsync -ai --delete-before /run/media/jdd/intenso5to2/
> /run/media/jdd/intenso5to/

I use:

OPTIONS="--archive --acls --xattrs --hard-links  --stats --humanreadable "

rsync $OPTIONS --one-file-system /SOURCE/ /DEST/

and sometimes I add "--checksum". I should also add "--sparse", I forgot.


-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
ie:
   --recursive --links --perms --times --group --owner --devices --specials

--
Cheers / Saludos,
               
                Carlos E. R.

  (from openSUSE Malachite, Leap 42.3 x86_64 (ssd))

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
Le 18/01/2018 à 19:51, Carlos E. R. a écrit :

> On 01/18/2018 05:45 PM, [hidden email] wrote:
>> I have two disks that should be identical. But Dolphin say there is a
>> 200Gb difference in file size, the "to2" (source) having 1.3 To free
>> when the target have 1.5To free
>>
>> I just run as root:
>>
>> rsync -ai --delete-before /run/media/jdd/intenso5to2/
>> /run/media/jdd/intenso5to/
>
> I use:
(...)

yes, me too, most of the time, but this don't seems to fit btrfs
(specially .snapshot)

there should be some (more) extensive  definition of -x, including
system folders (sys, proc...) and snapshots

like a "only sync a flat file system"

jdd

--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
In reply to this post by Nick LeRoy-3
Le 18/01/2018 à 19:41, Nick LeRoy a écrit :


> I'd suggest that there are probably sparse files on the original
> filesystem.  By default, rsync does not detect sparse files or handle
> them specially.  To change this behavior, specify '-S' or '--sparse'.
>
> -Nick
>
no change with --sparse, same disk.file size

thanks
jdd

--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
In reply to this post by Felix Miata-3
Le 18/01/2018 à 19:48, Felix Miata a écrit :
> [hidden email] composed on 2018-01-19 13:23 (UTC+0100):
>
>> the two of them are ext4
>
> Compare outputs of each using tune2fs -l. Do the attributes that are not
> supposed to be unique match?
>
You found the main problem. The "Reserved block count" was not the same

after setting this reserved count to 0 for the two disks, the size as
reported by Dolphin is the same

however, there is still the same disk usage difference

--  du -s /run/media/jdd/intenso5to*
3242684936      /run/media/jdd/intenso5to
3242680980      /run/media/jdd/intenso5to2

but

  # du --inodes -s /run/media/jdd/intenso5to*
1667104 /run/media/jdd/intenso5to
1667104 /run/media/jdd/intenso5to2

so some files do not have the exact same size??

thanks
jdd

http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Carlos E. R.-2
In reply to this post by jdd@dodin.org
On 2018-01-18 22:16, [hidden email] wrote:

> Le 18/01/2018 à 19:51, Carlos E. R. a écrit :
>> On 01/18/2018 05:45 PM, [hidden email] wrote:
>>> I have two disks that should be identical. But Dolphin say there is a
>>> 200Gb difference in file size, the "to2" (source) having 1.3 To free
>>> when the target have 1.5To free
>>>
>>> I just run as root:
>>>
>>> rsync -ai --delete-before /run/media/jdd/intenso5to2/
>>> /run/media/jdd/intenso5to/
>>
>> I use:
> (...)
>
> yes, me too, most of the time, but this don't seems to fit btrfs
> (specially .snapshot)
>
> there should be some (more) extensive  definition of -x, including
> system folders (sys, proc...) and snapshots
>
> like a "only sync a flat file system"
I would never include /dev, /proc/ or /sys.

Another useful option is "--one-file-system". With that, you can copy
the root filesystem without those directories or /home.

--exclude certainly works.

I don't use any btrfs, but you just have to exclude the snapshots directory.

--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Carlos E. R.-2
In reply to this post by jdd@dodin.org
On 2018-01-18 22:46, [hidden email] wrote:

> Le 18/01/2018 à 19:48, Felix Miata a écrit :
>> [hidden email] composed on 2018-01-19 13:23 (UTC+0100):
>>
>>> the two of them are ext4
>>
>> Compare outputs of each using tune2fs -l. Do the attributes that are not
>> supposed to be unique match?
>>
> You found the main problem. The "Reserved block count" was not the same
>
> after setting this reserved count to 0 for the two disks, the size as
> reported by Dolphin is the same
>
> however, there is still the same disk usage difference
>
> --  du -s /run/media/jdd/intenso5to*
> 3242684936      /run/media/jdd/intenso5to
> 3242680980      /run/media/jdd/intenso5to2
>
> but
>
>  # du --inodes -s /run/media/jdd/intenso5to*
> 1667104 /run/media/jdd/intenso5to
> 1667104 /run/media/jdd/intenso5to2
>
> so some files do not have the exact same size??
In this case, I would use 'mc', one disk in each panel, find directory
sizes, and then browse down the directories till finding those that are
different. Then find which files are different and why.


--
Cheers / Saludos,

                Carlos E. R.
                (from 42.2 x86_64 "Malachite" at Telcontar)


signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

David T-G-2
In reply to this post by jdd@dodin.org
jdd --

...and then [hidden email] said...
%
...
% --  du -s /run/media/jdd/intenso5to*
% 3242684936      /run/media/jdd/intenso5to
% 3242680980      /run/media/jdd/intenso5to2
...
%
% so some files do not have the exact same size??

Try something like

  cd /run/media/jdd/intenso5to && du -ka . | sort -k 2 >/tmp/a
  cd /run/media/jdd/intenso5to2 && du -ka . | sort -k 2 >/tmp/b
  diff -bu /tmp/a /tmp/b

and see how they compare.  Byt sorting by path rather than size, you
should have both listings in the same order.

I actually have an 8G uSD vfat card that shows 1.3G used with no content
that I can find, so this caught my eye :-)  I don't think I have quite
the same issue, though...


HTH & good luck

:-D
--
David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Felix Miata-3
In reply to this post by Carlos E. R.-2
Carlos E. R. composed on 2018-01-18 19:51 (UTC+0100):

> I use:

> OPTIONS="--archive --acls --xattrs --hard-links  --stats --humanreadable "

> rsync $OPTIONS --one-file-system /SOURCE/ /DEST/

> and sometimes I add "--checksum". I should also add "--sparse", I forgot.

> -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
> ie:
>    --recursive --links --perms --times --group --owner --devices --specials

I don't understand why archive excludes what it excludes, so on anything I know
or suspect has hard links, I typically use:

        -rlptgoDHAX

after perusing the man page yet again.
--
"Wisdom is supreme; therefore get wisdom. Whatever else you
get, get wisdom." Proverbs 4:7 (New Living Translation)

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Bernhard Voelker
In reply to this post by jdd@dodin.org
On 01/18/2018 10:46 PM, [hidden email] wrote:
> however, there is still the same disk usage difference
>
> --  du -s /run/media/jdd/intenso5to*
> 3242684936      /run/media/jdd/intenso5to
> 3242680980      /run/media/jdd/intenso5to2

Did you try du's --apparent-size option?

Have a nice day,
Berny

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
Le 19/01/2018 à 09:44, Bernhard Voelker a écrit :
> On 01/18/2018 10:46 PM, [hidden email] wrote:
>> however, there is still the same disk usage difference
>>
>> --  du -s /run/media/jdd/intenso5to*
>> 3242684936      /run/media/jdd/intenso5to
>> 3242680980      /run/media/jdd/intenso5to2
>
> Did you try du's --apparent-size option?
>

not yet

right now I test the -c (checksum) option - long but should fix the
problem (hope :-)

jdd

--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Wol's lists
In reply to this post by David T-G-2
On 19/01/18 03:45, David T-G wrote:
> I actually have an 8G uSD vfat card that shows 1.3G used with no content
> that I can find, so this caught my eye :-)  I don't think I have quite
> the same issue, though...

I guess you've looked for files with the exFAT hidden attribute, seeing
as it's likely to be an MS file system? But have you also looked for
things like .trash (nix dot directories)? I found a load of space lost
like that once.

And I thought emacs had a mode for reading directories? Might be
interesting if you can call that up to see what that thinks is there, as
opposed to what ls or dir tell you.

Cheers,
Wol

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

jdd@dodin.org
In reply to this post by jdd@dodin.org
I couldn't have two identical copies, I give up :-(

just hope than nothing important is missed

thanks
jdd


--
http://dodin.org

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: difference between two disks

Bernhard Voelker
On 01/22/2018 09:00 PM, [hidden email] wrote:
> I couldn't have two identical copies, I give up:-(

well, you could surely achieve this by creating a copy of
the (unmounted!) file system instead of the content therein.
E.g. with dd [1], you need the arguments:

   if=/dev/sdc          source device (disk or partition) with the file system to copy
   of=/dev/sdd          destination device to be (over-)written
   iflag=fullblock      a MUST to avoid effects of 'short reads'
   conv=noerror,sync    continue on read errors, pad those with NULs
   bs=32M               faster than using the default block size
   status=progress      get some periodical status on the terminal

   $ dd \
       if=/dev/sdc \
       of=/dev/sdd \
       iflag=fullblock \
       conv=noerror,sync \
       bs=32M \
       status=progress

[1] https://www.gnu.org/software/coreutils/dd

Have a nice day,
Berny



--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

123