LVM shapshots, udev, symlinks, and reboots -- ugliness

[Marlier, Ian]

Hey all --

Here's the problem that I'm having at the moment:

I created a xen machine that uses an LVM partition as it's root filesystem.
Using LVM's snapshot functionality I created multiple LVM snapshot
partitions of this master, each of which is assigned as the root filesystem
of another xen virtual machine.

All seemed fine, until it was necessary to reboot the physical machine on
which these LVM volumes exist.

Now one of the LVM partitions is a little bit gone.

Specifically:
- `lvs` shows the partition as present, but INACTIVE
- '/dev/dm-X' exists, and by mounting it I can get access to the partition;
fdisk on this device checks out fine.
- '/dev/mapper/volgroup-logicalvolume' exists, but cannot be mounted.  Fdisk
on this device fails.
- '/dev/volgroup/logicalvolume', which is generally a symlink to
/dev/mapper/volgroup-logicalvolume, does not exist at all.

The '/dev/volgroup/logicalvolume' syntax is far more convenient than the
/dev/dm-X syntax, since the value of X can change across reboots (or when
deleting a snapshot and recreating another with the same name, for example).
So I'd really like to have that approach functional...

`dmesg`, /var/log/messages, etc show nothing useful; no errors or anything
else, that I can see...

`uname -a` gives this: Linux xenhost0 2.6.16.13-4-xen #1 SMP Wed May 3
04:53:23 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux




So, questions:
- Any idea what would cause a snapshot to be marked INACTIVE on a reboot?
- Any idea why the INACTIVE snapshot would be mountable, when addressed as
/dev/dm-X?
- Any idea why the /dev/mapper/volgroup-logicalvolume device would no longer
be functional?  And why the /dev/volgroup/logicalvolume symlink wouldn't
have been created?

(And, well...is there a better place to ask these questions?  I've gotten ot
the point that I assume that someone on SLE can answer any question I might
have :-)


--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]

[gregfreemyer]

Ian,

No answer to your question, but a cautionary note that AFAIK lvm
snapshots are still marked experimental in the 2.6 kernel.  I know
I've had far more luck with them in 2.4 than I ever have in 2.6.  I
posted earlier that my SuSE 10.1 snapshot experience was improved, but
I'm still definately getting errors.  ie. failed snapshots.

Greg

On 8/11/06, Ian Marlier <[hidden email]> wrote:

> Hey all --
>
> Here's the problem that I'm having at the moment:
>
> I created a xen machine that uses an LVM partition as it's root filesystem.
> Using LVM's snapshot functionality I created multiple LVM snapshot
> partitions of this master, each of which is assigned as the root filesystem
> of another xen virtual machine.
>
> All seemed fine, until it was necessary to reboot the physical machine on
> which these LVM volumes exist.
>
> Now one of the LVM partitions is a little bit gone.
>
> Specifically:
> - `lvs` shows the partition as present, but INACTIVE
> - '/dev/dm-X' exists, and by mounting it I can get access to the partition;
> fdisk on this device checks out fine.
> - '/dev/mapper/volgroup-logicalvolume' exists, but cannot be mounted.  Fdisk
> on this device fails.
> - '/dev/volgroup/logicalvolume', which is generally a symlink to
> /dev/mapper/volgroup-logicalvolume, does not exist at all.
>
> The '/dev/volgroup/logicalvolume' syntax is far more convenient than the
> /dev/dm-X syntax, since the value of X can change across reboots (or when
> deleting a snapshot and recreating another with the same name, for example).
> So I'd really like to have that approach functional...
>
> `dmesg`, /var/log/messages, etc show nothing useful; no errors or anything
> else, that I can see...
>
> `uname -a` gives this: Linux xenhost0 2.6.16.13-4-xen #1 SMP Wed May 3
> 04:53:23 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
>
> So, questions:
> - Any idea what would cause a snapshot to be marked INACTIVE on a reboot?
> - Any idea why the INACTIVE snapshot would be mountable, when addressed as
> /dev/dm-X?
> - Any idea why the /dev/mapper/volgroup-logicalvolume device would no longer
> be functional?  And why the /dev/volgroup/logicalvolume symlink wouldn't
> have been created?
>
> (And, well...is there a better place to ask these questions?  I've gotten ot
> the point that I assume that someone on SLE can answer any question I might
> have :-)
>
>
> --
> Check the headers for your unsubscription address
> For additional commands send e-mail to [hidden email]
> Also check the archives at http://lists.suse.com
> Please read the FAQs: [hidden email]
>
>
>

--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]

[mourik jan c heupink-2]

> No answer to your question, but a cautionary note that AFAIK lvm
> snapshots are still marked experimental in the 2.6 kernel.  I know
> I've had far more luck with them in 2.4 than I ever have in 2.6.  I
> posted earlier that my SuSE 10.1 snapshot experience was improved, but
> I'm still definately getting errors.  ie. failed snapshots.

I wasn't aware of this. I've had a complete lockups of a system twice
while creating a snapshot.

I take snapshots daily, 99% of them successful, and I didn't understand
these two lockups. (no difference compared to the other days afaik...)

Are you generally staying away from lvm snapshots..? Anyone else seeing
that it's not 100% stable..?

mj

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]

[gregfreemyer]

On 8/11/06, mourik jan <[hidden email]> wrote:

>
> > No answer to your question, but a cautionary note that AFAIK lvm
> > snapshots are still marked experimental in the 2.6 kernel.  I know
> > I've had far more luck with them in 2.4 than I ever have in 2.6.  I
> > posted earlier that my SuSE 10.1 snapshot experience was improved, but
> > I'm still definately getting errors.  ie. failed snapshots.
>
> I wasn't aware of this. I've had a complete lockups of a system twice
> while creating a snapshot.
>
> I take snapshots daily, 99% of them successful, and I didn't understand
> these two lockups. (no difference compared to the other days afaik...)
>
> Are you generally staying away from lvm snapshots..? Anyone else seeing
> that it's not 100% stable..?
>
> mj

mj,

At least for me snapshots are not causing lockups.  I'm just not able
to mount the newly created snapshot.

On the one fileserver I've updated to a 2.6 kernel, I do a snapshot
every night of 3 different lvm volumes..  Looking at the last 30 days
logs I had

volume 1 - 1 failure
volume 2 - 19 failures
volume 3 - 26 failures

You can see why I'm not very comfortable with LVM snapshots yet.
Also, I monitor the redhat lvm mailing list and they get lots of posts
about snapshots.  The answer had typically been that it is still
experimental, so don't use it in production.

Recently they have been saying new patches have been posted to LKML,
so try the latest and greatest vanilla/redhat kernel.  I don't know if
any of those recent fixes have made it into the SuSE 10.1 kernel or
not.  I hope not, because then I can hope that 10.2 will be better.

Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]

[mourik jan c heupink-2]

> At least for me snapshots are not causing lockups.  I'm just not able
> to mount the newly created snapshot.
I've never had THAT problem. They always mount correctly.

> On the one fileserver I've updated to a 2.6 kernel, I do a snapshot
> every night of 3 different lvm volumes..  Looking at the last 30 days
> logs I had
>
> volume 1 - 1 failure
> volume 2 - 19 failures
> volume 3 - 26 failures
that's A LOT. 26 failures of 30 attempts is almost 100%...
Could it be related to the filesystem..? I'm using xfs, and have never
been unable to mount a snaphot.

> Recently they have been saying new patches have been posted to LKML,
> so try the latest and greatest vanilla/redhat kernel.  I don't know if
> any of those recent fixes have made it into the SuSE 10.1 kernel or
> not.  I hope not, because then I can hope that 10.2 will be better.
I noticed some YOU lvm updates the other week. I've installed them, and
hope to see that they help...

Regards,
mj


--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]

[gregfreemyer]

On 8/11/06, mourik jan <[hidden email]> wrote:

>
> > At least for me snapshots are not causing lockups.  I'm just not able
> > to mount the newly created snapshot.
> I've never had THAT problem. They always mount correctly.
>
> > On the one fileserver I've updated to a 2.6 kernel, I do a snapshot
> > every night of 3 different lvm volumes..  Looking at the last 30 days
> > logs I had
> >
> > volume 1 - 1 failure
> > volume 2 - 19 failures
> > volume 3 - 26 failures
> that's A LOT. 26 failures of 30 attempts is almost 100%...
> Could it be related to the filesystem..? I'm using xfs, and have never
> been unable to mount a snaphot.

Same here, xfs.  But when the snapshot fails the error is "no
superblock", so it is something pretty fundamental.  I would
troubleshoot it more if I didn't see so many issues on the lvm mailing
list.

This server does have a small amount of real data on it, but mostly it
is reproducable.  Once this one becomes stable I will think about
upgrading my 2.4.x kernel machines.

> > Recently they have been saying new patches have been posted to LKML,
> > so try the latest and greatest vanilla/redhat kernel.  I don't know if
> > any of those recent fixes have made it into the SuSE 10.1 kernel or
> > not.  I hope not, because then I can hope that 10.2 will be better.
> I noticed some YOU lvm updates the other week. I've installed them, and
> hope to see that they help...
>
> Regards,
> mj

I think I already have those.

Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]