LVM issue

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

LVM issue

gregfreemyer
All,

I had an air conditioner go out yesterday and a suse 10.1 based
fileserver got shutdown non-gracefully.

Today, one of my LVM virtual partitions is not working  (Yes, I have a
backup from the previous night, but I hope not to need it.)

A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
/dev/mapper/* that should be there is missing.  ie. 2 are present and
the corresponding filesystems are mounting fine.  The 3rd is missing
and it obviously will not mount.

The missing link is /dev/<my-lvm-domain>/data

I would simply recreate it, but looking at /dev/mapper I'm surprised
to find 2 entries that could be my lvm partition:
/dev/mapper/<my-lvm-domain>-data
/dev/mapper/<my-lvm-domain>-data-real

Is it safe for me to simply try creating a link for -data then
issueing a mount command?

Thanks
Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

Marlier, Ian



On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:

> All,
>
> I had an air conditioner go out yesterday and a suse 10.1 based
> fileserver got shutdown non-gracefully.
>
> Today, one of my LVM virtual partitions is not working  (Yes, I have a
> backup from the previous night, but I hope not to need it.)
>
> A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
> /dev/mapper/* that should be there is missing.  ie. 2 are present and
> the corresponding filesystems are mounting fine.  The 3rd is missing
> and it obviously will not mount.
>
> The missing link is /dev/<my-lvm-domain>/data
>
> I would simply recreate it, but looking at /dev/mapper I'm surprised
> to find 2 entries that could be my lvm partition:
> /dev/mapper/<my-lvm-domain>-data
> /dev/mapper/<my-lvm-domain>-data-real
>
> Is it safe for me to simply try creating a link for -data then
> issueing a mount command?

Same symptoms I'm seeing, with a snapshot!

If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?

What about the /dev/mapper/<my-lv-domain>-data-real?

Does fsck pass on either/both of them, or does it fail to read the
filesystem?


--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

gregfreemyer
On 8/11/06, Ian Marlier <[hidden email]> wrote:

>
>
>
> On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:
>
> > All,
> >
> > I had an air conditioner go out yesterday and a suse 10.1 based
> > fileserver got shutdown non-gracefully.
> >
> > Today, one of my LVM virtual partitions is not working  (Yes, I have a
> > backup from the previous night, but I hope not to need it.)
> >
> > A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
> > /dev/mapper/* that should be there is missing.  ie. 2 are present and
> > the corresponding filesystems are mounting fine.  The 3rd is missing
> > and it obviously will not mount.
> >
> > The missing link is /dev/<my-lvm-domain>/data
> >
> > I would simply recreate it, but looking at /dev/mapper I'm surprised
> > to find 2 entries that could be my lvm partition:
> > /dev/mapper/<my-lvm-domain>-data
> > /dev/mapper/<my-lvm-domain>-data-real
> >
> > Is it safe for me to simply try creating a link for -data then
> > issueing a mount command?
>
> Same symptoms I'm seeing, with a snapshot!
>
> If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
>
> What about the /dev/mapper/<my-lv-domain>-data-real?
>
> Does fsck pass on either/both of them, or does it fail to read the
> filesystem?

Thanks for the idea to try a mount.

I tried a readonly mount of -data and it failed (no superblock).

A readonly mount of -data-real works, so I will recreate the link
pointing to that.

Thanks again,
Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

Marlier, Ian



On 8/11/06 1:01 PM, "Greg Freemyer" <[hidden email]> wrote:

> On 8/11/06, Ian Marlier <[hidden email]> wrote:
>>
>>
>>
>> On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:
>>
>>> All,
>>>
>>> I had an air conditioner go out yesterday and a suse 10.1 based
>>> fileserver got shutdown non-gracefully.
>>>
>>> Today, one of my LVM virtual partitions is not working  (Yes, I have a
>>> backup from the previous night, but I hope not to need it.)
>>>
>>> A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
>>> /dev/mapper/* that should be there is missing.  ie. 2 are present and
>>> the corresponding filesystems are mounting fine.  The 3rd is missing
>>> and it obviously will not mount.
>>>
>>> The missing link is /dev/<my-lvm-domain>/data
>>>
>>> I would simply recreate it, but looking at /dev/mapper I'm surprised
>>> to find 2 entries that could be my lvm partition:
>>> /dev/mapper/<my-lvm-domain>-data
>>> /dev/mapper/<my-lvm-domain>-data-real
>>>
>>> Is it safe for me to simply try creating a link for -data then
>>> issueing a mount command?
>>
>> Same symptoms I'm seeing, with a snapshot!
>>
>> If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
>>
>> What about the /dev/mapper/<my-lv-domain>-data-real?
>>
>> Does fsck pass on either/both of them, or does it fail to read the
>> filesystem?
>
> Thanks for the idea to try a mount.
>
> I tried a readonly mount of -data and it failed (no superblock).
>
> A readonly mount of -data-real works, so I will recreate the link
> pointing to that.

Hmm...glad that worked.

After re-creating the symlink, you might want to try doing a clean reboot of
the machine; I'm not sure, but I think that those links are created
dynamically at boot time so yours may not persist across a reboot of the
machine.  (This also makes it possible, at least, that having a
manually-created symlink in there will break something -- you might want to
have a rescue disk handy as well.)

I've never managed to get a clear sense of whether the dynamic creation is
done by udev, or by something else.  The files in /etc/udev/rules.d don't
refer to LVM specifically, though they do somehow interact with LVM in order
to create the /dev/dm-X devices...suffice it to say that it's something that
I'm trying to figure out, in reference to my own issue posted a little
earlier today :-)

- Ian


--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

Marlier, Ian



On 8/11/06 1:11 PM, "Ian Marlier" <[hidden email]> wrote:

>
>
>
> On 8/11/06 1:01 PM, "Greg Freemyer" <[hidden email]> wrote:
>
>> On 8/11/06, Ian Marlier <[hidden email]> wrote:
>>>
>>>
>>>
>>> On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:
>>>
>>>> All,
>>>>
>>>> I had an air conditioner go out yesterday and a suse 10.1 based
>>>> fileserver got shutdown non-gracefully.
>>>>
>>>> Today, one of my LVM virtual partitions is not working  (Yes, I have a
>>>> backup from the previous night, but I hope not to need it.)
>>>>
>>>> A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
>>>> /dev/mapper/* that should be there is missing.  ie. 2 are present and
>>>> the corresponding filesystems are mounting fine.  The 3rd is missing
>>>> and it obviously will not mount.
>>>>
>>>> The missing link is /dev/<my-lvm-domain>/data
>>>>
>>>> I would simply recreate it, but looking at /dev/mapper I'm surprised
>>>> to find 2 entries that could be my lvm partition:
>>>> /dev/mapper/<my-lvm-domain>-data
>>>> /dev/mapper/<my-lvm-domain>-data-real
>>>>
>>>> Is it safe for me to simply try creating a link for -data then
>>>> issueing a mount command?
>>>
>>> Same symptoms I'm seeing, with a snapshot!
>>>
>>> If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
>>>
>>> What about the /dev/mapper/<my-lv-domain>-data-real?
>>>
>>> Does fsck pass on either/both of them, or does it fail to read the
>>> filesystem?
>>
>> Thanks for the idea to try a mount.
>>
>> I tried a readonly mount of -data and it failed (no superblock).
>>
>> A readonly mount of -data-real works, so I will recreate the link
>> pointing to that.
>
> Hmm...glad that worked.
>
> After re-creating the symlink, you might want to try doing a clean reboot of
> the machine; I'm not sure, but I think that those links are created
> dynamically at boot time so yours may not persist across a reboot of the
> machine.  (This also makes it possible, at least, that having a
> manually-created symlink in there will break something -- you might want to
> have a rescue disk handy as well.)
>
> I've never managed to get a clear sense of whether the dynamic creation is
> done by udev, or by something else.  The files in /etc/udev/rules.d don't
> refer to LVM specifically, though they do somehow interact with LVM in order
> to create the /dev/dm-X devices...suffice it to say that it's something that
> I'm trying to figure out, in reference to my own issue posted a little
> earlier today :-)
>
> - Ian
>

Also, there's this section of /etc/sysconfig/lvm:

     ## Type:        integer
     ## Default:     60
     #
     # Timeout for udev device detection. This is the upper limit which the
     # boot script will wait for udev to finish hotplug event processing.
     # If LVM fails to detect all devices during boot this value should be
     # increased. Setting this to '0' disables waiting for udev.
     #
     LVM_DEVICE_TIMEOUT="60"

Maybe increasing that timeout to, I dunno, 120 or 180 seconds would do the
trick?  (I can't try it myself right at the moment, as one of my coworkers
is doing some stuff on our symptomatic machine)


--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

gregfreemyer
On 8/11/06, Ian Marlier <[hidden email]> wrote:

> On 8/11/06 1:11 PM, "Ian Marlier" <[hidden email]> wrote:
> > On 8/11/06 1:01 PM, "Greg Freemyer" <[hidden email]> wrote:
> >> On 8/11/06, Ian Marlier <[hidden email]> wrote:
> >>> On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:
> >>>> All,
> >>>>
> >>>> I had an air conditioner go out yesterday and a suse 10.1 based
> >>>> fileserver got shutdown non-gracefully.
> >>>>
> >>>> Today, one of my LVM virtual partitions is not working  (Yes, I have a
> >>>> backup from the previous night, but I hope not to need it.)
> >>>>
> >>>> A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
> >>>> /dev/mapper/* that should be there is missing.  ie. 2 are present and
> >>>> the corresponding filesystems are mounting fine.  The 3rd is missing
> >>>> and it obviously will not mount.
> >>>>
> >>>> The missing link is /dev/<my-lvm-domain>/data
> >>>>
> >>>> I would simply recreate it, but looking at /dev/mapper I'm surprised
> >>>> to find 2 entries that could be my lvm partition:
> >>>> /dev/mapper/<my-lvm-domain>-data
> >>>> /dev/mapper/<my-lvm-domain>-data-real
> >>>>
> >>>> Is it safe for me to simply try creating a link for -data then
> >>>> issueing a mount command?
> >>>
> >>> Same symptoms I'm seeing, with a snapshot!
> >>>
> >>> If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
> >>>
> >>> What about the /dev/mapper/<my-lv-domain>-data-real?
> >>>
> >>> Does fsck pass on either/both of them, or does it fail to read the
> >>> filesystem?
> >>
> >> Thanks for the idea to try a mount.
> >>
> >> I tried a readonly mount of -data and it failed (no superblock).
> >>
> >> A readonly mount of -data-real works, so I will recreate the link
> >> pointing to that.
> >
> > Hmm...glad that worked.
> >
> > After re-creating the symlink, you might want to try doing a clean reboot of
> > the machine; I'm not sure, but I think that those links are created
> > dynamically at boot time so yours may not persist across a reboot of the
> > machine.  (This also makes it possible, at least, that having a
> > manually-created symlink in there will break something -- you might want to
> > have a rescue disk handy as well.)
> >
> > I've never managed to get a clear sense of whether the dynamic creation is
> > done by udev, or by something else.  The files in /etc/udev/rules.d don't
> > refer to LVM specifically, though they do somehow interact with LVM in order
> > to create the /dev/dm-X devices...suffice it to say that it's something that
> > I'm trying to figure out, in reference to my own issue posted a little
> > earlier today :-)
> >
> > - Ian
> >
>
> Also, there's this section of /etc/sysconfig/lvm:
>
>      ## Type:        integer
>      ## Default:     60
>      #
>      # Timeout for udev device detection. This is the upper limit which the
>      # boot script will wait for udev to finish hotplug event processing.
>      # If LVM fails to detect all devices during boot this value should be
>      # increased. Setting this to '0' disables waiting for udev.
>      #
>      LVM_DEVICE_TIMEOUT="60"
>
> Maybe increasing that timeout to, I dunno, 120 or 180 seconds would do the
> trick?  (I can't try it myself right at the moment, as one of my coworkers
> is doing some stuff on our symptomatic machine)
>
Ian,

Manually creating the link allows me to mount my volume, so I'm good
from that perspective.

As you suggested rebooting the server causes the manually created link
to disappear, so I still have a major problem.

I did not try the timeout change.  I really doubt it is my issue
because I have 3 volumes on the one lvm domain.  The other 2 are
working fine.

I'm going to repost my question on the lvm user list hosted by redhat
(lvm(device mapper) is a redhat sponsored portion of the kernel).  (I
just unsubscribed a couple weeks ago thinking I had not used it in a
long time.)

But, if your right that it is a udev issue, I'm not sure where to ask
about that.

Thanks
Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: LVM issue

gregfreemyer
Resolved by doing a lvremove of a corrupted LVM snapshot.

On 8/11/06, Greg Freemyer <[hidden email]> wrote:

> On 8/11/06, Ian Marlier <[hidden email]> wrote:
> > On 8/11/06 1:11 PM, "Ian Marlier" <[hidden email]> wrote:
> > > On 8/11/06 1:01 PM, "Greg Freemyer" <[hidden email]> wrote:
> > >> On 8/11/06, Ian Marlier <[hidden email]> wrote:
> > >>> On 8/11/06 12:24 PM, "Greg Freemyer" <[hidden email]> wrote:
> > >>>> All,
> > >>>>
> > >>>> I had an air conditioner go out yesterday and a suse 10.1 based
> > >>>> fileserver got shutdown non-gracefully.
> > >>>>
> > >>>> Today, one of my LVM virtual partitions is not working  (Yes, I have a
> > >>>> backup from the previous night, but I hope not to need it.)
> > >>>>
> > >>>> A quick look at "/dev/<my-lvm-domain>/* shows that one of the links to
> > >>>> /dev/mapper/* that should be there is missing.  ie. 2 are present and
> > >>>> the corresponding filesystems are mounting fine.  The 3rd is missing
> > >>>> and it obviously will not mount.
> > >>>>
> > >>>> The missing link is /dev/<my-lvm-domain>/data
> > >>>>
> > >>>> I would simply recreate it, but looking at /dev/mapper I'm surprised
> > >>>> to find 2 entries that could be my lvm partition:
> > >>>> /dev/mapper/<my-lvm-domain>-data
> > >>>> /dev/mapper/<my-lvm-domain>-data-real
> > >>>>
> > >>>> Is it safe for me to simply try creating a link for -data then
> > >>>> issueing a mount command?
> > >>>
> > >>> Same symptoms I'm seeing, with a snapshot!
> > >>>
> > >>> If you do a mount on /dev/mapper/<my-lv-domain>-data, does that work?
> > >>>
> > >>> What about the /dev/mapper/<my-lv-domain>-data-real?
> > >>>
> > >>> Does fsck pass on either/both of them, or does it fail to read the
> > >>> filesystem?
> > >>
> > >> Thanks for the idea to try a mount.
> > >>
> > >> I tried a readonly mount of -data and it failed (no superblock).
> > >>
> > >> A readonly mount of -data-real works, so I will recreate the link
> > >> pointing to that.
> > >
> > > Hmm...glad that worked.
> > >
> > > After re-creating the symlink, you might want to try doing a clean reboot of
> > > the machine; I'm not sure, but I think that those links are created
> > > dynamically at boot time so yours may not persist across a reboot of the
> > > machine.  (This also makes it possible, at least, that having a
> > > manually-created symlink in there will break something -- you might want to
> > > have a rescue disk handy as well.)
> > >
> > > I've never managed to get a clear sense of whether the dynamic creation is
> > > done by udev, or by something else.  The files in /etc/udev/rules.d don't
> > > refer to LVM specifically, though they do somehow interact with LVM in order
> > > to create the /dev/dm-X devices...suffice it to say that it's something that
> > > I'm trying to figure out, in reference to my own issue posted a little
> > > earlier today :-)
> > >
> > > - Ian
> > >
> >
> > Also, there's this section of /etc/sysconfig/lvm:
> >
> >      ## Type:        integer
> >      ## Default:     60
> >      #
> >      # Timeout for udev device detection. This is the upper limit which the
> >      # boot script will wait for udev to finish hotplug event processing.
> >      # If LVM fails to detect all devices during boot this value should be
> >      # increased. Setting this to '0' disables waiting for udev.
> >      #
> >      LVM_DEVICE_TIMEOUT="60"
> >
> > Maybe increasing that timeout to, I dunno, 120 or 180 seconds would do the
> > trick?  (I can't try it myself right at the moment, as one of my coworkers
> > is doing some stuff on our symptomatic machine)
> >
> Ian,
>
> Manually creating the link allows me to mount my volume, so I'm good
> from that perspective.
>
> As you suggested rebooting the server causes the manually created link
> to disappear, so I still have a major problem.
>
> I did not try the timeout change.  I really doubt it is my issue
> because I have 3 volumes on the one lvm domain.  The other 2 are
> working fine.
>
> I'm going to repost my question on the lvm user list hosted by redhat
> (lvm(device mapper) is a redhat sponsored portion of the kernel).  (I
> just unsubscribed a couple weeks ago thinking I had not used it in a
> long time.)
>
> But, if your right that it is a udev issue, I'm not sure where to ask
> about that.
>
> Thanks
> Greg
> --
> Greg Freemyer
> The Norcross Group
> Forensics for the 21st Century
>


--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

--
Check the headers for your unsubscription address
For additional commands send e-mail to [hidden email]
Also check the archives at http://lists.suse.com
Please read the FAQs: [hidden email]