Non-blocking %check

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Non-blocking %check

todd rme
There has been some work and advice recently about improving the
parallelization of project builds. However, one issue that occurred to
me is that a lot of time can be spent running the %check scriptlet
that shouldn't have any impact on the final RPMs.

My idea was that perhaps the RPM could be used for building packages
that depend on it while the %check scriptlet is running. If the %check
succeeds, then everything continues as normal. If not, then builds
that depend on it are canceled and RPMs that depend on it are
invalidated before being published.

I understand that this is a big change. In addition to needed big
changes in how OSC works, this would also require a fundamental change
in the %check macro.  It would mean that things that happen during the
%check scriptlet would no longer have any impact on the final RPMs.
Personally I think this is a feature rather than a bug, I always
thought it was strange that tests are allowed to change the package,
but it is a big behavior change.

I am not sure this is worth the effort to implement or worth breaking
existing RPM semantics. But if we are having a problem with the builds
not being as parallel as they could be, this might be something at
least to consider.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Bernhard Voelker
On 04/20/2017 05:08 AM, Todd Rme wrote:

> There has been some work and advice recently about improving the
> parallelization of project builds. However, one issue that occurred to
> me is that a lot of time can be spent running the %check scriptlet
> that shouldn't have any impact on the final RPMs.
>
> My idea was that perhaps the RPM could be used for building packages
> that depend on it while the %check scriptlet is running. If the %check
> succeeds, then everything continues as normal. If not, then builds
> that depend on it are canceled and RPMs that depend on it are
> invalidated before being published.
>
> I understand that this is a big change. In addition to needed big
> changes in how OSC works, this would also require a fundamental change
> in the %check macro.  It would mean that things that happen during the
> %check scriptlet would no longer have any impact on the final RPMs.
> Personally I think this is a feature rather than a bug, I always
> thought it was strange that tests are allowed to change the package,
> but it is a big behavior change.
>
> I am not sure this is worth the effort to implement or worth breaking
> existing RPM semantics. But if we are having a problem with the builds
> not being as parallel as they could be, this might be something at
> least to consider.

I'm not sure to which discussion you refer to (as you didn't point
to it).  I can only tell how it is done for the coreutils package [1]
where the full check (`make check-very-expensive`) may run between
~15min and ~1.5h depending on the architecture.  Without the checks,
the package is built in ~3-15min.
There's a 2nd spec file 'coreutils-testsuite.spec' which is generated
from the regular 'coreutils.spec', and the %check section is only used
by that secondary package:

  %check
  %if "%{name}" == "coreutils-testsuite"
    ...
    # Avoid parallel make, because otherwise some timeout based tests like
    # rm/ext3-perf may fail due to high CPU or IO load.
    make check-very-expensive \
      && ...
  %endif

Does this idea go into the direction you're thinking?

[1]
https://build.opensuse.org/package/view_file/Base:System/coreutils/coreutils.spec?expand=1

Have a nice day,
Berny
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Ruediger Meier
On Thursday 20 April 2017, Bernhard Voelker wrote:

> On 04/20/2017 05:08 AM, Todd Rme wrote:
> > There has been some work and advice recently about improving the
> > parallelization of project builds. However, one issue that occurred
> > to me is that a lot of time can be spent running the %check
> > scriptlet that shouldn't have any impact on the final RPMs.
> >
> > My idea was that perhaps the RPM could be used for building
> > packages that depend on it while the %check scriptlet is running.
> > If the %check succeeds, then everything continues as normal. If
> > not, then builds that depend on it are canceled and RPMs that
> > depend on it are invalidated before being published.
> >
> > I understand that this is a big change. In addition to needed big
> > changes in how OSC works, this would also require a fundamental
> > change in the %check macro.  It would mean that things that happen
> > during the %check scriptlet would no longer have any impact on the
> > final RPMs. Personally I think this is a feature rather than a bug,
> > I always thought it was strange that tests are allowed to change
> > the package, but it is a big behavior change.
> >
> > I am not sure this is worth the effort to implement or worth
> > breaking existing RPM semantics. But if we are having a problem
> > with the builds not being as parallel as they could be, this might
> > be something at least to consider.
>
> I'm not sure to which discussion you refer to (as you didn't point
> to it).  I can only tell how it is done for the coreutils package [1]
> where the full check (`make check-very-expensive`) may run between
> ~15min and ~1.5h depending on the architecture.  Without the checks,
> the package is built in ~3-15min.
> There's a 2nd spec file 'coreutils-testsuite.spec' which is generated
> from the regular 'coreutils.spec', and the %check section is only
> used by that secondary package:
>
>   %check
>   %if "%{name}" == "coreutils-testsuite"
>     ...
>     # Avoid parallel make, because otherwise some timeout based tests
> like # rm/ext3-perf may fail due to high CPU or IO load.
>     make check-very-expensive \
>       && ...
>   %endif
>
> Does this idea go into the direction you're thinking?

This is a nice idea but it has a few drawbacks:
 1.A bad coreutils-testsuite does not seem to prevent automatically
   that the coreutils package would be released to the users.
 2.coreutils-testsuite is not testing the *same* binaries which we
   install finally in the distro. Bugs or inconsistencies in OSB or
   rpm could cause that "coreutils" builds broken binaries
   but "coreutils-testsuite" looks ok.

What I would really like is that project's like coreutils would have
a "make check*" target which runs tests against the installed
coreutils. Similar to some of our python packages, users could even
test their final installation, like

    python -c "import numpy; numpy.test()"


cu,
Rudi
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Bernhard Voelker
On 04/20/2017 01:30 PM, Ruediger Meier wrote:
> This is a nice idea but it has a few drawbacks:
>  1.A bad coreutils-testsuite does not seem to prevent automatically
>    that the coreutils package would be released to the users.

there is still the package maintainer who is repsonsible for forwarding
packages ... so I don't see this danger.

>  2.coreutils-testsuite is not testing the *same* binaries which we
>    install finally in the distro. Bugs or inconsistencies in OSB or
>    rpm could cause that "coreutils" builds broken binaries
>    but "coreutils-testsuite" looks ok.

That's a valid point though ...

> What I would really like is that project's like coreutils would have
> a "make check*" target which runs tests against the installed
> coreutils.

... it has been like that for quite a while now.  I don't know what
was the motivation behind the split, but I can imagine that this speeded
up the build cycles on OBS dramatically.

Thus said, I'm not against removing 'coreutils-testsuite'.  Maybe it'd
be an improvement if only the smaller (faster) 'make check' is run for
the coreutils package while it would run the slower 'make check-very-expensive'
tests in the coreutils-testsuite case.

Have a nice day,
Berny
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Ruediger Meier
On Thursday 20 April 2017, Bernhard Voelker wrote:
> On 04/20/2017 01:30 PM, Ruediger Meier wrote:
> > This is a nice idea but it has a few drawbacks:
> >  1.A bad coreutils-testsuite does not seem to prevent automatically
> >    that the coreutils package would be released to the users.
>
> there is still the package maintainer who is repsonsible for
> forwarding packages ... so I don't see this danger.

> >  2.coreutils-testsuite is not testing the *same* binaries which we
> >    install finally in the distro. Bugs or inconsistencies in OSB or
> >    rpm could cause that "coreutils" builds broken binaries
> >    but "coreutils-testsuite" looks ok.
>
> That's a valid point though ...
>
> > What I would really like is that project's like coreutils would
> > have a "make check*" target which runs tests against the installed
> > coreutils.
>
> ... it has been like that for quite a while now.  I don't know what
> was the motivation behind the split, but I can imagine that this
> speeded up the build cycles on OBS dramatically.

Yes, the non-testsuite package has less "BuildRequires" to
prevent "unnecessary" rebuilds. But different "BuildRequires" even
increases the probability that the installed and tested binaries could
differ.

IHMO our OBS bootstrapping way to rebuild again and again until
everything converges to reproducable packages is bad especially when we
add such "build cycle optimizations". It makes spec files and tests
very ugly, complicated or even unsafe (e.g. util-linux). It's also a
pain to review submit requests for such packages having duplicated spec
files and changelogs.

> Thus said, I'm not against removing 'coreutils-testsuite'.  Maybe
> it'd be an improvement if only the smaller (faster) 'make check' is
> run for the coreutils package while it would run the slower 'make
> check-very-expensive' tests in the coreutils-testsuite case.
>
> Have a nice day,
> Berny


cu,
Rudi
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Adrian Schröter

JFYI, the idea to handle build results and QA results seperate
exists since quite some time. Means being able to switch in repository
or request configurations to accept or not accept QA failures independend
of source changes.

However, it is unlikely that we work on that soon, sorry.

Nevertheless, I think the seperate -testsuite packages are a proper
workaround for many cases. You can still see the results, and still get
build results soon.

On Donnerstag, 20. April 2017, 15:40:29 CEST wrote Ruediger Meier:

> On Thursday 20 April 2017, Bernhard Voelker wrote:
> > On 04/20/2017 01:30 PM, Ruediger Meier wrote:
> > > This is a nice idea but it has a few drawbacks:
> > >  1.A bad coreutils-testsuite does not seem to prevent automatically
> > >    that the coreutils package would be released to the users.
> >
> > there is still the package maintainer who is repsonsible for
> > forwarding packages ... so I don't see this danger.
>
> > >  2.coreutils-testsuite is not testing the *same* binaries which we
> > >    install finally in the distro. Bugs or inconsistencies in OSB or
> > >    rpm could cause that "coreutils" builds broken binaries
> > >    but "coreutils-testsuite" looks ok.
> >
> > That's a valid point though ...
> >
> > > What I would really like is that project's like coreutils would
> > > have a "make check*" target which runs tests against the installed
> > > coreutils.
> >
> > ... it has been like that for quite a while now.  I don't know what
> > was the motivation behind the split, but I can imagine that this
> > speeded up the build cycles on OBS dramatically.
>
> Yes, the non-testsuite package has less "BuildRequires" to
> prevent "unnecessary" rebuilds. But different "BuildRequires" even
> increases the probability that the installed and tested binaries could
> differ.
>
> IHMO our OBS bootstrapping way to rebuild again and again until
> everything converges to reproducable packages is bad especially when we
> add such "build cycle optimizations". It makes spec files and tests
> very ugly, complicated or even unsafe (e.g. util-linux). It's also a
> pain to review submit requests for such packages having duplicated spec
> files and changelogs.
>
> > Thus said, I'm not against removing 'coreutils-testsuite'.  Maybe
> > it'd be an improvement if only the smaller (faster) 'make check' is
> > run for the coreutils package while it would run the slower 'make
> > check-very-expensive' tests in the coreutils-testsuite case.
> >
> > Have a nice day,
> > Berny
>
>
> cu,
> Rudi
>


--

Adrian Schroeter
email: [hidden email]

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
 
Maxfeldstraße 5                        
90409 Nürnberg
Germany


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Ruediger Meier
On Thursday 20 April 2017, Adrian Schröter wrote:

> JFYI, the idea to handle build results and QA results seperate
> exists since quite some time. Means being able to switch in
> repository or request configurations to accept or not accept QA
> failures independend of source changes.
>
> However, it is unlikely that we work on that soon, sorry.
>
> Nevertheless, I think the seperate -testsuite packages are a proper
> workaround for many cases. You can still see the results, and still
> get build results soon.

No, you don't see *the* results. You can see *some* results of different
binaries which were built using a different build script (spec file),
different BuildRequires and probably on a different build host.

To fix that we could add a subpackage providing the whole original build
directory and install this later (at QA time) to run the missing check
only.

cu,
Rudi
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Bernhard Voelker
On 04/20/2017 04:18 PM, Ruediger Meier wrote:
> No, you don't see *the* results. You can see *some* results of different
> binaries which were built using a different build script (spec file),
> different BuildRequires and probably on a different build host.

The additional BuildRequires for the coreutils-testsuite are only used
for %check, but don't lead to a different build result:

  %if "%{name}" == "coreutils-testsuite"
  BuildRequires:  acl
  BuildRequires:  gdb
  BuildRequires:  perl-Expect
  BuildRequires:  python-pyinotify
  BuildRequires:  strace
  BuildRequires:  timezone
  %ifarch %ix86 x86_64 ppc ppc64 s390x armv7l armv7hl
  BuildRequires:  valgrind
  %endif
  %endif

All the other packages - which are used for building - are identical.

There might be still some risk, but I've never seen any fallout from
this (which should be theoretically be 50% of such cases, i.e., where
the binaries in coreutils are good but those in the testsuite are not).

I think it's more risky that sometimes you have to mask a certain test
which does simply not work on OBS; and even others are just SKIPped
because the prerequisites on OBS are not met.

And finally, I'd rather like to be able to run some tests as root on OBS
to get more test coverage (yes, there are some special tests written to
be run as root), but this is a different story.

After all, I'm fine with the current splitting.

Have a nice day,
Berny
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Michael Matz
In reply to this post by Ruediger Meier
Hi,

On Thu, 20 Apr 2017, Ruediger Meier wrote:

> No, you don't see *the* results. You can see *some* results of different
> binaries which were built using a different build script (spec file),
> different BuildRequires and probably on a different build host.

While your arguments are valid and true in the fullest generic sense, in
the particular case we're talking about here (and in fact for most such
cases) it's moot.  The binaries _are_ the same, so the tests are valid.

And while it is also true that it would be even nicer if more packages
would support installed testing, that's something for upstream to solve
usually.  (Sometimes you can get away with running make check with the
appropriate setting of various makefile variables).  But it would also be
nicer if I had a pink zombie pony; alas it is not so.

So, for the time being, and with the realities we're working in, and
considering that some packages are in fact in the critical path and hence
create a _real_ problem when they build for a long time, the %-testsuite
extra .spec files are the sensible work around.  The other realistic
work-around would be to not run the testsuite at all, and that would be
worse.

> To fix that we could add a subpackage providing the whole original build
> directory and install this later (at QA time) to run the missing check
> only.

True, that would work as well.  It would also create many new extremely
large binary packages nobody except automatic testers would use.  If it
actually solves any problem we in reality have is doubtful.


Ciao,
Michael.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Ruediger Meier
In reply to this post by Bernhard Voelker
On Thursday 20 April 2017, Bernhard Voelker wrote:
> On 04/20/2017 04:18 PM, Ruediger Meier wrote:
> > No, you don't see *the* results. You can see *some* results of
> > different binaries which were built using a different build script
> > (spec file), different BuildRequires and probably on a different
> > build host.
>
> The additional BuildRequires for the coreutils-testsuite are only
> used for %check, but don't lead to a different build result:

I believe you that this is true. But OBS is designed to "not believe" us
human maintainers. Normally it rebuilds indomitably whenever a
dependency changes. The OBS design is even somehow obsessed about
reproducible builds. But why not in this important case where we want
to check *the* *built* binaries?

>   %if "%{name}" == "coreutils-testsuite"
>   BuildRequires:  acl
>   BuildRequires:  gdb

Just an example. What if one day gdb would bring us another
whatever-devel package and ./configure would be happy to use it?

>   BuildRequires:  perl-Expect
>   BuildRequires:  python-pyinotify
>   BuildRequires:  strace
>   BuildRequires:  timezone
>   %ifarch %ix86 x86_64 ppc ppc64 s390x armv7l armv7hl
>   BuildRequires:  valgrind
>   %endif
>   %endif

Couldn't we just remove that ifdef above and tell OBS to not trigger
coreutils rebuild if these packages changes?

I mean I find it really too strange nowadays. For util-linux I was not
even allowed to install "procps" anymore. We optimzed away the
simple "ps" command which is an absolute basic POSIX requirement!

> All the other packages - which are used for building - are identical.
>
> There might be still some risk, but I've never seen any fallout from
> this (which should be theoretically be 50% of such cases, i.e., where
> the binaries in coreutils are good but those in the testsuite are
> not).

I have also never seen any fallout for many other cases. I would say the
most rebuilds are not nessesary. Nethertheless we do all theoretically
needed rebuilds to be on the safe side. Just the "unimportant" testing
has been disabled to make things faster ...

cu,
Rudi
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Ruediger Meier
In reply to this post by Michael Matz
On Thursday 20 April 2017, Michael Matz wrote:

> Hi,
>
> On Thu, 20 Apr 2017, Ruediger Meier wrote:
> > No, you don't see *the* results. You can see *some* results of
> > different binaries which were built using a different build script
> > (spec file), different BuildRequires and probably on a different
> > build host.
>
> While your arguments are valid and true in the fullest generic sense,
> in the particular case we're talking about here (and in fact for most
> such cases) it's moot.  The binaries _are_ the same, so the tests are
> valid.
>
> And while it is also true that it would be even nicer if more
> packages would support installed testing, that's something for
> upstream to solve usually.  (Sometimes you can get away with running
> make check with the appropriate setting of various makefile
> variables).  But it would also be nicer if I had a pink zombie pony;
> alas it is not so.
>
> So, for the time being, and with the realities we're working in, and
> considering that some packages are in fact in the critical path and
> hence create a _real_ problem when they build for a long time, the
> %-testsuite extra .spec files are the sensible work around.  The
> other realistic work-around would be to not run the testsuite at all,
> and that would be worse.

I agree that splitting off *testsuite is not the most critical thing.

However my concerns are not only about the *testsuite packages but
mainly about our general way how to optimize the build cycles. Look at
our util-linux package. It's almost impossible to understand these spec
files and AFAIK no other distro is building ul in a similar manner,
completely different than supposed by upstream and autotools inventors.

> > To fix that we could add a subpackage providing the whole original
> > build directory and install this later (at QA time) to run the
> > missing check only.
>
> True, that would work as well.  It would also create many new
> extremely large binary packages nobody except automatic testers would
> use.  If it actually solves any problem we in reality have is
> doubtful.

Regarding reality problems. Here you see an accepted and forwarded
submit request where all the duplicated spec files are not in sync,
applying different patch-sets:
https://build.opensuse.org/request/show/487529

cu,
Rudi

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Michael Matz
Hi,

On Thu, 20 Apr 2017, Ruediger Meier wrote:

> I agree that splitting off *testsuite is not the most critical thing.
>
> However my concerns are not only about the *testsuite packages but
> mainly about our general way how to optimize the build cycles. Look at
> our util-linux package.

So, you have a problem with one specific package.  I don't see how that's
related to build cycle length, the optimization of it or the %check
section.

> It's almost impossible to understand these spec files and AFAIK no other
> distro is building ul in a similar manner, completely different than
> supposed by upstream and autotools inventors.

Hmm, don't really see what you mean, some things therein could also be
done by patch instead of in place editing, but otherwise it looks fairly
standard if a bit verbose.

> Regarding reality problems. Here you see an accepted and forwarded
> submit request where all the duplicated spec files are not in sync,
> applying different patch-sets:
> https://build.opensuse.org/request/show/487529

That's a bug, and it shouldn't have been accepted.  Alex forgot to call
pre_checkin.sh which must be done for every package that contains one (and
is even hinted at in the very .spec file itself).  That should probably be
automatically checked somehow.


Ciao,
Michael.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Philipp Thomas
In reply to this post by Bernhard Voelker
* Bernhard Voelker ([hidden email]) [20170420 14:28]:

> ... it has been like that for quite a while now.  I don't know what
> was the motivation behind the split, but I can imagine that this speeded
> up the build cycles on OBS dramatically.

With coreutils being a very basic core package, the split had two main
reasons.  The first one was reducing the build time and the second one that
running the testsuite adds dependencies on packages that the build doesn't
have.  Being that at the time the split was requested it was hinted that it
was planned to add to obs so that you could fail the build of another
package I grudgingly did the split. Of course that feature never
materialised...

Philipp
 
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-blocking %check

Andreas Färber
In reply to this post by Ruediger Meier
Am 20.04.2017 um 16:18 schrieb Ruediger Meier:
> On Thursday 20 April 2017, Adrian Schröter wrote:
>> Nevertheless, I think the seperate -testsuite packages are a proper
>> workaround for many cases. You can still see the results, and still
>> get build results soon.
>
> No, you don't see *the* results. You can see *some* results of different
> binaries which were built using a different build script (spec file),
> different BuildRequires and probably on a different build host.

You're making wrong assumptions, those are individual packagers' faults.

> To fix that we could add a subpackage providing the whole original build
> directory and install this later (at QA time) to run the missing check
> only.

Take a look at qemu-testsuite. I tackled that very issue by overriding
variables and/or symlinking the installed binaries into the build tree,
avoiding lengthy rebuilds and testing the very binaries the user gets.
Ugly, but it worked sufficiently.

Regards,
Andreas

--
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Loading...