Build prioritization based on packages blocked and length of build

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Build prioritization based on packages blocked and length of build

Jimmy Berry-2
Does OBS prioritize large/important jobs to more powerful workers?

Leap:15.0 has 40% of packages blocked at this point by one very slow job. The
page shows it not even half done and over the 100% mark based on previous.
Contrast that with the OBS monitor page shows ~75% of workers idle.

It would seem straight-forward and very beneficial to prioritize packages that
block lots of others and secondary to that longer build times onto more
powerful machines. I have done this sort of optimization for job queues
elsewhere and seen major improvements.

I have also seen the same sort of behavior with libreoffice builds which are
far too long under the best of circumstances and awful when they are slower.

Does OBS have any prioritization besides based on the project in which the
package resides?

It seems one can even recognize the stalemate on the overview graph [1] which
shows the blocked flattening out much more quickly (even before making it to
the previous flat).

[1] https://i.imgur.com/qpk0nxE.png

--
Jimmy
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Build prioritization based on packages blocked and length of build

Adrian Schröter
On Freitag, 27. Oktober 2017, 07:49:10 CEST wrote Jimmy Berry:

> Does OBS prioritize large/important jobs to more powerful workers?
>
> Leap:15.0 has 40% of packages blocked at this point by one very slow job. The
> page shows it not even half done and over the 100% mark based on previous.
> Contrast that with the OBS monitor page shows ~75% of workers idle.
>
> It would seem straight-forward and very beneficial to prioritize packages that
> block lots of others and secondary to that longer build times onto more
> powerful machines. I have done this sort of optimization for job queues
> elsewhere and seen major improvements.
>
> I have also seen the same sort of behavior with libreoffice builds which are
> far too long under the best of circumstances and awful when they are slower.
>
> Does OBS have any prioritization besides based on the project in which the
> package resides?

yes, but not in this regard. The dispatcher takes the trigger reason
into account in first place (new builds or source changed builds a prefered).

A prefer of "more important jobs" it not so easy, given that there is not a single
value to rate a worker. Some jobs are disk IO bound, some are memory bound, others
are cpu bound and yet another ones need special cpu hardware features to be faster.

Also the build graph can change on every build, so it is not taken into account.

However, you can of course define constraints to require a certain number cpus, memory
or alike for packages which you see critical.

> It seems one can even recognize the stalemate on the overview graph [1] which
> shows the blocked flattening out much more quickly (even before making it to
> the previous flat).
>
> [1] https://i.imgur.com/qpk0nxE.png

That graph shows in first place that there the speed of the dispatcher was a bottle
neck in first place. Not all workers were used, but jobs did exist. This can happen
when some workers slowing down the answer on job assignment.

--

Adrian Schroeter
email: [hidden email]

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
 
Maxfeldstraße 5                        
90409 Nürnberg
Germany


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Build prioritization based on packages blocked and length of build

Jimmy Berry-2
On Friday, October 27, 2017 1:33:43 AM CDT Adrian Schröter wrote:

> On Freitag, 27. Oktober 2017, 07:49:10 CEST wrote Jimmy Berry:
> > Does OBS prioritize large/important jobs to more powerful workers?
> >
> > Leap:15.0 has 40% of packages blocked at this point by one very slow job.
> > The page shows it not even half done and over the 100% mark based on
> > previous. Contrast that with the OBS monitor page shows ~75% of workers
> > idle.
> >
> > It would seem straight-forward and very beneficial to prioritize packages
> > that block lots of others and secondary to that longer build times onto
> > more powerful machines. I have done this sort of optimization for job
> > queues elsewhere and seen major improvements.
> >
> > I have also seen the same sort of behavior with libreoffice builds which
> > are far too long under the best of circumstances and awful when they are
> > slower.
> >
> > Does OBS have any prioritization besides based on the project in which the
> > package resides?
>
> yes, but not in this regard. The dispatcher takes the trigger reason
> into account in first place (new builds or source changed builds a
> prefered).
>
> A prefer of "more important jobs" it not so easy, given that there is not a
> single value to rate a worker. Some jobs are disk IO bound, some are memory
> bound, others are cpu bound and yet another ones need special cpu hardware
> features to be faster.
>
> Also the build graph can change on every build, so it is not taken into
> account.
>
> However, you can of course define constraints to require a certain number
> cpus, memory or alike for packages which you see critical.
>
> > It seems one can even recognize the stalemate on the overview graph [1]
> > which shows the blocked flattening out much more quickly (even before
> > making it to the previous flat).
> >
> > [1] https://i.imgur.com/qpk0nxE.png
>
> That graph shows in first place that there the speed of the dispatcher was a
> bottle neck in first place. Not all workers were used, but jobs did exist.
> This can happen when some workers slowing down the answer on job
> assignment.

If metrics are recorded during job run it should be clear which resources are
the bottle neck and basically keep a psuedo constraint. Additionally, simply
assuming that jobs with hour+ (or some number based on the average build time)
should be assumed to be better on more powerful workers. For simplicity again
could be treated as an automatic constraint or weak constraint.

There clearly are very quick jobs (like non-compiled languages, or small
projects) and there are very slow jobs. The primary goal is to differentiate
between those two classes so the base 2-3 hours (or 6 hour) jobs do not end up
taking 12 hours due to unlucky scheduling on a slow worker.

The work required to do something like this seems reasonable and when
considering the much more efficient use of build power should be quite
noticeable.

Manual analysis and generation of constraint files could also be done, but
would really be a workaround for something that is really in the OBS
scheduling domain.

--
Jimmy
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]