RFC: slow ftp product builds

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

RFC: slow ftp product builds

Dinar Valeev
Hi,

I'm looking for a solution to fix a slow ftp tree build, it takes
about 5 hours to build it. It is fine to wait, but in Tumbleweed days
where many snapshots released during the week this is rather slow.

We build medias and ftp on special "local" arch, which have special
workers for it. x86 got fancy machines, while "Ports" have to share a
single machine.

Here are the numbers to compare:
2016-04-18 17:11:19  _product:openSUSE-ftp-ftp-ppc64_ppc64le
 meta change      succeeded            4h 51m 41s   build36:1

2016-04-17 20:59:09  _product:openSUSE-ftp-ftp-i586_x86_64
 source change    succeeded               39m 47s   typhoon1:1

Building on local have other disadvantages, like we depend on x86 to
get kiwi build env since local is x86 [1]. And while we were waiting
for x86 kiwi to finish build, x86 product is passed to openQA already,
since this moment a new source checking round might happen, which
brings me to the beginning of the loop (build packages, wait for kiwi,
start building product).

Having multiple machines for local will not help IMO, we can't predict
where build will be scheduled, on a host with cache or not.

There is an other option, to build ftp on native arch, basically bring
your own product builder. But then tooling around Tumbleweed assumes
products to work with are in 'local' architecture.

Given Ports family is growing (zSystems and possible Leap i586), all
ports architectures will benefit from it.

[1] https://github.com/openSUSE/open-build-service/blob/master/src/backend/BSConfig.pm.template#L166

Dinar,
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Srinidhi B-2
Hello Dinar,

>>> On Tuesday, 19 April, 2016    at 03:37 AM, Dinar Valeev <[hidden email]>
wrote:
> Hi,
>
> I'm looking for a solution to fix a slow ftp tree build, it takes
> about 5 hours to build it. It is fine to wait, but in Tumbleweed days
> where many snapshots released during the week this is rather slow.
>

First of all, I'm not an expert and this might not be a "solution", but a few months back I enabled "caching" on our KIWI worker and build times have significantly improved.

    # grep CACHE /etc/sysconfig/obs-worker
    OBS_CACHE_DIR="/path/to/pkg-cache"
    OBS_CACHE_SIZE="50000"
    OBS_WORKER_CACHE_SIZE=""

I'm not sure if there are any disadvantages of using caching, but purely from "build time" perspective, we have seen a lot of improvement!

Hope this helps,
Srinidhi.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Dinar Valeev
On Tue, Apr 19, 2016 at 12:21 AM, Srinidhi B <[hidden email]> wrote:

> Hello Dinar,
>
>>>> On Tuesday, 19 April, 2016    at 03:37 AM, Dinar Valeev <[hidden email]>
> wrote:
>> Hi,
>>
>> I'm looking for a solution to fix a slow ftp tree build, it takes
>> about 5 hours to build it. It is fine to wait, but in Tumbleweed days
>> where many snapshots released during the week this is rather slow.
>>
>
> First of all, I'm not an expert and this might not be a "solution", but a few months back I enabled "caching" on our KIWI worker and build times have significantly improved.
>
>     # grep CACHE /etc/sysconfig/obs-worker
>     OBS_CACHE_DIR="/path/to/pkg-cache"
>     OBS_CACHE_SIZE="50000"
>     OBS_WORKER_CACHE_SIZE=""
We need to have a cache big enough to fit all ports in.

I've looked at build36, it doesn't use cache at all. I reconfigured
it, let see what would be an improvement if any.

Thanks,

>
> I'm not sure if there are any disadvantages of using caching, but purely from "build time" perspective, we have seen a lot of improvement!
>
> Hope this helps,
> Srinidhi.
>
> --
> To unsubscribe, e-mail: [hidden email]
> To contact the owner, e-mail: [hidden email]
>
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Dinar Valeev
On Tue, Apr 19, 2016 at 11:43 AM, Dinar Valeev <[hidden email]> wrote:

> On Tue, Apr 19, 2016 at 12:21 AM, Srinidhi B <[hidden email]> wrote:
>> Hello Dinar,
>>
>>>>> On Tuesday, 19 April, 2016    at 03:37 AM, Dinar Valeev <[hidden email]>
>> wrote:
>>> Hi,
>>>
>>> I'm looking for a solution to fix a slow ftp tree build, it takes
>>> about 5 hours to build it. It is fine to wait, but in Tumbleweed days
>>> where many snapshots released during the week this is rather slow.
>>>
>>
>> First of all, I'm not an expert and this might not be a "solution", but a few months back I enabled "caching" on our KIWI worker and build times have significantly improved.
>>
>>     # grep CACHE /etc/sysconfig/obs-worker
>>     OBS_CACHE_DIR="/path/to/pkg-cache"
>>     OBS_CACHE_SIZE="50000"
>>     OBS_WORKER_CACHE_SIZE=""
> We need to have a cache big enough to fit all ports in.
>
> I've looked at build36, it doesn't use cache at all. I reconfigured
> it, let see what would be an improvement if any.
So far cache is growing but it is now 78G and keeps growing?

I wonder how much space we need per FTP tree?

>
> Thanks,
>>
>> I'm not sure if there are any disadvantages of using caching, but purely from "build time" perspective, we have seen a lot of improvement!
>>
>> Hope this helps,
>> Srinidhi.
>>
>> --
>> To unsubscribe, e-mail: [hidden email]
>> To contact the owner, e-mail: [hidden email]
>>
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Andreas Schwab
Dinar Valeev <[hidden email]> writes:

> So far cache is growing but it is now 78G and keeps growing?

That's about you need for a single biarch repo.

> I wonder how much space we need per FTP tree?

50G at minimum.

Andreas.

--
Andreas Schwab, [hidden email]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Dinar Valeev
On Tue, Apr 19, 2016 at 1:38 PM, Andreas Schwab <[hidden email]> wrote:
> Dinar Valeev <[hidden email]> writes:
>
>> So far cache is growing but it is now 78G and keeps growing?
>
> That's about you need for a single biarch repo.
It is 130GB now. So my native worker wouldn't fit here. It have only 96GB of RAM

>
>> I wonder how much space we need per FTP tree?
>
> 50G at minimum.
>
> Andreas.
>
> --
> Andreas Schwab, [hidden email]
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: slow ftp product builds

Srinidhi B-2
Hello Dinar,

>>> On Tuesday, 19 April, 2016    at 05:43 PM, Dinar Valeev <[hidden email]>
wrote:
> > That's about you need for a single biarch repo.
> It is 130GB now. So my native worker wouldn't fit here. It have only 96GB of
> RAM
> >

Might be a stupid question, but the cache is on physical disk and bs_worker will use this cache in a VM running in RAM. Correct? So you could actually store 150GB+ cache but may not use the exact size when building the product.

I have no experience in building such huge product ISOs, so I could be wrong. We build a relatively small (~800MB) add-on product ISO, but our package cache is already 49GB - almost all of the size allocated to the cache! :)

Srinidhi.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]