openSUSE staging workflow metrics path forward

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

openSUSE staging workflow metrics path forward

Jimmy Berry-2
A while back you may remember a request for read access to a few tables or a
snapshot of them from which one could work on extracting metrics of interest.
That request never went anywhere so when revisiting this subject recently I
went for a completely external approach using the API and some processing
code. Since InfluxDB and Grafana were mentioned as the tools which the OBS
team would use if they were to provide metrics I went ahead and used those.

The code is built atop the osc-plugin-factory/osclib [1] and primarily uses /
search/request API call to retrieve requests for given projects of interest.
The code then walks the combined state changes of all the requests to re-
create the staging state at any point in time. A variety of metrics are
generated along the way and some quite interesting graphs can be produced in
Grafana. Additionally, I plan to create an event log [2] for use by the
release team both for SLE and openSUSE to provide a clear overview of what is
being done by the various staging masters.

Long story short, I plan to cleanup and push the code which can be used to
update the metrics on the regular basis. The original responses indicated that
if something is setup there may be interest in moving forward with the
metrics. Is there interest in hosting an InfluxDB and Grafana instance that
could also be expanded with metrics added directly by OBS? If there is no
interest I will look into setting up a proper host for the release teams.

The URL to my local instances has been shared a few times, but I would rather
not include it in a public mailing list since it is not setup for any sort of
load or real public access. If individuals are interested feel free to reply
or contact me in IRC.

As a teaser, here are some of the work-in-progress graphs which only include
accepted submit requests. The negative dip on the first Factory graph is
documented in the OBS bug regarding dates [3]. In fact the graph would be much
worse without some major workarounds to detect and avoid such corrupted data.

- https://imgur.com/a/lqMPo

Lastly, I had quite a time walking the data structure to extract the desired
values and documented a number of bugs and problems [4].

[1] https://github.com/openSUSE/osc-plugin-factory/tree/master/osclib
[2] https://github.com/openSUSE/osc-plugin-factory/issues/730
[3] https://github.com/openSUSE/open-build-service/issues/3858
[4] https://github.com/openSUSE/open-build-service/issues/3898

--
Jimmy
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: openSUSE staging workflow metrics path forward

Jimmy Berry-2
Is it safe to say the silence indicates no interest and I should move forward
with hosting for the releaseteam?

--
Jimmy
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: openSUSE staging workflow metrics path forward

Henne Vogelsang-2
In reply to this post by Jimmy Berry-2
Hey,

On 22.09.2017 22:25, Jimmy Berry wrote:

> Is there interest in hosting an InfluxDB and Grafana instance that
> could also be expanded with metrics added directly by OBS?

There is interest, as you already know we are planing to use those tools
for application health monitoring. But we have no ETA for this, sorry.

If you need something now, you have to host it yourself.

Henne

--
Henne Vogelsang
http://www.opensuse.org
Everybody has a plan, until they get hit.
        - Mike Tyson
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: openSUSE staging workflow metrics path forward

Jimmy Berry-2
In reply to this post by Jimmy Berry-2
For those interested my VM request was fulfilled and I setup the machine [1].
Still some issues to be worked out and improvements that can be made to graph
titles and descriptions, but nice to have a solid basis publicly available.

Besides RAM limitations during Factory ingest the primary item to resolve is a
clean update path rather than re-requesting all data. See #4108: [api] allow
ordering to be controlled on /search/requests [2].

[1] https://metrics.opensuse.org/
[2] https://github.com/openSUSE/open-build-service/issues/4108

--
Jimmy
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]