Weblate upgrade planned (several days outage expected)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Weblate upgrade planned (several days outage expected)

Stanislav Brabec
Hi.

We finally solved problems that were blocking our Weblate server from
upgrade. We do not need to stick at the old Weblate 2.6 any more.

The bad news is the fact, that the upgrade and migration process will
take a non-trivial amount of machine time (about two days) to proceed.

For that time, Weblate has to be down or in read-only mode.

The date of the outage is not yet defined. I propose work days of the
next week. Please let us know if you plan any special plans for these
days.

We are planning to migrate:
from Weblate 2.6    on SLE12 SP1 and mysql
to   Weblate 2.13.1 on Leap 42.2 and postgresql

Leap has more python packages and packaging would require much less effort.


Here is the detailed description of the migration process:

1) Make Weblate read-only or off.

2) Backup the whole database and repositories.

3) Restore data on a staging machine with Leap 42.2.

4) Intermediate upgrade to Weblate 2.10:
https://build.opensuse.org/project/show/M17N:l10n.opensuse.org:weblate-2.10

It is not possible to upgrade directly to >=2.11, as
python-python-social-auth was replaced by python-social-auth-app-django,
and migration process is broken for our instance version.
https://github.com/python-social-auth/social-app-django/issues/26

5) Upgrade to patched Weblate 2.13.1. The upstream 2.13.1 has a broken
migration process and drops some comments.

The migration process changes the format of the database. Each

6) Dump mysql database by Django Shell commands.

7) Import database dump by Django Shell commands to a pre-initialized
database with some truncated tables.

This import will require at least 32GB of swap and will take many hours.

8) Switch to the new production server.

9) Create new permissions system.

--
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: [hidden email]
Křižíkova 148/34 (Corso IIa)                  tel: +49 911 7405384547
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Ludwig Nussel
Stanislav Brabec schrieb:
> We finally solved problems that were blocking our Weblate server from
> upgrade. We do not need to stick at the old Weblate 2.6 any more.
>
> The bad news is the fact, that the upgrade and migration process will
> take a non-trivial amount of machine time (about two days) to proceed.

Thanks for the heads up!
Given that we are six weeks from the release there is very little
room for failures of the new version. We're also about to deploy
software.o.o and I expect people to notice lots of missing
translations there as the last call to action didn't yield much.
I'm very much looking forward to a new version myself but I'm a bit
nervous to spoil the release fun :-) Do we need anything urgent or
could the migration be deferred to August, so after the release of
42.3?

cu
Ludwig

--
  (o_   Ludwig Nussel
  //\
  V_/_  http://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Ferdinand Galko
Hello all,

I also think that better time would be after Leap 42.3 release.

Best regards

Ferdinand
Slovak translator

2017-06-12 12:21 GMT+02:00 Ludwig Nussel <[hidden email]>:

> Stanislav Brabec schrieb:
>>
>> We finally solved problems that were blocking our Weblate server from
>> upgrade. We do not need to stick at the old Weblate 2.6 any more.
>>
>> The bad news is the fact, that the upgrade and migration process will
>> take a non-trivial amount of machine time (about two days) to proceed.
>
>
> Thanks for the heads up!
> Given that we are six weeks from the release there is very little
> room for failures of the new version. We're also about to deploy
> software.o.o and I expect people to notice lots of missing
> translations there as the last call to action didn't yield much.
> I'm very much looking forward to a new version myself but I'm a bit
> nervous to spoil the release fun :-) Do we need anything urgent or
> could the migration be deferred to August, so after the release of
> 42.3?
>
> cu
> Ludwig
>
> --
>  (o_   Ludwig Nussel
>  //\
>  V_/_  http://www.suse.com/
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
> Graham Norton, HRB 21284 (AG Nürnberg)
> --
> To unsubscribe, e-mail: [hidden email]
> To contact the owner, e-mail: [hidden email]
>
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Stanislav Brabec
In reply to this post by Ludwig Nussel
Ludwig Nussel wrote:

> Stanislav Brabec schrieb:
>> We finally solved problems that were blocking our Weblate server from
>> upgrade. We do not need to stick at the old Weblate 2.6 any more.
>>
>> The bad news is the fact, that the upgrade and migration process will
>> take a non-trivial amount of machine time (about two days) to proceed.
>
> Thanks for the heads up!
> Given that we are six weeks from the release there is very little
> room for failures of the new version. We're also about to deploy
> software.o.o and I expect people to notice lots of missing
> translations there as the last call to action didn't yield much.
> I'm very much looking forward to a new version myself but I'm a bit
> nervous to spoil the release fun :-) Do we need anything urgent or
> could the migration be deferred to August, so after the release of
> 42.3?

The risk of migration itself is zero. The old instance would not be
altered at all. The whole migration will be performed on an offline
backup instance. After completing of the migration process, new server
and new database will be used. If anything will fail, it is easy to
switch back.

You can preview new weblate versions on https://hosted.weblate.org/

The turning the old Weblate down has the only one purpose: Perform
migration on an up-to-date data set.


Testing and fixing of the migration process took a long time. Skipping
many versions and two years of development is far from being trivial.
Longer waiting will complicate the migration even more.

Even now, weblate released a newer version 2.14.1. But the migration was
tuned and tested for 2.13.1, so we will do it for 2.13.1 and migrate to
the latest version afterwards. Migration from 2.6 to any recent version
could require different hacks, but migration from 2.13.1 to the next
version should be seamless.

If anything will fail, it is easily possible to switch to Weblate 2.6
(one line in the opensuse.org routing tables).

We can tune up the new instance for a few days in a "Testing period",
where users will be discouraged to perform big changes (comments,
suggestions etc.). If any serious problem will be found, we can revert.
If not, stay at the new version.

--
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: [hidden email]
Křižíkova 148/34 (Corso IIa)                  tel: +49 911 7405384547
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Sarah Julia Kriesch
> Gesendet: Montag, 12. Juni 2017 um 18:01 Uhr
> Von: "Sarah Julia Kriesch" <[hidden email]>
> An: "Stanislav Brabec" <[hidden email]>
> Betreff: Aw: Re: [opensuse-translation] Weblate upgrade planned (several days outage expected)
>
> I can see different options:
> 1) You can do the migration during the working week. Most translotors are working then and want to translate during the weekend.
> 2) We wait for the release day of Leap. If Stanislav wants to use a special version of Weblate, he can download it now. Generally open source projects have got a download page with a list with all released (old) versions. He can use it, too.
> I
> Another question about the migration migration process: What is the reason for the change from mysql to postgresql? That will be a difficult change. Postgres isn't the same and I was allowed to repair such a database in the last year. I believe, you will have such problems during the migration, too.
>
> But "Thank you for the upgrade in the future! The change management process sounds really good."
>
> Best regards,
> Sarah
>
>
> > Gesendet: Montag, 12. Juni 2017 um 17:36 Uhr
> > Von: "Stanislav Brabec" <[hidden email]>
> > An: "Ludwig Nussel" <[hidden email]>, [hidden email]
> > Betreff: Re: [opensuse-translation] Weblate upgrade planned (several days outage expected)
> >
> > Ludwig Nussel wrote:
> > > Stanislav Brabec schrieb:
> > >> We finally solved problems that were blocking our Weblate server from
> > >> upgrade. We do not need to stick at the old Weblate 2.6 any more.
> > >>
> > >> The bad news is the fact, that the upgrade and migration process will
> > >> take a non-trivial amount of machine time (about two days) to proceed.
> > >
> > > Thanks for the heads up!
> > > Given that we are six weeks from the release there is very little
> > > room for failures of the new version. We're also about to deploy
> > > software.o.o and I expect people to notice lots of missing
> > > translations there as the last call to action didn't yield much.
> > > I'm very much looking forward to a new version myself but I'm a bit
> > > nervous to spoil the release fun :-) Do we need anything urgent or
> > > could the migration be deferred to August, so after the release of
> > > 42.3?
> >
> > The risk of migration itself is zero. The old instance would not be
> > altered at all. The whole migration will be performed on an offline
> > backup instance. After completing of the migration process, new server
> > and new database will be used. If anything will fail, it is easy to
> > switch back.
> >
> > You can preview new weblate versions on https://hosted.weblate.org/
> >
> > The turning the old Weblate down has the only one purpose: Perform
> > migration on an up-to-date data set.
> >
> >
> > Testing and fixing of the migration process took a long time. Skipping
> > many versions and two years of development is far from being trivial.
> > Longer waiting will complicate the migration even more.
> >
> > Even now, weblate released a newer version 2.14.1. But the migration was
> > tuned and tested for 2.13.1, so we will do it for 2.13.1 and migrate to
> > the latest version afterwards. Migration from 2.6 to any recent version
> > could require different hacks, but migration from 2.13.1 to the next
> > version should be seamless.
> >
> > If anything will fail, it is easily possible to switch to Weblate 2.6
> > (one line in the opensuse.org routing tables).
> >
> > We can tune up the new instance for a few days in a "Testing period",
> > where users will be discouraged to perform big changes (comments,
> > suggestions etc.). If any serious problem will be found, we can revert.
> > If not, stay at the new version.
> >
> > --
> > Best Regards / S pozdravem,
> >
> > Stanislav Brabec
> > software developer
> > ---------------------------------------------------------------------
> > SUSE LINUX, s. r. o.                         e-mail: [hidden email]
> > Křižíkova 148/34 (Corso IIa)                  tel: +49 911 7405384547
> > 186 00 Praha 8-Karlín                          fax:  +420 284 084 001
> > Czech Republic                                    http://www.suse.cz/
> > PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76
> > --
> > To unsubscribe, e-mail: [hidden email]
> > To contact the owner, e-mail: [hidden email]
> >
> >
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Stanislav Brabec
Sarah Julia Kriesch wrote:

> 1) You can do the migration during the working week. Most translotors
are working then and want to translate during the weekend.

Keeping weekends on-line was my intention as well.

> 2) We wait for the release day of Leap. If Stanislav wants to use a
special version of Weblate, he can download it now. Generally open
source projects have got a download page with a list with all released
(old) versions. He can use it, too.

I always wanted to use the latest version. But SLE12 (SP1) was not a
good decision for Weblate. It lacks too many packages, backporting them
took many time, and often failed.

It caused delay in regular updates. It not only became outdated, running
an unsupported weblate version on an unsupported OS. It is so old
version that the migration to the new version is even not supported! We
have to use intermediate upgrade to succeed.

Now we finally have a working packages for Leap 42.2 and we also solved
all migration problems. Everything compiles, upgrade succeeds with a
special process on a testing copy of the current L10N snapshot, no data
is lost.

> Another question about the migration migration process: What is the
reason for the change from mysql to postgresql?

Postgres is the main development platform of Weblate. We are probably
the only large instance of Weblate using MariaDB and Apache.

Also openSUSE infrastructure people prefer postgres, as it already
serves large part of the infrastructure.

> That will be a difficult change. Postgres isn't the same and I was
allowed to repair such a database in the last year. I believe, you will
have such problems during the migration, too.

Hopefully, Weblate is a Django application, and most database
differences are hidden behind Django.

We already performed the change on a testing instance, and everything
worked.

We met only technical problems in this migration:

- Current Django migration tools import the whole database into the
memory. It requires about 32 GB RAM or swap.

- Some tables are dependent on other tables. These have to be created in
a correct order in a destination database.

> But "Thank you for the upgrade in the future! The change management
process sounds really good."

We tried the migration process on a L10N snapshot many times before I
was able to send this mail.

--
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: [hidden email]
Křižíkova 148/34 (Corso IIa)                  tel: +49 911 7405384547
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Sarah Julia Kriesch


> Gesendet: Montag, 12. Juni 2017 um 19:51 Uhr
> Von: "Stanislav Brabec" <[hidden email]>
> An: "Sarah Julia Kriesch" <[hidden email]>, OS-trans <[hidden email]>
> Betreff: Re: [opensuse-translation] Weblate upgrade planned (several days outage expected)
>
> Sarah Julia Kriesch wrote:
>
> > 1) You can do the migration during the working week. Most translotors
> are working then and want to translate during the weekend.
>
> Keeping weekends on-line was my intention as well.
Then I would say: "Do it!" :-)
But look after the activity in different languages before choosing the day.
It seems the middle of the week is the best time. Some people have contributed on Monday a lot, too.
This week is bad, because we have got a public holiday on Thursday and some people want to have a free Friday then.
Tell us what you want to choose. So we can plan our translations.

>
> > 2) We wait for the release day of Leap. If Stanislav wants to use a
> special version of Weblate, he can download it now. Generally open
> source projects have got a download page with a list with all released
> (old) versions. He can use it, too.
>
> I always wanted to use the latest version. But SLE12 (SP1) was not a
> good decision for Weblate. It lacks too many packages, backporting them
> took many time, and often failed.
>
> It caused delay in regular updates. It not only became outdated, running
> an unsupported weblate version on an unsupported OS. It is so old
> version that the migration to the new version is even not supported! We
> have to use intermediate upgrade to succeed.
>
> Now we finally have a working packages for Leap 42.2 and we also solved
> all migration problems. Everything compiles, upgrade succeeds with a
> special process on a testing copy of the current L10N snapshot, no data
> is lost.
>
Did you think about using 42.3 for that?
Christian and I want to rollout the test wiki on 42.3 before the release day.
 

> > Another question about the migration migration process: What is the
> reason for the change from mysql to postgresql?
>
> Postgres is the main development platform of Weblate. We are probably
> the only large instance of Weblate using MariaDB and Apache.
>
> Also openSUSE infrastructure people prefer postgres, as it already
> serves large part of the infrastructure.
>
> > That will be a difficult change. Postgres isn't the same and I was
> allowed to repair such a database in the last year. I believe, you will
> have such problems during the migration, too.
>
> Hopefully, Weblate is a Django application, and most database
> differences are hidden behind Django.
>
> We already performed the change on a testing instance, and everything
> worked.
>
> We met only technical problems in this migration:
>
> - Current Django migration tools import the whole database into the
> memory. It requires about 32 GB RAM or swap.
>
> - Some tables are dependent on other tables. These have to be created in
> a correct order in a destination database.
>
> > But "Thank you for the upgrade in the future! The change management
> process sounds really good."
>
> We tried the migration process on a L10N snapshot many times before I
> was able to send this mail.
>
> --
> Best Regards / S pozdravem,
>
> Stanislav Brabec
> software developer
 
I am happy having you in the Heroes Team. :-)

One hint for the next change:
Split your information mail into 2 emails (heroes and translation)in the future, please.
The change management and downtime is important for most heroes. We can review that and you can receive your "Go!" there.
Sometimes that will be a topic in the Heroes meeting.
Not every Translator is interested for the change management process. Translators want to have a online system and need only the planned downtime. You can ask on the translation mailinglist for the preferred time/ date.
After that you can have a perfect organized change. :-)

We are a minimum of 3 Heroes on this mailinglist now (Ludwig, I and you). So you can receive feedback by us.
But a review by others would give you more security.

Best regards,
Sarah
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Stanislav Brabec
Sarah Julia Kriesch wrote:
>
> But look after the activity in different languages before choosing the day.
> It seems the middle of the week is the best time. Some people have contributed on Monday a lot, too.
> This week is bad, because we have got a public holiday on Thursday and some people want to have a free Friday then.
> Tell us what you want to choose. So we can plan our translations.

What about next week, starting with Tuesday? If everything will succeed
on the first try, it will be back online with a new version on Thursday.
In case or one problem, it will be back in Friday. In case of two
problems (power outage, out of memory etc.), i will switch the old
instance back to R/W mode in Friday, and start next try on the next Tuesday.

Once we will get a consensus on the date, I will send a separate mail.

> Did you think about using 42.3 for that?
> Christian and I want to rollout the test wiki on 42.3 before the release day.

Not yet. But after the release, upgrade to 42.3 will be surely a good
option. I enabled 42.3 repository in the OBS as a first step. Weblate is
sensitive to python modules versions, but hopefully, the latest version
works best in most cases.

I am now trying to package weblate-2.14.1, but the testsuite fails on
segfault, so I will probably to to Leap 42.2 + weblate-2.13.1.

> One hint for the next change:
> Split your information mail into 2 emails (heroes and translation)in the future, please.
> The change management and downtime is important for most heroes. We can review that and you can receive your "Go!" there.
> Sometimes that will be a topic in the Heroes meeting.

Good hint.

It's true that translators don't care about details of migration of
large Django projects. They need basically only Up/Down and when.


--
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: [hidden email]
Křižíkova 148/34 (Corso IIa)                  tel: +49 911 7405384547
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Sarah Julia Kriesch


> Gesendet: Dienstag, 13. Juni 2017 um 15:19 Uhr
> Von: "Stanislav Brabec" <[hidden email]>
> An: "Sarah Julia Kriesch" <[hidden email]>
> Cc: OS-trans <[hidden email]>
> Betreff: Re: [opensuse-translation] Weblate upgrade planned (several days outage expected)
>
> Sarah Julia Kriesch wrote:
> >
> > But look after the activity in different languages before choosing the day.
> > It seems the middle of the week is the best time. Some people have contributed on Monday a lot, too.
> > This week is bad, because we have got a public holiday on Thursday and some people want to have a free Friday then.
> > Tell us what you want to choose. So we can plan our translations.
>
> What about next week, starting with Tuesday? If everything will succeed
> on the first try, it will be back online with a new version on Thursday.
> In case or one problem, it will be back in Friday. In case of two
> problems (power outage, out of memory etc.), i will switch the old
> instance back to R/W mode in Friday, and start next try on the next Tuesday.
>
> Once we will get a consensus on the date, I will send a separate mail.
That sounds good for me. Let's wait for other opinions.
I hope the first try would succeed. ;-)

Or will other translators have public holidays next week and you want to translate Thursday until Tuesday?
Then say it, please.

Best regards,
Sarah
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Weblate upgrade planned (several days outage expected)

Ludwig Nussel
In reply to this post by Stanislav Brabec
Stanislav Brabec schrieb:

> Sarah Julia Kriesch wrote:
>>
>> But look after the activity in different languages before choosing the day.
>> It seems the middle of the week is the best time. Some people have contributed on Monday a lot, too.
>> This week is bad, because we have got a public holiday on Thursday and some people want to have a free Friday then.
>> Tell us what you want to choose. So we can plan our translations.
>
> What about next week, starting with Tuesday? If everything will succeed
> on the first try, it will be back online with a new version on Thursday.
> In case or one problem, it will be back in Friday. In case of two
> problems (power outage, out of memory etc.), i will switch the old
> instance back to R/W mode in Friday, and start next try on the next Tuesday.

Ok.

cu
Ludwig

--
  (o_   Ludwig Nussel
  //\
  V_/_  http://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Loading...