python singlespec: how to convert your package

[jan matejek-4]

Dear packagers,

this is an overview of how singlespec works and how to use it for your
package. First, some key concepts:

* It will allow you to build Python 2 and Python 3 (and PyPy 3 in the
future) package from a single spec file, with very little manual effort.
This means that if you're maintaining a separate python3-something in
d:l:py3, you can now switch over to the common python-something in d:l:py.

* It is designed to work on specs called "python-something". There is
some support for "python2-something" and "python3-something", but it is
mostly untested.
If your spec file name doesn't start with "python", don't even try.
(Yet.)

* (if you're building for Factory,) all your dependencies must be
converted first.
That's because the macros will BuildRequire "python2-things", which only
exist for packages that are already converted.
The devel:languages:python:singlespec already has basic packages, and
not all of them will be in d:l:py immediately, so if you want to play
now, build against this project. I'll also be accepting SRs into it.

* In general, if you're unsure, please submit to d:l:py:singlespec
first. Also examine linkdiffs of packages in d:l:py:singlespec to get an
idea of how it can look like.
https://build.opensuse.org/project/show/devel:languages:python:singlespec

* Individual macros are documented on github
https://github.com/openSUSE/python-rpm-macros

* I'm working on an autoconverter that performs most of the following
steps automatically. I intend to use it to convert most of d:l:py into
d:l:py:singlespec:staging, and then review and submit packages one by
one. So maybe if you wait, you can save yourself some work ;)

Conversion steps:

0. If you want to build for anything other than Factory, you need to
include compatibility shims.
This definition of the %python_module macro goes on top of your spec:
%{?!python_module:%define python_module() python-%1 python3-%1}
and a buildrequire:
BuildRequires: python-rpm-macros

1. Wrap all your "BuildRequires: python-something":
BuildRequires: %{python_module something}
Do not do this in Requires or any other fields.
If you have a buildrequire that should only apply to one python, say,
python-enum34 only for python 2, do not wrap it.

2. At the end of the spec preamble, before %description, place the macro:
%python_subpackages
on a line by itself

3. Replace "python setup.py build" with %python_build
and "python setup.py install" with %python_install.
These already contain the usual arguments (--prefix, --root), so usually
you don't need to add any.
If you configure CFLAGS or something like that, export them first:

CFLAGS="-fwrapv" python setup.py build
->
export CFLAGS="-fwrapv"
%python_build

If you are running any other commands based on python, use %python_exec.
"python setup.py test" -> "%python_exec setup.py test"
This works for usual test runners too:
"nosetests" -> "%python_exec %{_bindir}/nosetests"

4. If you're doing something more complicated ... say, removing files
from %python2_sitelib and %python3_sitelib, use %python_expand.
"%python_expand rm -f %{$python_sitelib}/foo/undesirable.txt"

Yes, that is "%$python_sitelib". Beware. If you put only
"%python_sitelib", it will not work, because "%python_sitelib" gets
expanded before %python_expand touches it.

In general, after %python_expand, "$python" is replaced by all pythons
even in macro names.
If you need more lines, wrap in braces:
%{python_expand firstline
secondline $python
thirdline}
Note that the first line must not be empty.

5. change "%files" to "%files %{python_files}"
If you have subpackages, say "%files foo", then
change to "%files %{python_files foo}".

6. If some files are only in one python version, mark them as
%python2_only or %python3_only in the filelist.
If there's more of them, you can use %ifpython2 ... %endif
If the entry in question is a __pycache__ directory, use %pycache_only
or %ifpycache.

7. If you are using update-alternatives, remove them.
Instead mark the executable as %python3_only:
%python3_only %{_bindir}/yourbinary
In most cases, we only need one version of the executable. If the
purpose of the tool is, e.g., reading EXIF metadata, we don't care if
python 2 or 3 reads them. (And the library files are still installed for
both.)

(If you are sure that you need executables for all python versions, look
at what python-bottle does:
https://build.opensuse.org/package/view_file/devel:languages:python:singlespec/python-bottle/python-bottle.spec?expand=1
note the installation inside %python_expand. Usually, packages don't
install versioned executables by themselves, so you need to do it
manually. If your package does, good for you.)

8. Take the time to enable automatic tests :)

have a nice weekend
m.

[Dominique Leuenberger / DimStar]

On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
>
> * Individual macros are documented on github
> https://github.com/openSUSE/python-rpm-macros

As this now becomes 'law' for the openSUSE packaging: can you please
ensure that the openSUSE wiki reflects the method of singlespec, giving
guidance on the whole thing?

This is quite important for users as well as for the review team to
have clear info on what must be, what should be and what must not be
present.

Thanks,
Dominique

[gregfreemyer]

On Fri, Feb 17, 2017 at 12:31 PM, jan matejek <[hidden email]> wrote:

> Dear packagers,
>
> this is an overview of how singlespec works and how to use it for your
> package. First, some key concepts:
>
> * It will allow you to build Python 2 and Python 3 (and PyPy 3 in the
> future) package from a single spec file, with very little manual effort.
> This means that if you're maintaining a separate python3-something in
> d:l:py3, you can now switch over to the common python-something in d:l:py.
>
> * It is designed to work on specs called "python-something". There is
> some support for "python2-something" and "python3-something", but it is
> mostly untested.
> If your spec file name doesn't start with "python", don't even try.
> (Yet.)
>
> * (if you're building for Factory,) all your dependencies must be
> converted first.
> That's because the macros will BuildRequire "python2-things", which only
> exist for packages that are already converted.
> The devel:languages:python:singlespec already has basic packages, and
> not all of them will be in d:l:py immediately, so if you want to play
> now, build against this project. I'll also be accepting SRs into it.

I have 13 python- packages not in d:l:p.  Should I SR a copy to singlespec?

I also have a bunch of non-python packages that have python-
sub-packages for python bindings.  Should I just ensure manually that
they build python2- and python3- sub-packages?

I'm the only consumer of these python binding sub-packages.  Should I
just drop python- versions of the sub-packages?

Thanks
Greg
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[jan matejek-4]

hello,

On 22.2.2017 06:08, Greg Freemyer wrote:
>
> I have 13 python- packages not in d:l:p.  Should I SR a copy to singlespec?

No. I'll be submitting all of singlespec to d:l:py at some point, this
would only cause confusion.

Let me know if you want me to look at the packages though.

>
> I also have a bunch of non-python packages that have python-
> sub-packages for python bindings.  Should I just ensure manually that
> they build python2- and python3- sub-packages?

At the moment, old-style packages should work fine, so I suggest leaving
it as-is, until there is singlespec support for packages like this.
(it's the next thing on my todo-list once the transition is on its way)

You can rename python- bindings to python2-, but this will probably only
be acceptable for Factory.

> I'm the only consumer of these python binding sub-packages.  Should I
> just drop python- versions of the sub-packages?

I'm not sure I understand the question. Are you asking if you should
only build Python 3 version? Probably not -- you might be the only
consumer now, but why not have both (or all) pythons supported.

Or are you asking about moving from python- to python2- ? Again, that is
up to you.

>
> Thanks
> Greg
>

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[gregfreemyer]

On Wed, Feb 22, 2017 at 8:17 AM, jan matejek <[hidden email]> wrote:
> Or are you asking about moving from python- to python2- ? Again, that is
> up to you

Thanks, then I will leave them as they are.

Greg
--
Greg Freemyer
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Thomas Bechtold]

Hi Jan,

what should I do if I want to specify a minimum version for a
BuildRequires? Something like:

BuildRequires: python-six >= 1.9.0

It seems that %{python_module} does not handle this case.

Cheers,

Tom

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Thomas Bechtold]

On Tue, 2017-02-28 at 11:43 +0100, Thomas Bechtold wrote:
> Hi Jan,
>
> what should I do if I want to specify a minimum version for a
> BuildRequires? Something like:
>
> BuildRequires: python-six >= 1.9.0
>
> It seems that %{python_module} does not handle this case.

Seems to be already fixed in git and devel:languages:python . Just not
yet in Factory.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Thomas Bechtold]

Hi,

On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
> 7. If you are using update-alternatives, remove them.
> Instead mark the executable as %python3_only:
> %python3_only %{_bindir}/yourbinary
> In most cases, we only need one version of the executable. If the
> purpose of the tool is, e.g., reading EXIF metadata, we don't care if
> python 2 or 3 reads them. (And the library files are still installed
> for
> both.)

Why? What if I only want the py2 version (including only the py2
dependencies)? Having the "binary" only shipped in the py3 package
means that I need all python3 packages installed.

Best

Tom
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[jan matejek-5]

On 28.2.2017 12:00, Thomas Bechtold wrote:

> On Tue, 2017-02-28 at 11:43 +0100, Thomas Bechtold wrote:
>> Hi Jan,
>>
>> what should I do if I want to specify a minimum version for a
>> BuildRequires? Something like:
>>
>> BuildRequires: python-six >= 1.9.0
>>
>> It seems that %{python_module} does not handle this case.
>
> Seems to be already fixed in git and devel:languages:python . Just not
> yet in Factory.

the right way to spell requirements like this is %{python_module six >=
1.9.0}

there was a bug in the Factory definition that I only discovered
recently when a package broke :)
the Factory definition is already updated in prjconf so the resolver
should pull in the right dependencies; updated definition in
python-rpm-macros should appear in Factory soon.

and for the record, the standard compatibility shim should spell,
instead of:
%{?!python_module:%define python_module() python-%1 python3-%1}
should be:
%{?!python_module:%define python_module() python-%{**} python3-%{**}}

i'm in the process of putting all of this in the wiki

m.

>

[jan matejek-5]

On 28.2.2017 12:31, Thomas Bechtold wrote:

> Hi,
>
> On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
>> 7. If you are using update-alternatives, remove them.
>> Instead mark the executable as %python3_only:
>> %python3_only %{_bindir}/yourbinary
>> In most cases, we only need one version of the executable. If the
>> purpose of the tool is, e.g., reading EXIF metadata, we don't care if
>> python 2 or 3 reads them. (And the library files are still installed
>> for
>> both.)
>
> Why? What if I only want the py2 version (including only the py2
> dependencies)? Having the "binary" only shipped in the py3 package
> means that I need all python3 packages installed.

Why would you want "only the py2 version?"
if it's a tool for which python is an implementation detail, you *should
not care* which version of python it is using.

The point of this is gradually moving towards python3-as-default. This
should mean that after enough packages are converted, the typical
installation will only have a Python 3 stack, not a Python 2 one.

In the meantime, you will need both, which is impractical, but it seems
better than doing a big switch-flip and mass-changing every "python-*"
to "python3-*". Also IIRC this (having both stacks) is already the case
because of some stacks (KDE, also GNOME i think?) moving to python 3.


OTOH, If it actually matters which python version you're using, then of
course you keep both and optionally provide update-alternatives. That's
what the addendum to this point says.

>
> Best
>
> Tom
>

[Thomas Bechtold]

On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote:

> On 28.2.2017 12:31, Thomas Bechtold wrote:
> > Hi,
> >
> > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
> > > 7. If you are using update-alternatives, remove them.
> > > Instead mark the executable as %python3_only:
> > > %python3_only %{_bindir}/yourbinary
> > > In most cases, we only need one version of the executable. If the
> > > purpose of the tool is, e.g., reading EXIF metadata, we don't
> > > care if
> > > python 2 or 3 reads them. (And the library files are still
> > > installed
> > > for
> > > both.)
> >
> > Why? What if I only want the py2 version (including only the py2
> > dependencies)? Having the "binary" only shipped in the py3 package
> > means that I need all python3 packages installed.
>
> Why would you want "only the py2 version?"

There is a product called SUSE OpenStack Cloud and that uses only
python2 . And we don't want to add (and maintain) all the python3
versions for a huge dependency list just because some binaries need the
py3 version.

> if it's a tool for which python is an implementation detail, you
> *should
> not care* which version of python it is using.
>
> The point of this is gradually moving towards python3-as-default.
> This
> should mean that after enough packages are converted, the typical
> installation will only have a Python 3 stack, not a Python 2 one.
>
> In the meantime, you will need both, which is impractical, but it
> seems
> better than doing a big switch-flip and mass-changing every "python-
> *"
> to "python3-*". Also IIRC this (having both stacks) is already the
> case
> because of some stacks (KDE, also GNOME i think?) moving to python 3.
>
>
> OTOH, If it actually matters which python version you're using, then
> of
> course you keep both and optionally provide update-alternatives.
> That's
> what the addendum to this point says.

The question is what we want to recommend. I stumbled over this while I
 started to create a py2pack spec template for the single spec
approach.

Cheers,

Tom
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Neal Gompa]

On Tue, Feb 28, 2017 at 8:36 AM, Thomas Bechtold <[hidden email]> wrote:

> On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote:
>> On 28.2.2017 12:31, Thomas Bechtold wrote:
>> > Hi,
>> >
>> > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
>> > > 7. If you are using update-alternatives, remove them.
>> > > Instead mark the executable as %python3_only:
>> > > %python3_only %{_bindir}/yourbinary
>> > > In most cases, we only need one version of the executable. If the
>> > > purpose of the tool is, e.g., reading EXIF metadata, we don't
>> > > care if
>> > > python 2 or 3 reads them. (And the library files are still
>> > > installed
>> > > for
>> > > both.)
>> >
>> > Why? What if I only want the py2 version (including only the py2
>> > dependencies)? Having the "binary" only shipped in the py3 package
>> > means that I need all python3 packages installed.
>>
>> Why would you want "only the py2 version?"
>
> There is a product called SUSE OpenStack Cloud and that uses only
> python2 . And we don't want to add (and maintain) all the python3
> versions for a huge dependency list just because some binaries need the
> py3 version.
>

As far as I knew, OpenStack upstream is mostly done on Python 3
porting. So why not just move everything over?




--
真実はいつも一つ！/ Always, there's only one truth!
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Thomas Bechtold]

On Tue, 2017-02-28 at 08:38 -0500, Neal Gompa wrote:

> On Tue, Feb 28, 2017 at 8:36 AM, Thomas Bechtold <[hidden email]>
> wrote:
> > On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote:
> > > On 28.2.2017 12:31, Thomas Bechtold wrote:
> > > > Hi,
> > > >
> > > > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
> > > > > 7. If you are using update-alternatives, remove them.
> > > > > Instead mark the executable as %python3_only:
> > > > > %python3_only %{_bindir}/yourbinary
> > > > > In most cases, we only need one version of the executable. If
> > > > > the
> > > > > purpose of the tool is, e.g., reading EXIF metadata, we don't
> > > > > care if
> > > > > python 2 or 3 reads them. (And the library files are still
> > > > > installed
> > > > > for
> > > > > both.)
> > > >
> > > > Why? What if I only want the py2 version (including only the
> > > > py2
> > > > dependencies)? Having the "binary" only shipped in the py3
> > > > package
> > > > means that I need all python3 packages installed.
> > >
> > > Why would you want "only the py2 version?"
> >
> > There is a product called SUSE OpenStack Cloud and that uses only
> > python2 . And we don't want to add (and maintain) all the python3
> > versions for a huge dependency list just because some binaries need
> > the
> > py3 version.
> >
>
> As far as I knew, OpenStack upstream is mostly done on Python 3
> porting. So why not just move everything over?

"Mostly done", that's why. I'm pretty sure that not all OpenStack
projects we include have already python3 support.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[jan matejek-5]

On 28.2.2017 14:36, Thomas Bechtold wrote:
> There is a product called SUSE OpenStack Cloud and that uses only
> python2 . And we don't want to add (and maintain) all the python3
> versions for a huge dependency list just because some binaries need the
> py3 version.

(...)

> The question is what we want to recommend. I stumbled over this while I
>  started to create a py2pack spec template for the single spec
> approach.

well, the "recommended approach" so far has been to do alternatives for
every throwaway executable the package would install, and that seems
wrong to me. It's made worse by the fact that the vast majority of
packages install the executable with an unversioned name, so it's
additional work to either a) patch the setup.py file or b) catch the
executable after installation and rename it --- made worse with
singlespec by the fact that %python_install is now a single step.

So let's look at the possible ways to resolve this.

First, it is still perfectly possible to use alternatives, there's a
couple of rather helpful macros for that too. So if the list of "some
binaries" is small enough, we could keep what I recommended, and do
alternatives only for the packages in the list.

Then there's the fact that in a typical setuptools-based case, the
"binary" is the exact same script with a different shebang. I could
write a macro, i dunno, %python_clone_executable, that takes the named
executable and creates "executable-%{python_version}" for every flavor.

(I would still very much prefer the default, simplest case to be "only
put executable in one package". Even if the other option is simplified
to a macro and a couple of filelist entries.)

Then it's a question of whether we need *alternatives* or whether
shipping versioned binaries is good enough for your case? And if the
latter, I'd stay off alternatives where not required, and keep with
%python3_only %{_bindir}/unversioned-executable

m.

[Luigi Baldoni]

jan matejek-4 wrote

hello,

On 22.2.2017 06:08, Greg Freemyer wrote:
>
> I have 13 python- packages not in d:l:p.  Should I SR a copy to singlespec?

No. I'll be submitting all of singlespec to d:l:py at some point, this
would only cause confusion.

Let me know if you want me to look at the packages though.

Oh, i just submitted a few to d:l:p:singlespec, should I have sent them to plain d:l:p instead?

And also, since I've started playing with singlespec only today, could you please tell me
what I'm doing wrong here?

%ifpython2
BuildRequires:  %{python_module argparse}
%endif

OBS is still complaining about a missing python3-argparse.

Regards

[Luigi Baldoni]

Luigi Baldoni wrote

And also, since I've started playing with singlespec only today, could you please tell me
what I'm doing wrong here?

%ifpython2
BuildRequires:  %{python_module argparse}
%endif

OBS is still complaining about a missing python3-argparse.

I read the introduction messages more attentively and found the answer there.
Sorry, sometimes I jump the gun.

Anyway I have another question: is %ifpython2 only usable for Requires and the like?
I couldn't make it work for a generic command in %prep.

Regards

[Oliver Kurz-2]

I am trying to build some new package directly in the singlespec way and I
like the approach. To be able to find dependencies I added d:l:p and
d:l:p:singlespec to my project for now but hit the problem of
"have choice for…"
How can I ensure that one choice is selected and the package can build?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

[Olaf Hering-2]

Am Thu, 02 Mar 2017 00:18:47 +0100
schrieb Oliver Kurz <[hidden email]>:

> How can I ensure that one choice is selected and the package can
> build?

Add "Prefer: string" with "osc meta prjconf -e".

Olaf

[jan matejek-4]

On 1.3.2017 21:35, Luigi Baldoni wrote:
> Luigi Baldoni wrote
>> And also, since I've started playing with singlespec only today, could you
>> please tell me
>> what I'm doing wrong here?
>>
>> %ifpython2
>> BuildRequires:  %{python_module argparse}
>> %endif

for completeness:
instead of using %ifpython here, simply write:

BuildRequires: python-argparse

without conditionals and without %python_module macro

> I read the introduction messages more attentively and found the answer
> there.
> Sorry, sometimes I jump the gun.

:)

> Anyway I have another question: is %ifpython2 only usable for Requires and
> the like?
> I couldn't make it work for a generic command in %prep.

%ifpython2 and friends don't work in %prep/%build/%install/%check,
because these sections are common to all the subpackages.
IOW, the code in %prep and friends is only run once per build, and must
handle all pythons at once.

You can test for %if 0%{?have_python2}. But this will be true when
*python2 is one of the pythons for which we build*. It does not mean
that the section will only run for building the python2 version.

What are you trying to do? Can't you simply do it without the conditional?

m.

>
> Regards
>
>
>
>
> --
> View this message in context: http://opensuse.14.x6.nabble.com/python-singlespec-how-to-convert-your-package-tp5080741p5081843.html
> Sent from the opensuse-packaging mailing list archive at Nabble.com.
>

[Luigi Baldoni]

jan matejek-4 wrote

On 1.3.2017 21:35, Luigi Baldoni wrote:
> Luigi Baldoni wrote
>> And also, since I've started playing with singlespec only today, could you
>> please tell me
>> what I'm doing wrong here?
>>
>> %ifpython2
>> BuildRequires:  %{python_module argparse}
>> %endif

for completeness:
instead of using %ifpython here, simply write:

BuildRequires: python-argparse

without conditionals and without %python_module macro

And that did the trick.

> Anyway I have another question: is %ifpython2 only usable for Requires and
> the like?
> I couldn't make it work for a generic command in %prep.

%ifpython2 and friends don't work in %prep/%build/%install/%check,
because these sections are common to all the subpackages.
IOW, the code in %prep and friends is only run once per build, and must
handle all pythons at once.

In other words I need to put the logic into an external script or setup.py
itself to differentiate between py2 and py3, right?

What are you trying to do? Can't you simply do it without the conditional?

In the case in question, argparse doesn't exist for python3, so setup.py
fails setup_requires. I need a patch, I suppose.

Also, what about libraries that exist only for py2 or py3? Does it make
sense to use singlespec for those? (I failed when I tried)
Will we have to submit plain python3-foo to d:l:p once d:l:p3 is eliminated?

Regards

P.S: Should I keep submitting stuff to d:l:p:singlespec? If the conversion
script is around the corner I'll just wait.