Dear packagers,
this is an overview of how singlespec works and how to use it for your package. First, some key concepts: * It will allow you to build Python 2 and Python 3 (and PyPy 3 in the future) package from a single spec file, with very little manual effort. This means that if you're maintaining a separate python3-something in d:l:py3, you can now switch over to the common python-something in d:l:py. * It is designed to work on specs called "python-something". There is some support for "python2-something" and "python3-something", but it is mostly untested. If your spec file name doesn't start with "python", don't even try. (Yet.) * (if you're building for Factory,) all your dependencies must be converted first. That's because the macros will BuildRequire "python2-things", which only exist for packages that are already converted. The devel:languages:python:singlespec already has basic packages, and not all of them will be in d:l:py immediately, so if you want to play now, build against this project. I'll also be accepting SRs into it. * In general, if you're unsure, please submit to d:l:py:singlespec first. Also examine linkdiffs of packages in d:l:py:singlespec to get an idea of how it can look like. https://build.opensuse.org/project/show/devel:languages:python:singlespec * Individual macros are documented on github https://github.com/openSUSE/python-rpm-macros * I'm working on an autoconverter that performs most of the following steps automatically. I intend to use it to convert most of d:l:py into d:l:py:singlespec:staging, and then review and submit packages one by one. So maybe if you wait, you can save yourself some work ;) Conversion steps: 0. If you want to build for anything other than Factory, you need to include compatibility shims. This definition of the %python_module macro goes on top of your spec: %{?!python_module:%define python_module() python-%1 python3-%1} and a buildrequire: BuildRequires: python-rpm-macros 1. Wrap all your "BuildRequires: python-something": BuildRequires: %{python_module something} Do not do this in Requires or any other fields. If you have a buildrequire that should only apply to one python, say, python-enum34 only for python 2, do not wrap it. 2. At the end of the spec preamble, before %description, place the macro: %python_subpackages on a line by itself 3. Replace "python setup.py build" with %python_build and "python setup.py install" with %python_install. These already contain the usual arguments (--prefix, --root), so usually you don't need to add any. If you configure CFLAGS or something like that, export them first: CFLAGS="-fwrapv" python setup.py build -> export CFLAGS="-fwrapv" %python_build If you are running any other commands based on python, use %python_exec. "python setup.py test" -> "%python_exec setup.py test" This works for usual test runners too: "nosetests" -> "%python_exec %{_bindir}/nosetests" 4. If you're doing something more complicated ... say, removing files from %python2_sitelib and %python3_sitelib, use %python_expand. "%python_expand rm -f %{$python_sitelib}/foo/undesirable.txt" Yes, that is "%$python_sitelib". Beware. If you put only "%python_sitelib", it will not work, because "%python_sitelib" gets expanded before %python_expand touches it. In general, after %python_expand, "$python" is replaced by all pythons even in macro names. If you need more lines, wrap in braces: %{python_expand firstline secondline $python thirdline} Note that the first line must not be empty. 5. change "%files" to "%files %{python_files}" If you have subpackages, say "%files foo", then change to "%files %{python_files foo}". 6. If some files are only in one python version, mark them as %python2_only or %python3_only in the filelist. If there's more of them, you can use %ifpython2 ... %endif If the entry in question is a __pycache__ directory, use %pycache_only or %ifpycache. 7. If you are using update-alternatives, remove them. Instead mark the executable as %python3_only: %python3_only %{_bindir}/yourbinary In most cases, we only need one version of the executable. If the purpose of the tool is, e.g., reading EXIF metadata, we don't care if python 2 or 3 reads them. (And the library files are still installed for both.) (If you are sure that you need executables for all python versions, look at what python-bottle does: https://build.opensuse.org/package/view_file/devel:languages:python:singlespec/python-bottle/python-bottle.spec?expand=1 note the installation inside %python_expand. Usually, packages don't install versioned executables by themselves, so you need to do it manually. If your package does, good for you.) 8. Take the time to enable automatic tests :) have a nice weekend m. |
On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote:
> > * Individual macros are documented on github > https://github.com/openSUSE/python-rpm-macros As this now becomes 'law' for the openSUSE packaging: can you please ensure that the openSUSE wiki reflects the method of singlespec, giving guidance on the whole thing? This is quite important for users as well as for the review team to have clear info on what must be, what should be and what must not be present. Thanks, Dominique |
In reply to this post by jan matejek-4
On Fri, Feb 17, 2017 at 12:31 PM, jan matejek <[hidden email]> wrote:
> Dear packagers, > > this is an overview of how singlespec works and how to use it for your > package. First, some key concepts: > > * It will allow you to build Python 2 and Python 3 (and PyPy 3 in the > future) package from a single spec file, with very little manual effort. > This means that if you're maintaining a separate python3-something in > d:l:py3, you can now switch over to the common python-something in d:l:py. > > * It is designed to work on specs called "python-something". There is > some support for "python2-something" and "python3-something", but it is > mostly untested. > If your spec file name doesn't start with "python", don't even try. > (Yet.) > > * (if you're building for Factory,) all your dependencies must be > converted first. > That's because the macros will BuildRequire "python2-things", which only > exist for packages that are already converted. > The devel:languages:python:singlespec already has basic packages, and > not all of them will be in d:l:py immediately, so if you want to play > now, build against this project. I'll also be accepting SRs into it. I have 13 python- packages not in d:l:p. Should I SR a copy to singlespec? I also have a bunch of non-python packages that have python- sub-packages for python bindings. Should I just ensure manually that they build python2- and python3- sub-packages? I'm the only consumer of these python binding sub-packages. Should I just drop python- versions of the sub-packages? Thanks Greg -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
hello,
On 22.2.2017 06:08, Greg Freemyer wrote: > > I have 13 python- packages not in d:l:p. Should I SR a copy to singlespec? No. I'll be submitting all of singlespec to d:l:py at some point, this would only cause confusion. Let me know if you want me to look at the packages though. > > I also have a bunch of non-python packages that have python- > sub-packages for python bindings. Should I just ensure manually that > they build python2- and python3- sub-packages? At the moment, old-style packages should work fine, so I suggest leaving it as-is, until there is singlespec support for packages like this. (it's the next thing on my todo-list once the transition is on its way) You can rename python- bindings to python2-, but this will probably only be acceptable for Factory. > I'm the only consumer of these python binding sub-packages. Should I > just drop python- versions of the sub-packages? I'm not sure I understand the question. Are you asking if you should only build Python 3 version? Probably not -- you might be the only consumer now, but why not have both (or all) pythons supported. Or are you asking about moving from python- to python2- ? Again, that is up to you. > > Thanks > Greg > -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
On Wed, Feb 22, 2017 at 8:17 AM, jan matejek <[hidden email]> wrote:
> Or are you asking about moving from python- to python2- ? Again, that is > up to you Thanks, then I will leave them as they are. Greg -- Greg Freemyer -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
In reply to this post by jan matejek-4
Hi Jan,
what should I do if I want to specify a minimum version for a BuildRequires? Something like: BuildRequires: python-six >= 1.9.0 It seems that %{python_module} does not handle this case. Cheers, Tom -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
On Tue, 2017-02-28 at 11:43 +0100, Thomas Bechtold wrote:
> Hi Jan, > > what should I do if I want to specify a minimum version for a > BuildRequires? Something like: > > BuildRequires: python-six >= 1.9.0 > > It seems that %{python_module} does not handle this case. Seems to be already fixed in git and devel:languages:python . Just not yet in Factory. -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
In reply to this post by jan matejek-4
Hi,
On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote: > 7. If you are using update-alternatives, remove them. > Instead mark the executable as %python3_only: > %python3_only %{_bindir}/yourbinary > In most cases, we only need one version of the executable. If the > purpose of the tool is, e.g., reading EXIF metadata, we don't care if > python 2 or 3 reads them. (And the library files are still installed > for > both.) Why? What if I only want the py2 version (including only the py2 dependencies)? Having the "binary" only shipped in the py3 package means that I need all python3 packages installed. Best Tom -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
In reply to this post by Thomas Bechtold
On 28.2.2017 12:00, Thomas Bechtold wrote:
> On Tue, 2017-02-28 at 11:43 +0100, Thomas Bechtold wrote: >> Hi Jan, >> >> what should I do if I want to specify a minimum version for a >> BuildRequires? Something like: >> >> BuildRequires: python-six >= 1.9.0 >> >> It seems that %{python_module} does not handle this case. > > Seems to be already fixed in git and devel:languages:python . Just not > yet in Factory. 1.9.0} there was a bug in the Factory definition that I only discovered recently when a package broke :) the Factory definition is already updated in prjconf so the resolver should pull in the right dependencies; updated definition in python-rpm-macros should appear in Factory soon. and for the record, the standard compatibility shim should spell, instead of: %{?!python_module:%define python_module() python-%1 python3-%1} should be: %{?!python_module:%define python_module() python-%{**} python3-%{**}} i'm in the process of putting all of this in the wiki m. > |
In reply to this post by Thomas Bechtold
On 28.2.2017 12:31, Thomas Bechtold wrote:
> Hi, > > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote: >> 7. If you are using update-alternatives, remove them. >> Instead mark the executable as %python3_only: >> %python3_only %{_bindir}/yourbinary >> In most cases, we only need one version of the executable. If the >> purpose of the tool is, e.g., reading EXIF metadata, we don't care if >> python 2 or 3 reads them. (And the library files are still installed >> for >> both.) > > Why? What if I only want the py2 version (including only the py2 > dependencies)? Having the "binary" only shipped in the py3 package > means that I need all python3 packages installed. if it's a tool for which python is an implementation detail, you *should not care* which version of python it is using. The point of this is gradually moving towards python3-as-default. This should mean that after enough packages are converted, the typical installation will only have a Python 3 stack, not a Python 2 one. In the meantime, you will need both, which is impractical, but it seems better than doing a big switch-flip and mass-changing every "python-*" to "python3-*". Also IIRC this (having both stacks) is already the case because of some stacks (KDE, also GNOME i think?) moving to python 3. OTOH, If it actually matters which python version you're using, then of course you keep both and optionally provide update-alternatives. That's what the addendum to this point says. > > Best > > Tom > |
On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote:
> On 28.2.2017 12:31, Thomas Bechtold wrote: > > Hi, > > > > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote: > > > 7. If you are using update-alternatives, remove them. > > > Instead mark the executable as %python3_only: > > > %python3_only %{_bindir}/yourbinary > > > In most cases, we only need one version of the executable. If the > > > purpose of the tool is, e.g., reading EXIF metadata, we don't > > > care if > > > python 2 or 3 reads them. (And the library files are still > > > installed > > > for > > > both.) > > > > Why? What if I only want the py2 version (including only the py2 > > dependencies)? Having the "binary" only shipped in the py3 package > > means that I need all python3 packages installed. > > Why would you want "only the py2 version?" There is a product called SUSE OpenStack Cloud and that uses only python2 . And we don't want to add (and maintain) all the python3 versions for a huge dependency list just because some binaries need the py3 version. > if it's a tool for which python is an implementation detail, you > *should > not care* which version of python it is using. > > The point of this is gradually moving towards python3-as-default. > This > should mean that after enough packages are converted, the typical > installation will only have a Python 3 stack, not a Python 2 one. > > In the meantime, you will need both, which is impractical, but it > seems > better than doing a big switch-flip and mass-changing every "python- > *" > to "python3-*". Also IIRC this (having both stacks) is already the > case > because of some stacks (KDE, also GNOME i think?) moving to python 3. > > > OTOH, If it actually matters which python version you're using, then > of > course you keep both and optionally provide update-alternatives. > That's > what the addendum to this point says. The question is what we want to recommend. I stumbled over this while I started to create a py2pack spec template for the single spec approach. Cheers, Tom -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
On Tue, Feb 28, 2017 at 8:36 AM, Thomas Bechtold <[hidden email]> wrote:
> On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote: >> On 28.2.2017 12:31, Thomas Bechtold wrote: >> > Hi, >> > >> > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote: >> > > 7. If you are using update-alternatives, remove them. >> > > Instead mark the executable as %python3_only: >> > > %python3_only %{_bindir}/yourbinary >> > > In most cases, we only need one version of the executable. If the >> > > purpose of the tool is, e.g., reading EXIF metadata, we don't >> > > care if >> > > python 2 or 3 reads them. (And the library files are still >> > > installed >> > > for >> > > both.) >> > >> > Why? What if I only want the py2 version (including only the py2 >> > dependencies)? Having the "binary" only shipped in the py3 package >> > means that I need all python3 packages installed. >> >> Why would you want "only the py2 version?" > > There is a product called SUSE OpenStack Cloud and that uses only > python2 . And we don't want to add (and maintain) all the python3 > versions for a huge dependency list just because some binaries need the > py3 version. > As far as I knew, OpenStack upstream is mostly done on Python 3 porting. So why not just move everything over? -- 真実はいつも一つ!/ Always, there's only one truth! -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
On Tue, 2017-02-28 at 08:38 -0500, Neal Gompa wrote:
> On Tue, Feb 28, 2017 at 8:36 AM, Thomas Bechtold <[hidden email]> > wrote: > > On Tue, 2017-02-28 at 14:23 +0100, jan matejek wrote: > > > On 28.2.2017 12:31, Thomas Bechtold wrote: > > > > Hi, > > > > > > > > On Fri, 2017-02-17 at 18:31 +0100, jan matejek wrote: > > > > > 7. If you are using update-alternatives, remove them. > > > > > Instead mark the executable as %python3_only: > > > > > %python3_only %{_bindir}/yourbinary > > > > > In most cases, we only need one version of the executable. If > > > > > the > > > > > purpose of the tool is, e.g., reading EXIF metadata, we don't > > > > > care if > > > > > python 2 or 3 reads them. (And the library files are still > > > > > installed > > > > > for > > > > > both.) > > > > > > > > Why? What if I only want the py2 version (including only the > > > > py2 > > > > dependencies)? Having the "binary" only shipped in the py3 > > > > package > > > > means that I need all python3 packages installed. > > > > > > Why would you want "only the py2 version?" > > > > There is a product called SUSE OpenStack Cloud and that uses only > > python2 . And we don't want to add (and maintain) all the python3 > > versions for a huge dependency list just because some binaries need > > the > > py3 version. > > > > As far as I knew, OpenStack upstream is mostly done on Python 3 > porting. So why not just move everything over? "Mostly done", that's why. I'm pretty sure that not all OpenStack projects we include have already python3 support. -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
In reply to this post by Thomas Bechtold
On 28.2.2017 14:36, Thomas Bechtold wrote: > There is a product called SUSE OpenStack Cloud and that uses only > python2 . And we don't want to add (and maintain) all the python3 > versions for a huge dependency list just because some binaries need the > py3 version. (...) > The question is what we want to recommend. I stumbled over this while I > started to create a py2pack spec template for the single spec > approach. well, the "recommended approach" so far has been to do alternatives for every throwaway executable the package would install, and that seems wrong to me. It's made worse by the fact that the vast majority of packages install the executable with an unversioned name, so it's additional work to either a) patch the setup.py file or b) catch the executable after installation and rename it --- made worse with singlespec by the fact that %python_install is now a single step. So let's look at the possible ways to resolve this. First, it is still perfectly possible to use alternatives, there's a couple of rather helpful macros for that too. So if the list of "some binaries" is small enough, we could keep what I recommended, and do alternatives only for the packages in the list. Then there's the fact that in a typical setuptools-based case, the "binary" is the exact same script with a different shebang. I could write a macro, i dunno, %python_clone_executable, that takes the named executable and creates "executable-%{python_version}" for every flavor. (I would still very much prefer the default, simplest case to be "only put executable in one package". Even if the other option is simplified to a macro and a couple of filelist entries.) Then it's a question of whether we need *alternatives* or whether shipping versioned binaries is good enough for your case? And if the latter, I'd stay off alternatives where not required, and keep with %python3_only %{_bindir}/unversioned-executable m. |
In reply to this post by jan matejek-4
Oh, i just submitted a few to d:l:p:singlespec, should I have sent them to plain d:l:p instead? And also, since I've started playing with singlespec only today, could you please tell me what I'm doing wrong here? %ifpython2 BuildRequires: %{python_module argparse} %endif OBS is still complaining about a missing python3-argparse. Regards |
I read the introduction messages more attentively and found the answer there. Sorry, sometimes I jump the gun. Anyway I have another question: is %ifpython2 only usable for Requires and the like? I couldn't make it work for a generic command in %prep. Regards |
In reply to this post by jan matejek-5
I am trying to build some new package directly in the singlespec way and I
like the approach. To be able to find dependencies I added d:l:p and d:l:p:singlespec to my project for now but hit the problem of "have choice for…" How can I ensure that one choice is selected and the package can build? -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
Am Thu, 02 Mar 2017 00:18:47 +0100
schrieb Oliver Kurz <[hidden email]>: > How can I ensure that one choice is selected and the package can > build? Add "Prefer: string" with "osc meta prjconf -e". Olaf |
In reply to this post by Luigi Baldoni
On 1.3.2017 21:35, Luigi Baldoni wrote:
> Luigi Baldoni wrote >> And also, since I've started playing with singlespec only today, could you >> please tell me >> what I'm doing wrong here? >> >> %ifpython2 >> BuildRequires: %{python_module argparse} >> %endif for completeness: instead of using %ifpython here, simply write: BuildRequires: python-argparse without conditionals and without %python_module macro > I read the introduction messages more attentively and found the answer > there. > Sorry, sometimes I jump the gun. :) > Anyway I have another question: is %ifpython2 only usable for Requires and > the like? > I couldn't make it work for a generic command in %prep. %ifpython2 and friends don't work in %prep/%build/%install/%check, because these sections are common to all the subpackages. IOW, the code in %prep and friends is only run once per build, and must handle all pythons at once. You can test for %if 0%{?have_python2}. But this will be true when *python2 is one of the pythons for which we build*. It does not mean that the section will only run for building the python2 version. What are you trying to do? Can't you simply do it without the conditional? m. > > Regards > > > > > -- > View this message in context: http://opensuse.14.x6.nabble.com/python-singlespec-how-to-convert-your-package-tp5080741p5081843.html > Sent from the opensuse-packaging mailing list archive at Nabble.com. > |
And that did the trick. In other words I need to put the logic into an external script or setup.py itself to differentiate between py2 and py3, right? In the case in question, argparse doesn't exist for python3, so setup.py fails setup_requires. I need a patch, I suppose. Also, what about libraries that exist only for py2 or py3? Does it make sense to use singlespec for those? (I failed when I tried) Will we have to submit plain python3-foo to d:l:p once d:l:p3 is eliminated? Regards P.S: Should I keep submitting stuff to d:l:p:singlespec? If the conversion script is around the corner I'll just wait. |
Free forum by Nabble | Edit this page |