New thoughts on python singlespec macros

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

New thoughts on python singlespec macros

todd rme
Thinking about what we need for python3 conversion long-term, a few
issues have come to mind that currently aren't addressed well in the
macros or previous discussions:

1. We really need to get rid of using unversioned "python" in shebangs.
2. We need to switch everything over to use versioned rpm names
(particularly "python2-foo"), while still supporting older SLE/Leap
versions.
3. We need to be able to handle builds that don't include python2.
4. pytest often checks the directories being used, which breaks when
using "%python_expand"

Regarding 1, there are currently two pain points. First and most
obvious is the fact that the shebangs have to be edit manually.  The
second is that if we want versioned shebangs, we need to manually
recompile the files to avoid mtime issues.

Ideally changing the shebangs could be handled automatically during
%python_build, but there could still be a separate macro that could be
called in cases where manual changes are necessary.  Recompiling is
needed in other situations as well, so having a macro to handle that
would be useful in general.

Regarding 2, this is more of a policy issue. I think all packages,
even python2-only ones, should be switched over to using singlespec.
This makes sure that they have the correct name on all platforms.
Backports packages should have a conditional "skip_python3" that is
set to disable building for python versions equal to or greater than
the version they were backported from, even when those versions are no
longer shipped by openSUSE. There should also be a policy that, when
an upstream package drops python2 support, the last-supported python2
version should be made available in the "python2-foo" package.  Only
"python-foo-doc" and "python-foo-data" can have the unversioned
"python-" name, and that is only for version-independent documentation
and assets, respectively.

Regarding 3, there are again two main pain points. The first is that
there is no macro I am aware of to reliable determine if a given
python version is being built.  %have_python2 and %have_python3 no
longer work reliably AFAIK.

The second is that build dependencies for particular python versions
cannot be reliably disabled if that python version is not built.
Currently most packages that have a python2-only build dependency just
unconditionally require it, which means it will be pulled in even if
python2 builds are disabled.  Ideally I would like to see
"%{python2_module foo}" and "%{python3_module foo}" that will only
pull in that dependency if that version of python is being used.  This
also has the advantage of not needing to care about
backwards-compatibility issues of "python-foo" vs. "python2-foo"
names, which is handled inconsistently right now.  If that is not
feasible, just having a reliable check would be an improvement.

Regarding 4, I don't really understand what "%python_expand" is doing
behind the scenes, and whether there is a simple workaround or whether
the macro needs to do something different internally.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

Jan Engelhardt-4

On Tuesday 2017-11-14 19:29, Todd Rme wrote:

>
>1. We really need to get rid of using unversioned "python" in shebangs.
>
>Regarding 1, there are currently two pain points. First and most
>obvious is the fact that the shebangs have to be edit manually.  The
>second is that if we want versioned shebangs, we need to manually
>recompile the files to avoid mtime issues.
>
>Ideally changing the shebangs could be handled automatically during
>%python_build, but there could still be a separate macro that could be
>called in cases where manual changes are necessary.  Recompiling is
>needed in other situations as well, so having a macro to handle that
>would be useful in general.

Aside from unversioned pythons in the #! line, "#!/usr/bin/env" has
also been made out to be a problem, so there is an opportunity here
to kill two birds with one stone.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

Henne Vogelsang-2
In reply to this post by todd rme
Hey,

On 14.11.2017 19:29, Todd Rme wrote:

> Thinking about what we need for python3 conversion long-term, a few
> issues have come to mind that currently aren't addressed well in the
> macros or previous discussions:
>
> 1. -> 4.
5. packages that include files outside of sitelib/arch. This fails
during verification because of the file conflicts.

Henne

--
Henne Vogelsang
http://www.opensuse.org
Everybody has a plan, until they get hit.
        - Mike Tyson
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

jan matejek-4
In reply to this post by todd rme
hello,

On 14.11.2017 19:29, Todd Rme wrote:
> Regarding 1, there are currently two pain points. First and most
> obvious is the fact that the shebangs have to be edit manually.  The
> second is that if we want versioned shebangs, we need to manually
> recompile the files to avoid mtime issues.

The second is sort of a "wrong point". Files that need shebangs are different from files that need
bytecode. Scripts that you directly execute don't create pyc files.
Of course, some upstreams put shebangs everywhere, and in rare cases you get a file that is both a
module (gets a pyc when used by others) and an executable.

But in general i wouldn't want to bend over backwards for this.

> Ideally changing the shebangs could be handled automatically during
> %python_build,

It is for files declared as executables. Distutils / setuptools do that.

> but there could still be a separate macro that could be
> called in cases where manual changes are necessary.  Recompiling is
> needed in other situations as well, so having a macro to handle that
> would be useful in general.

yes, we need $flavor_compile. I've had some blocking issues with it, i don't recall what they were...

Also yes, we need the shebang macros. I'll just, uh, go ahead and put them in python-rpm-macros.

> Regarding 3, there are again two main pain points. The first is that
> there is no macro I am aware of to reliable determine if a given
> python version is being built.  %have_python2 and %have_python3 no
> longer work reliably AFAIK.

Never worked in the first place ;) But yes.
I've been setting %bcond_without python2, then wrapping python2 parts in %if %{with python2}, same
for other flavors if necessary. This works even for buildrequires.
This is a possibly good convention. I'll look into if we can do the %with part without explicitly
setting the bcond in every package.

> python2 builds are disabled.  Ideally I would like to see
> "%{python2_module foo}" and "%{python3_module foo}" that will only
> pull in that dependency if that version of python is being used.  This
> also has the advantage of not needing to care about
> backwards-compatibility issues of "python-foo" vs. "python2-foo"
> names, which is handled inconsistently right now.  If that is not
> feasible, just having a reliable check would be an improvement.

The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of
%python2_module. We'd need to have something like %python2_buildrequires, but that sounds too
specific and impractical.
Having a reliable guard conditions, like the %with thing, is probably a better way.

> Regarding 4, I don't really understand what "%python_expand" is doing
> behind the scenes, and whether there is a simple workaround or whether
> the macro needs to do something different internally.

I suspect this might be better fixed by a patch to pytest....

(setuptools create a "build" directory and all python_expanding macros move this directory around to
"_build.$flavor", so that python is not reusing pycs from the wrong version. This should not be
necessary in theory, but in practice it is. pytest knows to ignore "build" directory, but doesn't
know about "_build". Probably easiest to teach pytest about its existence.)


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

todd rme
In reply to this post by Henne Vogelsang-2
On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang <[hidden email]> wrote:

> Hey,
>
> On 14.11.2017 19:29, Todd Rme wrote:
>
>> Thinking about what we need for python3 conversion long-term, a few
>> issues have come to mind that currently aren't addressed well in the
>> macros or previous discussions:
>>
>> 1. -> 4.
> 5. packages that include files outside of sitelib/arch. This fails
> during verification because of the file conflicts.
>
> Henne

Those are usually python version-independent.  That is the idea of
having a "python-foo-data" package convention for such files.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

todd rme
In reply to this post by jan matejek-4
On Wed, Nov 15, 2017 at 9:43 AM, jan matejek <[hidden email]> wrote:

> hello,
>
> On 14.11.2017 19:29, Todd Rme wrote:
>> Regarding 1, there are currently two pain points. First and most
>> obvious is the fact that the shebangs have to be edit manually.  The
>> second is that if we want versioned shebangs, we need to manually
>> recompile the files to avoid mtime issues.
>
> The second is sort of a "wrong point". Files that need shebangs are different from files that need
> bytecode. Scripts that you directly execute don't create pyc files.
> Of course, some upstreams put shebangs everywhere, and in rare cases you get a file that is both a
> module (gets a pyc when used by others) and an executable.
>
> But in general i wouldn't want to bend over backwards for this.

It is pretty common for tests.

>> python2 builds are disabled.  Ideally I would like to see
>> "%{python2_module foo}" and "%{python3_module foo}" that will only
>> pull in that dependency if that version of python is being used.  This
>> also has the advantage of not needing to care about
>> backwards-compatibility issues of "python-foo" vs. "python2-foo"
>> names, which is handled inconsistently right now.  If that is not
>> feasible, just having a reliable check would be an improvement.
>
> The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of
> %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too
> specific and impractical.

Would it be possible to replace disabled requires with some dummy
package that is a buildrequires by default anyway?


>> Regarding 4, I don't really understand what "%python_expand" is doing
>> behind the scenes, and whether there is a simple workaround or whether
>> the macro needs to do something different internally.
>
> I suspect this might be better fixed by a patch to pytest....
>
> (setuptools create a "build" directory and all python_expanding macros move this directory around to
> "_build.$flavor", so that python is not reusing pycs from the wrong version. This should not be
> necessary in theory, but in practice it is. pytest knows to ignore "build" directory, but doesn't
> know about "_build". Probably easiest to teach pytest about its existence.)

Can the macros move "_build.$flavor" to "build" during the
corresponding part of "python_expand"?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

Henne Vogelsang-2
In reply to this post by todd rme
Hey,

On 15.11.2017 16:33, Todd Rme wrote:

> On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang <[hidden email]> wrote:
>> On 14.11.2017 19:29, Todd Rme wrote:
>>
>>> Thinking about what we need for python3 conversion long-term, a few
>>> issues have come to mind that currently aren't addressed well in the
>>> macros or previous discussions:
>>>
>>> 1. -> 4.
>> 5. packages that include files outside of sitelib/arch. This fails
>> during verification because of the file conflicts.
>
> Those are usually python version-independent. That is the idea of
> having a "python-foo-data" package convention for such files.

And who decides which python*-foo package you use? :-)

Henne

--
Henne Vogelsang
http://www.opensuse.org
Everybody has a plan, until they get hit.
        - Mike Tyson
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

todd rme
On Wed, Nov 15, 2017 at 10:45 AM, Henne Vogelsang <[hidden email]> wrote:

> Hey,
>
> On 15.11.2017 16:33, Todd Rme wrote:
>> On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang <[hidden email]> wrote:
>>> On 14.11.2017 19:29, Todd Rme wrote:
>>>
>>>> Thinking about what we need for python3 conversion long-term, a few
>>>> issues have come to mind that currently aren't addressed well in the
>>>> macros or previous discussions:
>>>>
>>>> 1. -> 4.
>>> 5. packages that include files outside of sitelib/arch. This fails
>>> during verification because of the file conflicts.
>>
>> Those are usually python version-independent. That is the idea of
>> having a "python-foo-data" package convention for such files.
>
> And who decides which python*-foo package you use? :-)
>
> Henne

From all the examples I have seen the "python*-foo" packages require
the "python-foo-data" package (the name varies). If the files are
version-independent it doesn't matter which one they are pulled from.

In cases I have seen where they are version-dependent, they can be
renamed to use "%{$python_bin_suffix}".  That is typically the case
when .desktop files are involved, but those need to use
"%{$python_bin_suffix}" anyway.

The only case I am aware of where neither of these approaches work is
with jupyter notebook extensions, but since we only ship the python3
version of the notebook executable, we also only ship the python3
version of the notebook extensions.

Do you have a specific situation where none of these approaches work?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

jan matejek-4
In reply to this post by todd rme
On 15.11.2017 16:45, Todd Rme wrote:

> On Wed, Nov 15, 2017 at 9:43 AM, jan matejek <[hidden email]> wrote:
>>> python2 builds are disabled.  Ideally I would like to see
>>> "%{python2_module foo}" and "%{python3_module foo}" that will only
>>> pull in that dependency if that version of python is being used.  This
>>> also has the advantage of not needing to care about
>>> backwards-compatibility issues of "python-foo" vs. "python2-foo"
>>> names, which is handled inconsistently right now.  If that is not
>>> feasible, just having a reliable check would be an improvement.
>>
>> The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of
>> %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too
>> specific and impractical.
>
> Would it be possible to replace disabled requires with some dummy
> package that is a buildrequires by default anyway?
python-rpm-macros for instance? :)
That's certainly possible, although rather kludgey. It would solve the inconsistency. OTOH another
possible inconsistency is when python2 version requires something that doesn't start with "python-".
That's why I like guard conditions better.

> Can the macros move "_build.$flavor" to "build" during the
> corresponding part of "python_expand"?

That is precisely what happens. The problems you see exist because the *other* versions are lying
around. An alternate solution would be to hide the directories better. I'm not sure where though.

m.



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New thoughts on python singlespec macros

todd rme
On Wed, Nov 15, 2017 at 12:04 PM, jan matejek <[hidden email]> wrote:

> On 15.11.2017 16:45, Todd Rme wrote:
>> On Wed, Nov 15, 2017 at 9:43 AM, jan matejek <[hidden email]> wrote:
>>>> python2 builds are disabled.  Ideally I would like to see
>>>> "%{python2_module foo}" and "%{python3_module foo}" that will only
>>>> pull in that dependency if that version of python is being used.  This
>>>> also has the advantage of not needing to care about
>>>> backwards-compatibility issues of "python-foo" vs. "python2-foo"
>>>> names, which is handled inconsistently right now.  If that is not
>>>> feasible, just having a reliable check would be an improvement.
>>>
>>> The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of
>>> %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too
>>> specific and impractical.
>>
>> Would it be possible to replace disabled requires with some dummy
>> package that is a buildrequires by default anyway?
>
> python-rpm-macros for instance? :)
> That's certainly possible, although rather kludgey. It would solve the inconsistency. OTOH another
> possible inconsistency is when python2 version requires something that doesn't start with "python-".
> That's why I like guard conditions better.

Yes, the guard conditions would still be needed.

The reason I like the approach is because it fits well with the
existing pattern, where "%python_foo" is the multi-python version,
"%python2_foo" is the python2 version, and "%python3_foo".  There are
only a couple cases where this pattern doesn't hold.

>> Can the macros move "_build.$flavor" to "build" during the
>> corresponding part of "python_expand"?
>
> That is precisely what happens. The problems you see exist because the *other* versions are lying
> around. An alternate solution would be to hide the directories better. I'm not sure where though.

What about moving it into buildroot?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]