Some more python singlespec questions/suggestions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Some more python singlespec questions/suggestions

todd rme
A few questions and ideas have popped up for me while doing the
singlespec conversion:

First, is there a way to use the macros without actually setting up
the multiple packages?  Particularly I need to use %{python_module
foo}, but %python_expand and %python_exec would also be useful.  The
main use-case is for packages where the unit tests have been moved
into a separate -doc spec file to avoid dependency cycles.  These
packages need to import both python2 and python3 dependencies to for
the unit tests, and ideally would use %python_expand or %python_exec
to run the tests.  But the documentation itself would only be built
once, so we don't need multiple rpms.  I tried %_python_macro_init but
it didn't work.

Second, is there a single-spec version of %py(3)_compile?  Ideally it
would be nice to have %python2_compile and %python3_compile that could
be used with %python_expand, as well as a macro that automatically run
them both on %{buildroot}%{python(2/3)_sitelib} and
%{buildroot}%{python(2/3)_sitearch} (whichever exists).

Third, is there a macros to get just the current python major version
number?  So "2" for python2 and "3" for python3?  So projects, such as
the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to
support, so having a macro to get the major number is pretty critical
especially since IPython is dropping python2 support soon so we need
to be able to adjust the build targets easily.

Third, the wiki recommends defining %oldpython when you need a
specific python-foo dependency.  Considering how often the backports
are used these days in python2 packages, might this be a macro that
would be good to include so people don't have to manually define it?

Fourth, I notice the python2-foo versions of the packages (for
Tumbleweed) provide python-foo, but the python-foo versions (for older
releases) do not provide python2-foo.  For maximum simplicity and
compatibility, might it be good to have the python-foo versions also
provide python2-foo?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Some more python singlespec questions/suggestions

jan matejek-4
On 10.4.2017 17:09, Todd Rme wrote:

> A few questions and ideas have popped up for me while doing the
> singlespec conversion:
>
> First, is there a way to use the macros without actually setting up
> the multiple packages?  Particularly I need to use %{python_module
> foo}, but %python_expand and %python_exec would also be useful.  The
> main use-case is for packages where the unit tests have been moved
> into a separate -doc spec file to avoid dependency cycles.  These
> packages need to import both python2 and python3 dependencies to for
> the unit tests, and ideally would use %python_expand or %python_exec
> to run the tests.  But the documentation itself would only be built
> once, so we don't need multiple rpms.  I tried %_python_macro_init but
> it didn't work.
AFAIK the macros you list work fine by themselves.
(of course, you still need to require python-rpm-macros; and if you are using %python_module in
BuildRequires, you still need to shim the definition)
If you're seeing failures, show me :)

The only macros whose presence depend on each other are %python_subpackages and %python_files.
Either will cause an error if the other is not used.

> Second, is there a single-spec version of %py(3)_compile?  Ideally it
> would be nice to have %python2_compile and %python3_compile that could
> be used with %python_expand, as well as a macro that automatically run
> them both on %{buildroot}%{python(2/3)_sitelib} and
> %{buildroot}%{python(2/3)_sitearch} (whichever exists).

Not as of now; so far the singlespec macros were geared towards setuptools-based installations where
you don't use %py_compile.
We can add it.

> Third, is there a macros to get just the current python major version
> number?  So "2" for python2 and "3" for python3?  So projects, such as
> the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to
> support, so having a macro to get the major number is pretty critical
> especially since IPython is dropping python2 support soon so we need
> to be able to adjust the build targets easily.

what will happen to such packages when they encounter pypy3?

In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on
"major version of python", as opposed to "is this python2 or something newer", does not look like a
good practice.
(and if it's limited to one package, you can easily get, e.g. from %python_version_nodots)
OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?

> Third, the wiki recommends defining %oldpython when you need a
> specific python-foo dependency.  Considering how often the backports
> are used these days in python2 packages, might this be a macro that
> would be good to include so people don't have to manually define it?

If a backport doesn't provide python2-name, it's a bug and should be fixed.

As for adding %oldpython to the macro set for other cases (obsoletes/provides usually), maybe?
(Or maybe find a different spelling; %oldpython is good because it is easy to fit it into any spec
file, but it is a hack. Something like "Provides: %{literally python-foo}" would be nicer)

>
> Fourth, I notice the python2-foo versions of the packages (for
> Tumbleweed) provide python-foo, but the python-foo versions (for older
> releases) do not provide python2-foo.  For maximum simplicity and
> compatibility, might it be good to have the python-foo versions also
> provide python2-foo?

Not as easy as it looks, unfortunately .e( I can add auto-provides to the main package, but I can't
touch subpackages. That way the "maximum simplicity and compatibility" is a half-promise.
However it seems that practicality does beat purity here, as most packages don't have subpackages
anyway, so I guess i'll put it on my TODO list.

regards
m.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Some more python singlespec questions/suggestions

todd rme
On Wed, Apr 12, 2017 at 8:39 AM, jan matejek <[hidden email]> wrote:

> On 10.4.2017 17:09, Todd Rme wrote:
>> A few questions and ideas have popped up for me while doing the
>> singlespec conversion:
>>
>> First, is there a way to use the macros without actually setting up
>> the multiple packages?  Particularly I need to use %{python_module
>> foo}, but %python_expand and %python_exec would also be useful.  The
>> main use-case is for packages where the unit tests have been moved
>> into a separate -doc spec file to avoid dependency cycles.  These
>> packages need to import both python2 and python3 dependencies to for
>> the unit tests, and ideally would use %python_expand or %python_exec
>> to run the tests.  But the documentation itself would only be built
>> once, so we don't need multiple rpms.  I tried %_python_macro_init but
>> it didn't work.
>
> AFAIK the macros you list work fine by themselves.
> (of course, you still need to require python-rpm-macros; and if you are using %python_module in
> BuildRequires, you still need to shim the definition)
> If you're seeing failures, show me :)

Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:languages:python/python-testtools-doc

I get errors like this outside Tumbleweed: "nothing provides
%{python_module, nothing provides testtools = %{version}}"

>
>> Second, is there a single-spec version of %py(3)_compile?  Ideally it
>> would be nice to have %python2_compile and %python3_compile that could
>> be used with %python_expand, as well as a macro that automatically run
>> them both on %{buildroot}%{python(2/3)_sitelib} and
>> %{buildroot}%{python(2/3)_sitearch} (whichever exists).
>
> Not as of now; so far the singlespec macros were geared towards setuptools-based installations where
> you don't use %py_compile.
> We can add it.

Even in setuptools, you sometimes need to run %py_compile due to, for
example, "inconsistent mtime" rpmlint warnings.

But more and more packages are shipping wheels only, so having support
for %py_compile, and less importantly "pip install", is becoming
important.

>> Third, is there a macros to get just the current python major version
>> number?  So "2" for python2 and "3" for python3?  So projects, such as
>> the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to
>> support, so having a macro to get the major number is pretty critical
>> especially since IPython is dropping python2 support soon so we need
>> to be able to adjust the build targets easily.
>
> what will happen to such packages when they encounter pypy3?

No idea, that is up to the upstream project to decide.

> In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on
> "major version of python", as opposed to "is this python2 or something newer", does not look like a
> good practice.
> (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots)
> OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?

I agree with you, but this is an upstream decision.

>> Third, the wiki recommends defining %oldpython when you need a
>> specific python-foo dependency.  Considering how often the backports
>> are used these days in python2 packages, might this be a macro that
>> would be good to include so people don't have to manually define it?
>
> If a backport doesn't provide python2-name, it's a bug and should be fixed.
>
> As for adding %oldpython to the macro set for other cases (obsoletes/provides usually), maybe?
> (Or maybe find a different spelling; %oldpython is good because it is easy to fit it into any spec
> file, but it is a hack. Something like "Provides: %{literally python-foo}" would be nicer)

I think that whatever is recommended on the wiki should be supported
out-of-the-box.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Some more python singlespec questions/suggestions

jan matejek-4
On 12.4.2017 17:01, Todd Rme wrote:
> Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:languages:python/python-testtools-doc
>
> I get errors like this outside Tumbleweed: "nothing provides
> %{python_module, nothing provides testtools = %{version}}"

You're missing this line:
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
Because it's used in BuildRequires, it must be either in prjconf (in Factory) or in the spec file.

> Even in setuptools, you sometimes need to run %py_compile due to, for
> example, "inconsistent mtime" rpmlint warnings.

If you get "inconsistent mtime" from setup.py install, it's either the installer's or your packaging
bug. Usually you're running tests that touch things they shouldn't.

(But yes, %py_compiling it again is often easier than fixing it properly.)

>
> But more and more packages are shipping wheels only, so having support
> for %py_compile, and less importantly "pip install", is becoming
> important.

wouldn't "pip install" take care of it too?

>> In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on
>> "major version of python", as opposed to "is this python2 or something newer", does not look like a
>> good practice.
>> (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots)
>> OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
>
> I agree with you, but this is an upstream decision.

And also easily resolvable inside the spec file.
The question here is whether this should be directly supported by a singlespec macro.

One more point against it that it's not general: for instance, in LXML, you run "make test" for
python2 tests and "make test3" for python3. Here you would need a variable that is "" for python2
and "3" for python3. In other cases, you need "2" for python2 and "3" for python3. Next time it
might be "2" for python2, "pp" for PyPy and "" for python3?

...and as long as you're if'ing it anyway, you can do this:
if [ %{$python_version_nodots} -ge 30 ]; then
 mysuffix=3
else
 mysuffix=2
fi

or even:
mysuffix=`expr %{$python_version_nodots} / 10`

m.



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Some more python singlespec questions/suggestions

todd rme
On Thu, Apr 13, 2017 at 9:06 AM, jan matejek <[hidden email]> wrote:
> On 12.4.2017 17:01, Todd Rme wrote:
>> Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:languages:python/python-testtools-doc
>>
>> I get errors like this outside Tumbleweed: "nothing provides
>> %{python_module, nothing provides testtools = %{version}}"
>
> You're missing this line:
> %{?!python_module:%define python_module() python-%{**} python3-%{**}}
> Because it's used in BuildRequires, it must be either in prjconf (in Factory) or in the spec file.

Next time I will wait until after my 101 degree fever is over before
asking ;) Sorry.

>> Even in setuptools, you sometimes need to run %py_compile due to, for
>> example, "inconsistent mtime" rpmlint warnings.
>
> If you get "inconsistent mtime" from setup.py install, it's either the installer's or your packaging
> bug. Usually you're running tests that touch things they shouldn't.
>
> (But yes, %py_compiling it again is often easier than fixing it properly.)

I have seen it in other situations as well, and it isn't necessarily
easy to track down.

>> But more and more packages are shipping wheels only, so having support
>> for %py_compile, and less importantly "pip install", is becoming
>> important.
>
> wouldn't "pip install" take care of it too?

Once pypi starts providing consistent URLs for wheels we can do that
when they are provided.

But even then there are packages with their own home-made build
systems that still need to be byte-compiled manually.

>>> In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on
>>> "major version of python", as opposed to "is this python2 or something newer", does not look like a
>>> good practice.
>>> (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots)
>>> OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
>>
>> I agree with you, but this is an upstream decision.
>
> And also easily resolvable inside the spec file.
> The question here is whether this should be directly supported by a singlespec macro.
>
> One more point against it that it's not general: for instance, in LXML, you run "make test" for
> python2 tests and "make test3" for python3. Here you would need a variable that is "" for python2
> and "3" for python3. In other cases, you need "2" for python2 and "3" for python3. Next time it
> might be "2" for python2, "pp" for PyPy and "" for python3?
>
> ...and as long as you're if'ing it anyway, you can do this:
> if [ %{$python_version_nodots} -ge 30 ]; then
>  mysuffix=3
> else
>  mysuffix=2
> fi
>
> or even:
> mysuffix=`expr %{$python_version_nodots} / 10`

I agree that it isn't a desirable situation, but if we want our users
to be able to use upstream documentation then we need to support
upstream naming conventions. We can create links with our own naming
patterns, but if upstream provides executable with certain names we
need to have those executables as well. And I think this is a common
enough upstream naming pattern that there should be support for it.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]