Automatic Python Provides

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Automatic Python Provides

todd rme
Fedora is implementing a system to automatically add reliable,
distro-agnostic provides to their python packages [1].  This
apparently originally came from Mandriva and Mageia, according to the
mailing list discussion [2].  They are also adding macros to make
these easier to use, so you can use the macros to depend on the right
provides automatically.

This seems like something that would make it much easier to handle
python packages in a distro-agnostic manner.  The downside is it
requires rebuilding all python packages to work, which may limit its
applicably to previous openSUSE/SLE releases.

What does everything think about implementing this on our end as well?
 I am not exactly clear from the discussion what exactly it would
entail.

[1] https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages
[2] https://lists.fedoraproject.org/archives/list/python-devel@.../thread/UZ5IZFW7AZVIU3XH6EX4262NVVJRJZUZ/
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

jan matejek-4
hello,

On 17.8.2016 18:09, Todd Rme wrote:
> Fedora is implementing a system to automatically add reliable,
> distro-agnostic provides to their python packages [1].  This
(...)
> What does everything think about implementing this on our end as well?

yes, let's please do this. :)

>  I am not exactly clear from the discussion what exactly it would
> entail.

We'd need to extend the autoreqprov generator in our RPM. There is
already some python-related code, IIRC, something that can generate
provides from files/directories in site-packages directory. It would be
beneficial to have both: the canonical name from pypi, what fedora is
doing now, and also provides for imports -- even though the latter would
never be perfect.

Some RPM person please speak up?

m.

>
> [1] https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages
> [2] https://lists.fedoraproject.org/archives/list/python-devel@.../thread/UZ5IZFW7AZVIU3XH6EX4262NVVJRJZUZ/
>



signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Adam Spiers
jan matejek <[hidden email]> wrote:
> hello,
>
> On 17.8.2016 18:09, Todd Rme wrote:
> > Fedora is implementing a system to automatically add reliable,
> > distro-agnostic provides to their python packages [1].  This
> (...)
> > What does everything think about implementing this on our end as well?
>
> yes, let's please do this. :)

+1 !

We already do this for Ruby:

  https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled

(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
changed at least once, and I am not sure if that doc was correctly
updated at the same time.  But hopefully it's good enough to give you
an idea how it works.)

> >  I am not exactly clear from the discussion what exactly it would
> > entail.
>
> We'd need to extend the autoreqprov generator in our RPM. There is
> already some python-related code, IIRC, something that can generate
> provides from files/directories

Yes, this was newly completed in py2pack by Tom:

  https://github.com/openSUSE/py2pack/pull/45

The next step would be to make this happen automatically at build-time
via autoreqprov and macros, instead of at .spec-editing-time.

> in site-packages directory.

I don't think site-packages is the right source for this metadata.
It should come from the setup.py infrastructure (which may or may
not source the dependencies from somewhere like requires.txt).

> It would be
> beneficial to have both: the canonical name from pypi, what fedora is
> doing now, and also provides for imports -- even though the latter would
> never be perfect.

I guess the most important part is having the correct dependency
names, which means using the pypi names.  I'm not sure whether
providing virtual packages for import namespace names is a good idea.

> Some RPM person please speak up?

Hope the above info helps.  I'm sure that any contributions to improve
this will be very welcome.  Maybe it could be done by extending
py2pack somehow so that it gets called in a special mode at build-time
to extract the metadata - Tom, what do you think?

BTW there is a new github team:

  https://github.com/orgs/openSUSE/teams/python-packaging

Anyone who works on this should probably be added to that team.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Thomas Bechtold
Hi,

On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:

> jan matejek <[hidden email]> wrote:
> > hello,
> >
> > On 17.8.2016 18:09, Todd Rme wrote:
> > > Fedora is implementing a system to automatically add reliable,
> > > distro-agnostic provides to their python packages [1].  This
> > (...)
> > > What does everything think about implementing this on our end as well?
> >
> > yes, let's please do this. :)
>
> +1 !
>
> We already do this for Ruby:
>
>   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
>
> (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> changed at least once, and I am not sure if that doc was correctly
> updated at the same time.  But hopefully it's good enough to give you
> an idea how it works.)
>
> > >  I am not exactly clear from the discussion what exactly it would
> > > entail.
> >
> > We'd need to extend the autoreqprov generator in our RPM. There is
> > already some python-related code, IIRC, something that can generate
> > provides from files/directories

Where can I find the autoreqprov code?

> Yes, this was newly completed in py2pack by Tom:
>
>   https://github.com/openSUSE/py2pack/pull/45
>
> The next step would be to make this happen automatically at build-time
> via autoreqprov and macros, instead of at .spec-editing-time.
>
> > in site-packages directory.
>
> I don't think site-packages is the right source for this metadata.
> It should come from the setup.py infrastructure (which may or may
> not source the dependencies from somewhere like requires.txt).
>
> > It would be
> > beneficial to have both: the canonical name from pypi, what fedora is
> > doing now, and also provides for imports -- even though the latter would
> > never be perfect.
>
> I guess the most important part is having the correct dependency
> names, which means using the pypi names.  I'm not sure whether
> providing virtual packages for import namespace names is a good idea.
>
> > Some RPM person please speak up?
>
> Hope the above info helps.  I'm sure that any contributions to improve
> this will be very welcome.  Maybe it could be done by extending
> py2pack somehow so that it gets called in a special mode at build-time
> to extract the metadata - Tom, what do you think?

I would like to have something like gem2rpm for python. This could be
py2pack. It would generate a .spec file from a given pypi tarball. The
process to generate the spec could be tweaked via a config file (similar
to gem2prm). Is that what you have in mind?

Best,

Tom
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Adam Spiers
Thomas Bechtold <[hidden email]> wrote:

> On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
> > jan matejek <[hidden email]> wrote:
> > > On 17.8.2016 18:09, Todd Rme wrote:
> > > > Fedora is implementing a system to automatically add reliable,
> > > > distro-agnostic provides to their python packages [1].  This
> > > (...)
> > > > What does everything think about implementing this on our end as well?
> > >
> > > yes, let's please do this. :)
> >
> > +1 !
> >
> > We already do this for Ruby:
> >
> >   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
> >
> > (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> > changed at least once, and I am not sure if that doc was correctly
> > updated at the same time.  But hopefully it's good enough to give you
> > an idea how it works.)
> >
> > > >  I am not exactly clear from the discussion what exactly it would
> > > > entail.
> > >
> > > We'd need to extend the autoreqprov generator in our RPM. There is
> > > already some python-related code, IIRC, something that can generate
> > > provides from files/directories
>
> Where can I find the autoreqprov code?

Please see the above link :)

> > Yes, this was newly completed in py2pack by Tom:
> >
> >   https://github.com/openSUSE/py2pack/pull/45
> >
> > The next step would be to make this happen automatically at build-time
> > via autoreqprov and macros, instead of at .spec-editing-time.
> >
> > > in site-packages directory.
> >
> > I don't think site-packages is the right source for this metadata.
> > It should come from the setup.py infrastructure (which may or may
> > not source the dependencies from somewhere like requires.txt).
> >
> > > It would be
> > > beneficial to have both: the canonical name from pypi, what fedora is
> > > doing now, and also provides for imports -- even though the latter would
> > > never be perfect.
> >
> > I guess the most important part is having the correct dependency
> > names, which means using the pypi names.  I'm not sure whether
> > providing virtual packages for import namespace names is a good idea.
> >
> > > Some RPM person please speak up?
> >
> > Hope the above info helps.  I'm sure that any contributions to improve
> > this will be very welcome.  Maybe it could be done by extending
> > py2pack somehow so that it gets called in a special mode at build-time
> > to extract the metadata - Tom, what do you think?
>
> I would like to have something like gem2rpm for python. This could be
> py2pack.

I think py2pack already *is* the gem2rpm equivalent for Python.

> It would generate a .spec file from a given pypi tarball.

Uhh ... isn't that what it does already?  Or am I going senile?

> The
> process to generate the spec could be tweaked via a config file (similar
> to gem2prm). Is that what you have in mind?

Well, there's certainly no harm in stealing ideas for features from
gem2rpm.  But I thought the main point of this thread was to calculate
the Requires and Provides at build-time, so that py2pack doesn't need
to put them in the .spec file at all?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Thomas Bechtold
On Mon, Aug 22, 2016 at 02:13:14PM +0100, Adam Spiers wrote:

> Thomas Bechtold <[hidden email]> wrote:
> > On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
> > > jan matejek <[hidden email]> wrote:
> > > > On 17.8.2016 18:09, Todd Rme wrote:
> > > > > Fedora is implementing a system to automatically add reliable,
> > > > > distro-agnostic provides to their python packages [1].  This
> > > > (...)
> > > > > What does everything think about implementing this on our end as well?
> > > >
> > > > yes, let's please do this. :)
> > >
> > > +1 !
> > >
> > > We already do this for Ruby:
> > >
> > >   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
> > >
> > > (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> > > changed at least once, and I am not sure if that doc was correctly
> > > updated at the same time.  But hopefully it's good enough to give you
> > > an idea how it works.)
> > >
> > > > >  I am not exactly clear from the discussion what exactly it would
> > > > > entail.
> > > >
> > > > We'd need to extend the autoreqprov generator in our RPM. There is
> > > > already some python-related code, IIRC, something that can generate
> > > > provides from files/directories
> >
> > Where can I find the autoreqprov code?
>
> Please see the above link :)
>
> > > Yes, this was newly completed in py2pack by Tom:
> > >
> > >   https://github.com/openSUSE/py2pack/pull/45
> > >
> > > The next step would be to make this happen automatically at build-time
> > > via autoreqprov and macros, instead of at .spec-editing-time.
> > >
> > > > in site-packages directory.
> > >
> > > I don't think site-packages is the right source for this metadata.
> > > It should come from the setup.py infrastructure (which may or may
> > > not source the dependencies from somewhere like requires.txt).
> > >
> > > > It would be
> > > > beneficial to have both: the canonical name from pypi, what fedora is
> > > > doing now, and also provides for imports -- even though the latter would
> > > > never be perfect.
> > >
> > > I guess the most important part is having the correct dependency
> > > names, which means using the pypi names.  I'm not sure whether
> > > providing virtual packages for import namespace names is a good idea.
> > >
> > > > Some RPM person please speak up?
> > >
> > > Hope the above info helps.  I'm sure that any contributions to improve
> > > this will be very welcome.  Maybe it could be done by extending
> > > py2pack somehow so that it gets called in a special mode at build-time
> > > to extract the metadata - Tom, what do you think?
> >
> > I would like to have something like gem2rpm for python. This could be
> > py2pack.
>
> I think py2pack already *is* the gem2rpm equivalent for Python.

Except that you usually need to adjust the generated spec file by hand
(i.e. update-alternative handling)

> > It would generate a .spec file from a given pypi tarball.
>
> Uhh ... isn't that what it does already?  Or am I going senile?

Yes. But as I said, the spec needs to be adjusted by hand usually. You
can't just call py2pack for a new upstream release. It's currently only
useful for the initial convertion from pypi to a spec.

>
> > The
> > process to generate the spec could be tweaked via a config file (similar
> > to gem2prm). Is that what you have in mind?
>
> Well, there's certainly no harm in stealing ideas for features from
> gem2rpm.  But I thought the main point of this thread was to calculate
> the Requires and Provides at build-time, so that py2pack doesn't need
> to put them in the .spec file at all?

That's the question I guess. Do we want to have the
Requires/BuildRequires explicit in the spec (added by a py2pack rerun
whenever you update the spec) or implicit via a macro (which may also
run py2pack to extract the Requires/BuildRequires).

Or am I missunderstanding something?

Best,

Tom
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Adam Spiers
Thomas Bechtold <[hidden email]> wrote:
> On Mon, Aug 22, 2016 at 02:13:14PM +0100, Adam Spiers wrote:
> > Thomas Bechtold <[hidden email]> wrote:
> > > I would like to have something like gem2rpm for python. This could be
> > > py2pack.
> >
> > I think py2pack already *is* the gem2rpm equivalent for Python.
>
> Except that you usually need to adjust the generated spec file by hand
> (i.e. update-alternative handling)

Last time I looked, you did for gem2rpm too.  Maybe that's changed
more recently?

> > > It would generate a .spec file from a given pypi tarball.
> >
> > Uhh ... isn't that what it does already?  Or am I going senile?
>
> Yes. But as I said, the spec needs to be adjusted by hand usually. You
> can't just call py2pack for a new upstream release. It's currently only
> useful for the initial convertion from pypi to a spec.

That's certainly also how gem2rpm started out, at least.

> > > The
> > > process to generate the spec could be tweaked via a config file (similar
> > > to gem2prm). Is that what you have in mind?
> >
> > Well, there's certainly no harm in stealing ideas for features from
> > gem2rpm.  But I thought the main point of this thread was to calculate
> > the Requires and Provides at build-time, so that py2pack doesn't need
> > to put them in the .spec file at all?
>
> That's the question I guess. Do we want to have the
> Requires/BuildRequires explicit in the spec (added by a py2pack rerun
> whenever you update the spec) or implicit via a macro (which may also
> run py2pack to extract the Requires/BuildRequires).

Implicit via a macro!  Why have extra mess in the .spec if we can
avoid it?  As per the OP, Fedora is doing it via macros, and we
already do it via macros for Ruby, so I can't think why we wouldn't
want to do this.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Jason Craig-3
In reply to this post by todd rme
On 08/17/2016 10:09 AM, Todd Rme wrote:

> Fedora is implementing a system to automatically add reliable,
> distro-agnostic provides to their python packages [1].  This
> apparently originally came from Mandriva and Mageia, according to the
> mailing list discussion [2].  They are also adding macros to make
> these easier to use, so you can use the macros to depend on the right
> provides automatically.
>
> This seems like something that would make it much easier to handle
> python packages in a distro-agnostic manner.  The downside is it
> requires rebuilding all python packages to work, which may limit its
> applicably to previous openSUSE/SLE releases.
>
> What does everything think about implementing this on our end as well?
>  I am not exactly clear from the discussion what exactly it would
> entail.

I am supportive of the idea, but with a caveat. As I understand from the
first link provided, they are using a "canonical name" in the "Provides"
tag. This canonical name "can differ slightly from the name displayed,
for example, on PyPI". If this is the case, we would be setting
ourselves up for a situation where at some point (if not already) two
packages made for different PyPi distributions end up having the same
Provides tag.

PyPi is the de facto standard for Python distributions, and beyond that
PEP 426 specifically defines what a metadata "name" for a distribution
is. To go against PyPi and PEP 426 would be, IMO incorrect.

The PEP 426 name can only be letters, digits, underscores, hyphens and
periods and must start and end with a letter or digit. Can the Provides
tag not include such a token?

--
Jason Craig
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Thomas Bechtold
In reply to this post by Adam Spiers
Hi,
On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:

> jan matejek <[hidden email]> wrote:
> > hello,
> >
> > On 17.8.2016 18:09, Todd Rme wrote:
> > > Fedora is implementing a system to automatically add reliable,
> > > distro-agnostic provides to their python packages [1].  This
> > (...)
> > > What does everything think about implementing this on our end as well?
> >
> > yes, let's please do this. :)
>
> +1 !
>
> We already do this for Ruby:
>
>   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
>
> (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> changed at least once, and I am not sure if that doc was correctly
> updated at the same time.  But hopefully it's good enough to give you
> an idea how it works.)
>
> > >  I am not exactly clear from the discussion what exactly it would
> > > entail.
> >
> > We'd need to extend the autoreqprov generator in our RPM. There is
> > already some python-related code, IIRC, something that can generate
> > provides from files/directories
>
> Yes, this was newly completed in py2pack by Tom:
>
>   https://github.com/openSUSE/py2pack/pull/45

I plan to remove this from py2pack again because I split the code to a
new tool called metaextract (https://pypi.python.org/pypi/metaextract)
I plan to use that also in another project (OpenStack packaging related)

>
> The next step would be to make this happen automatically at build-time
> via autoreqprov and macros, instead of at .spec-editing-time.
>
> > in site-packages directory.
>
> I don't think site-packages is the right source for this metadata.
> It should come from the setup.py infrastructure (which may or may
> not source the dependencies from somewhere like requires.txt).
>
> > It would be
> > beneficial to have both: the canonical name from pypi, what fedora is
> > doing now, and also provides for imports -- even though the latter would
> > never be perfect.
>
> I guess the most important part is having the correct dependency
> names, which means using the pypi names.  I'm not sure whether
> providing virtual packages for import namespace names is a good idea.
>
> > Some RPM person please speak up?
>
> Hope the above info helps.  I'm sure that any contributions to improve
> this will be very welcome.  Maybe it could be done by extending
> py2pack somehow so that it gets called in a special mode at build-time
> to extract the metadata - Tom, what do you think?
>
> BTW there is a new github team:
>
>   https://github.com/orgs/openSUSE/teams/python-packaging
>
> Anyone who works on this should probably be added to that team.
> --
> To unsubscribe, e-mail: [hidden email]
> To contact the owner, e-mail: [hidden email]
>
>
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Adam Spiers
Thomas Bechtold <[hidden email]> wrote:

> On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
> > jan matejek <[hidden email]> wrote:
> > > hello,
> > >
> > > On 17.8.2016 18:09, Todd Rme wrote:
> > > > Fedora is implementing a system to automatically add reliable,
> > > > distro-agnostic provides to their python packages [1].  This
> > > (...)
> > > > What does everything think about implementing this on our end as well?
> > >
> > > yes, let's please do this. :)
> >
> > +1 !
> >
> > We already do this for Ruby:
> >
> >   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
> >
> > (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> > changed at least once, and I am not sure if that doc was correctly
> > updated at the same time.  But hopefully it's good enough to give you
> > an idea how it works.)
> >
> > > >  I am not exactly clear from the discussion what exactly it would
> > > > entail.
> > >
> > > We'd need to extend the autoreqprov generator in our RPM. There is
> > > already some python-related code, IIRC, something that can generate
> > > provides from files/directories
> >
> > Yes, this was newly completed in py2pack by Tom:
> >
> >   https://github.com/openSUSE/py2pack/pull/45
>
> I plan to remove this from py2pack again because I split the code to a
> new tool called metaextract (https://pypi.python.org/pypi/metaextract)
> I plan to use that also in another project (OpenStack packaging related)

Splitting it out sounds good.  But just to be clear - you aren't
removing the *functionality* from py2pack, right?  I mean, py2pack
will still use metaextract to retain the metadata extraction
functionality it already has, rather than regressing to the unreliable
mechanism of parsing setup.py?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Automatic Python Provides

Thomas Bechtold
On Fri, Aug 26, 2016 at 11:20:35AM +0100, Adam Spiers wrote:

> Thomas Bechtold <[hidden email]> wrote:
> > On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
> > > jan matejek <[hidden email]> wrote:
> > > > hello,
> > > >
> > > > On 17.8.2016 18:09, Todd Rme wrote:
> > > > > Fedora is implementing a system to automatically add reliable,
> > > > > distro-agnostic provides to their python packages [1].  This
> > > > (...)
> > > > > What does everything think about implementing this on our end as well?
> > > >
> > > > yes, let's please do this. :)
> > >
> > > +1 !
> > >
> > > We already do this for Ruby:
> > >
> > >   https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_automatically_handled
> > >
> > > (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has
> > > changed at least once, and I am not sure if that doc was correctly
> > > updated at the same time.  But hopefully it's good enough to give you
> > > an idea how it works.)
> > >
> > > > >  I am not exactly clear from the discussion what exactly it would
> > > > > entail.
> > > >
> > > > We'd need to extend the autoreqprov generator in our RPM. There is
> > > > already some python-related code, IIRC, something that can generate
> > > > provides from files/directories
> > >
> > > Yes, this was newly completed in py2pack by Tom:
> > >
> > >   https://github.com/openSUSE/py2pack/pull/45
> >
> > I plan to remove this from py2pack again because I split the code to a
> > new tool called metaextract (https://pypi.python.org/pypi/metaextract)
> > I plan to use that also in another project (OpenStack packaging related)
>
> Splitting it out sounds good.  But just to be clear - you aren't
> removing the *functionality* from py2pack, right?  I mean, py2pack
> will still use metaextract to retain the metadata extraction
> functionality it already has, rather than regressing to the unreliable
> mechanism of parsing setup.py?

Yes. Just the code moved to an external module. Sorry for not being
precise here :-)

Best,

Tom
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]