fixes for rpmlint ? python singlespec pyc, opt-1.pyc

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

fixes for rpmlint ? python singlespec pyc, opt-1.pyc

Bruno Friedmann-2
For any packages build with the new singlespec way there those warnings

[   21s] RPMLINT report:
[   21s] ===============
[   21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site-
packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc
[   21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/
python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc
[   21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site-
packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt
[   21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/
python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc
[   21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.

Should RPMLINT be fixed to become smarter, and just ignore those,
or something still need to be fixed in the python part ?



--

Bruno Friedmann
 Ioda-Net Sàrl www.ioda-net.ch
 Bareos Partner, openSUSE Member, fsfe fellowship
 GPG KEY : D5C9B751C4653227
 irc: tigerfoot


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc

Brüns, Stefan
On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:

> For any packages build with the new singlespec way there those warnings
>
> [   21s] RPMLINT report:
> [   21s] ===============
> [   21s] python3-mappyfile.noarch: W: files-duplicate
> /usr/lib/python3.6/site-
> packages/mappyfile/__pycache__/tokens.cpython-36.pyc
> /usr/lib/python3.6/site-
> packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc
> [   21s] python3-mappyfile.noarch: W: files-duplicate
> /usr/lib/python3.6/site-
> packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/
> python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [  
> 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
> packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe
> /usr/lib/python3.6/site-
> packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt
> [   21s] python3-mappyfile.noarch: W: files-duplicate
> /usr/lib/python3.6/site-
> packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/
> python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [
>  21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.
>
> Should RPMLINT be fixed to become smarter, and just ignore those,
> or something still need to be fixed in the python part ?

RPMLINT is right on this, we should not ship duplicate files. The files are
duplicate because the opt-1 optimization level has no effect on the bytecode
files.

At least run %fdupes -s on the target directory, this will remove (symlink)
the duplicates.

The next question is, should we even ship both no-optimization and opt-1
bytecode files? The bytecode itself improves startup time significantly, but
the opt-1 is dubious:

- it removes any assert(...) statements. Although may improve speed if there
are assertions in a hot path, I doubt this is a good idea in general

- opt-1 bytecode files will be ignored until we use "python3 -O1" or set
PYTONOPTIMIZE=1

- if there are no assertions, both bytecode files will be identical

To test it yourself, put the following three files in the same directory, and
execute the tests with:
a) python3 ./opttest.py
b) python3 -O ./opttest.py
c) ./opttest.py
d) PYTHONOPTIMIZE=1 ./opttest.py

---
# optmodule_noassert.py
def test_no_assert():
        print("no assertion")

---
# optmodule.py
def test_assert_good():
        assert(1)
        print("assert good")

def test_assert_fail():
        assert(0)
        print("assert fail")
---
#! /usr/bin/python3
# opttest.py
import optmodule
import optmodule_noassert

print("opttest")

optmodule_noassert.test_no_assert()
optmodule.test_assert_good()
optmodule.test_assert_fail()
---

Command a) and c) will throw an assertion and generate __pycache__/
optmodule{_noassert}.cpython-36.pyc.

Command b) and d) will throw no assertion and generate __pycache__/
optmodule{_noassert}.cpython-36.opt-1.pyc.

Both optmodule_noassert bytecode files are binary identical.

Kind regards,

Stefan
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

fdupes -s [was: Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc]

jan matejek-4
on a sidenote:

On 5.4.2017 17:35, Brüns, Stefan wrote:
> At least run %fdupes -s on the target directory, this will remove (symlink)
> the duplicates.

why do you recommend "-s" and symlinks?

The default is to create hardlinks, but apparently some people consistently use the "-s" flag.
Is there a good reason for this?
ISTM hardlinks are better from performance standpoint and have no downsides otherwise, unless you're
fdupe'ing across mountpoints.

thanks
m.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rpmlint broken for hardlink ,was fdupes, fixes for rpmlint :-)

Bruno Friedmann-2
On mercredi, 5 avril 2017 18.17:52 h CEST jan matejek wrote:

> on a sidenote:
>
> On 5.4.2017 17:35, Brüns, Stefan wrote:
> > At least run %fdupes -s on the target directory, this will remove
> > (symlink)
> > the duplicates.
>
> why do you recommend "-s" and symlinks?
>
> The default is to create hardlinks, but apparently some people consistently
> use the "-s" flag. Is there a good reason for this?
> ISTM hardlinks are better from performance standpoint and have no downsides
> otherwise, unless you're fdupe'ing across mountpoints.
>
> thanks
> m.

by the way -s doesn't make rpmlint happy I got a symlink but this the complain
:-(

So if I expand rpm --eval %fdupes on my TW I'm getting this

 _target="";
 _symlinks=0;

 fdupes -q -p -n -H -o name -r %1 |
  while read _file; do
    if test -z "$_target" ; then
      _target="$_file";
    else
      if test -z "$_file" ; then
        _target="";
        continue ;
      fi ;
      if test "$_symlinks" = 1; then
        ln -sf "${_target#/home/bruno/rpmbuild/BUILDROOT/%{name}-%{version}-%
{release}.x86_64}" "$_file";
      else
        ln -f "$_target" "$_file";
      fi ;
    fi ;
 done

fdupes -q -p -n -H -o name -r /some/path/with/files
give the duplicates correctly

/tmp/mappyfile/__init__.pyc
/tmp/mappyfile/__init__.pyo

/tmp/mappyfile/ordereddict.pyc
/tmp/mappyfile/ordereddict.pyo

/tmp/mappyfile/tokens.pyc
/tmp/mappyfile/tokens.pyo

/tmp/mappyfile/utils.pyc
/tmp/mappyfile/utils.pyo

And the result extracted is right

ls -ld __init__.py*
-rw-r--r-- 1 root root 147 Apr  5 18:26 __init__.py
-rw-r--r-- 2 root root 307 Apr  5 18:26 __init__.pyc
-rw-r--r-- 2 root root 307 Apr  5 18:26 __init__.pyo

We got the 2 links for the duplicate
confirmed by stat

stat __init__.py*
  File: __init__.py
  Size: 147             Blocks: 8          IO Block: 4096   regular file
Device: 27h/39d Inode: 1030758     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-05 18:26:52.287792333 +0200
Modify: 2017-04-05 18:26:52.287792333 +0200
Change: 2017-04-05 18:26:52.291792345 +0200
 Birth: -
  File: __init__.pyc
  Size: 307             Blocks: 8          IO Block: 4096   regular file
Device: 27h/39d Inode: 1030772     Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-05 18:28:13.052033536 +0200
Modify: 2017-04-05 18:26:52.291792345 +0200
Change: 2017-04-05 18:26:52.291792345 +0200
 Birth: -
  File: __init__.pyo
  Size: 307             Blocks: 8          IO Block: 4096   regular file
Device: 27h/39d Inode: 1030772     Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-05 18:28:13.052033536 +0200
Modify: 2017-04-05 18:26:52.291792345 +0200
Change: 2017-04-05 18:26:52.291792345 +0200
 Birth: -

So rpmlint is broken :-)

--

Bruno Friedmann
 Ioda-Net Sàrl www.ioda-net.ch
 Bareos Partner, openSUSE Member, fsfe fellowship
 GPG KEY : D5C9B751C4653227
 irc: tigerfoot


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc

Bruno Friedmann-2
In reply to this post by Brüns, Stefan
On mercredi, 5 avril 2017 17.35:06 h CEST Brüns, Stefan wrote:

> On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:
> > For any packages build with the new singlespec way there those warnings
> >
> > [   21s] RPMLINT report:
> > [   21s] ===============
> > [   21s] python3-mappyfile.noarch: W: files-duplicate
> > /usr/lib/python3.6/site-
> > packages/mappyfile/__pycache__/tokens.cpython-36.pyc
> > /usr/lib/python3.6/site-
> > packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc
> > [   21s] python3-mappyfile.noarch: W: files-duplicate
> > /usr/lib/python3.6/site-
> > packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/
> > python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [
> > 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
> > packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe
> > /usr/lib/python3.6/site-
> > packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt
> > [   21s] python3-mappyfile.noarch: W: files-duplicate
> > /usr/lib/python3.6/site-
> > packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/
> > python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [
> >
> >  21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.
> >
> > Should RPMLINT be fixed to become smarter, and just ignore those,
> > or something still need to be fixed in the python part ?
>
> RPMLINT is right on this, we should not ship duplicate files. The files are
> duplicate because the opt-1 optimization level has no effect on the bytecode
> files.
>
> At least run %fdupes -s on the target directory, this will remove (symlink)
> the duplicates.
>
> The next question is, should we even ship both no-optimization and opt-1
> bytecode files? The bytecode itself improves startup time significantly, but
> the opt-1 is dubious:
>
> - it removes any assert(...) statements. Although may improve speed if there
> are assertions in a hot path, I doubt this is a good idea in general
>
> - opt-1 bytecode files will be ignored until we use "python3 -O1" or set
> PYTONOPTIMIZE=1
>
> - if there are no assertions, both bytecode files will be identical
>
> To test it yourself, put the following three files in the same directory,
> and execute the tests with:
> a) python3 ./opttest.py
> b) python3 -O ./opttest.py
> c) ./opttest.py
> d) PYTHONOPTIMIZE=1 ./opttest.py
>
> ---
> # optmodule_noassert.py
> def test_no_assert():
>         print("no assertion")
>
> ---
> # optmodule.py
> def test_assert_good():
>         assert(1)
>         print("assert good")
>
> def test_assert_fail():
>         assert(0)
>         print("assert fail")
> ---
> #! /usr/bin/python3
> # opttest.py
> import optmodule
> import optmodule_noassert
>
> print("opttest")
>
> optmodule_noassert.test_no_assert()
> optmodule.test_assert_good()
> optmodule.test_assert_fail()
> ---
>
> Command a) and c) will throw an assertion and generate __pycache__/
> optmodule{_noassert}.cpython-36.pyc.
>
> Command b) and d) will throw no assertion and generate __pycache__/
> optmodule{_noassert}.cpython-36.opt-1.pyc.
>
> Both optmodule_noassert bytecode files are binary identical.
>
> Kind regards,
>
> Stefan

The fault is due to rpmlint, mostly there's a patch in it but it has 3.5
hardcoded :-)

Reported
https://bugzilla.opensuse.org/show_bug.cgi?id=1032748


--

Bruno Friedmann
 Ioda-Net Sàrl www.ioda-net.ch
 Bareos Partner, openSUSE Member, fsfe fellowship
 GPG KEY : D5C9B751C4653227
 irc: tigerfoot


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc

todd rme
In reply to this post by Brüns, Stefan
On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan
<[hidden email]> wrote:

> On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:
>> For any packages build with the new singlespec way there those warnings
>>
>> [   21s] RPMLINT report:
>> [   21s] ===============
>> [   21s] python3-mappyfile.noarch: W: files-duplicate
>> /usr/lib/python3.6/site-
>> packages/mappyfile/__pycache__/tokens.cpython-36.pyc
>> /usr/lib/python3.6/site-
>> packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc
>> [   21s] python3-mappyfile.noarch: W: files-duplicate
>> /usr/lib/python3.6/site-
>> packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/
>> python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [
>> 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site-
>> packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe
>> /usr/lib/python3.6/site-
>> packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt
>> [   21s] python3-mappyfile.noarch: W: files-duplicate
>> /usr/lib/python3.6/site-
>> packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/
>> python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [
>>  21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.
>>
>> Should RPMLINT be fixed to become smarter, and just ignore those,
>> or something still need to be fixed in the python part ?
>
> RPMLINT is right on this, we should not ship duplicate files. The files are
> duplicate because the opt-1 optimization level has no effect on the bytecode
> files.
>
> At least run %fdupes -s on the target directory, this will remove (symlink)
> the duplicates.
>
> The next question is, should we even ship both no-optimization and opt-1
> bytecode files? The bytecode itself improves startup time significantly, but
> the opt-1 is dubious:
>
> - it removes any assert(...) statements. Although may improve speed if there
> are assertions in a hot path, I doubt this is a good idea in general
>
> - opt-1 bytecode files will be ignored until we use "python3 -O1" or set
> PYTONOPTIMIZE=1
>
> - if there are no assertions, both bytecode files will be identical
>
> To test it yourself, put the following three files in the same directory, and
> execute the tests with:
> a) python3 ./opttest.py
> b) python3 -O ./opttest.py
> c) ./opttest.py
> d) PYTHONOPTIMIZE=1 ./opttest.py
>
> ---
> # optmodule_noassert.py
> def test_no_assert():
>         print("no assertion")
>
> ---
> # optmodule.py
> def test_assert_good():
>         assert(1)
>         print("assert good")
>
> def test_assert_fail():
>         assert(0)
>         print("assert fail")
> ---
> #! /usr/bin/python3
> # opttest.py
> import optmodule
> import optmodule_noassert
>
> print("opttest")
>
> optmodule_noassert.test_no_assert()
> optmodule.test_assert_good()
> optmodule.test_assert_fail()
> ---
>
> Command a) and c) will throw an assertion and generate __pycache__/
> optmodule{_noassert}.cpython-36.pyc.
>
> Command b) and d) will throw no assertion and generate __pycache__/
> optmodule{_noassert}.cpython-36.opt-1.pyc.
>
> Both optmodule_noassert bytecode files are binary identical.
>
> Kind regards,
>
> Stefan

Can the python singlespec macros be updated to automatically hardlink
or symlink these files?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc

Bruno Friedmann-2
On vendredi, 7 avril 2017 16.25:43 h CEST Todd Rme wrote:

> On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan
>
> <[hidden email]> wrote:
> > On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:
> >> For any packages build with the new singlespec way there those warnings
> >>
> >> [   21s] RPMLINT report:
> >> [   21s] ===============
> >> [   21s] python3-mappyfile.noarch: W: files-duplicate
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile/__pycache__/tokens.cpython-36.pyc
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc
> >> [   21s] python3-mappyfile.noarch: W: files-duplicate
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/
> >> python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [
> >> 21s] python3-mappyfile.noarch: W: files-duplicate
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt
> >> [   21s] python3-mappyfile.noarch: W: files-duplicate
> >> /usr/lib/python3.6/site-
> >> packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/
> >> python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc
> >> [
> >>
> >>  21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.
> >>
> >> Should RPMLINT be fixed to become smarter, and just ignore those,
> >> or something still need to be fixed in the python part ?
> >
> > RPMLINT is right on this, we should not ship duplicate files. The files
> > are
> > duplicate because the opt-1 optimization level has no effect on the
> > bytecode files.
> >
> > At least run %fdupes -s on the target directory, this will remove
> > (symlink)
> > the duplicates.
> >
> > The next question is, should we even ship both no-optimization and opt-1
> > bytecode files? The bytecode itself improves startup time significantly,
> > but the opt-1 is dubious:
> >
> > - it removes any assert(...) statements. Although may improve speed if
> > there are assertions in a hot path, I doubt this is a good idea in
> > general
> >
> > - opt-1 bytecode files will be ignored until we use "python3 -O1" or set
> > PYTONOPTIMIZE=1
> >
> > - if there are no assertions, both bytecode files will be identical
> >
> > To test it yourself, put the following three files in the same directory,
> > and execute the tests with:
> > a) python3 ./opttest.py
> > b) python3 -O ./opttest.py
> > c) ./opttest.py
> > d) PYTHONOPTIMIZE=1 ./opttest.py
> >
> > ---
> > # optmodule_noassert.py
> >
> > def test_no_assert():
> >         print("no assertion")
> >
> > ---
> > # optmodule.py
> >
> > def test_assert_good():
> >         assert(1)
> >         print("assert good")
> >
> > def test_assert_fail():
> >         assert(0)
> >         print("assert fail")
> >
> > ---
> > #! /usr/bin/python3
> > # opttest.py
> > import optmodule
> > import optmodule_noassert
> >
> > print("opttest")
> >
> > optmodule_noassert.test_no_assert()
> > optmodule.test_assert_good()
> > optmodule.test_assert_fail()
> > ---
> >
> > Command a) and c) will throw an assertion and generate __pycache__/
> > optmodule{_noassert}.cpython-36.pyc.
> >
> > Command b) and d) will throw no assertion and generate __pycache__/
> > optmodule{_noassert}.cpython-36.opt-1.pyc.
> >
> > Both optmodule_noassert bytecode files are binary identical.
> >
> > Kind regards,
> >
> > Stefan
>
> Can the python singlespec macros be updated to automatically hardlink
> or symlink these files?

I've found where we need to be really carefull with singlespecs ....
The fdupes line HAS to be written like this

%python_expand %fdupes %{buildroot}%{$python_sitelib}

Then it works

--

Bruno Friedmann
 Ioda-Net Sàrl www.ioda-net.ch
 Bareos Partner, openSUSE Member, fsfe fellowship
 GPG KEY : D5C9B751C4653227
 irc: tigerfoot


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]