fixes for rpmlint ? python singlespec pyc, opt-1.pyc
Locked
fixes for rpmlint ? python singlespec pyc, opt-1.pyc
 
|
For any packages build with the new singlespec way there those warnings
[ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ? -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc
|
|
On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:
> For any packages build with the new singlespec way there those warnings > > [ 21s] RPMLINT report: > [ 21s] =============== > [ 21s] python3-mappyfile.noarch: W: files-duplicate > /usr/lib/python3.6/site- > packages/mappyfile/__pycache__/tokens.cpython-36.pyc > /usr/lib/python3.6/site- > packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc > [ 21s] python3-mappyfile.noarch: W: files-duplicate > /usr/lib/python3.6/site- > packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ > python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ > 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- > packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe > /usr/lib/python3.6/site- > packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt > [ 21s] python3-mappyfile.noarch: W: files-duplicate > /usr/lib/python3.6/site- > packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ > python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ > 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. > > Should RPMLINT be fixed to become smarter, and just ignore those, > or something still need to be fixed in the python part ? RPMLINT is right on this, we should not ship duplicate files. The files are duplicate because the opt-1 optimization level has no effect on the bytecode files. At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates. The next question is, should we even ship both no-optimization and opt-1 bytecode files? The bytecode itself improves startup time significantly, but the opt-1 is dubious: - it removes any assert(...) statements. Although may improve speed if there are assertions in a hot path, I doubt this is a good idea in general - opt-1 bytecode files will be ignored until we use "python3 -O1" or set PYTONOPTIMIZE=1 - if there are no assertions, both bytecode files will be identical To test it yourself, put the following three files in the same directory, and execute the tests with: a) python3 ./opttest.py b) python3 -O ./opttest.py c) ./opttest.py d) PYTHONOPTIMIZE=1 ./opttest.py --- # optmodule_noassert.py def test_no_assert(): print("no assertion") --- # optmodule.py def test_assert_good(): assert(1) print("assert good") def test_assert_fail(): assert(0) print("assert fail") --- #! /usr/bin/python3 # opttest.py import optmodule import optmodule_noassert print("opttest") optmodule_noassert.test_no_assert() optmodule.test_assert_good() optmodule.test_assert_fail() --- Command a) and c) will throw an assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.pyc. Command b) and d) will throw no assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.opt-1.pyc. Both optmodule_noassert bytecode files are binary identical. Kind regards, Stefan -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
fdupes -s [was: Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc]
|
|
on a sidenote:
On 5.4.2017 17:35, Brüns, Stefan wrote: > At least run %fdupes -s on the target directory, this will remove (symlink) > the duplicates. why do you recommend "-s" and symlinks? The default is to create hardlinks, but apparently some people consistently use the "-s" flag. Is there a good reason for this? ISTM hardlinks are better from performance standpoint and have no downsides otherwise, unless you're fdupe'ing across mountpoints. thanks m. |
signature.asc (849 bytes)
Re: rpmlint broken for hardlink ,was fdupes, fixes for rpmlint :-)
|
|
On mercredi, 5 avril 2017 18.17:52 h CEST jan matejek wrote:
> on a sidenote: > > On 5.4.2017 17:35, Brüns, Stefan wrote: > > At least run %fdupes -s on the target directory, this will remove > > (symlink) > > the duplicates. > > why do you recommend "-s" and symlinks? > > The default is to create hardlinks, but apparently some people consistently > use the "-s" flag. Is there a good reason for this? > ISTM hardlinks are better from performance standpoint and have no downsides > otherwise, unless you're fdupe'ing across mountpoints. > > thanks > m. by the way -s doesn't make rpmlint happy I got a symlink but this the complain :-( So if I expand rpm --eval %fdupes on my TW I'm getting this _target=""; _symlinks=0; fdupes -q -p -n -H -o name -r %1 | while read _file; do if test -z "$_target" ; then _target="$_file"; else if test -z "$_file" ; then _target=""; continue ; fi ; if test "$_symlinks" = 1; then ln -sf "${_target#/home/bruno/rpmbuild/BUILDROOT/%{name}-%{version}-% {release}.x86_64}" "$_file"; else ln -f "$_target" "$_file"; fi ; fi ; done fdupes -q -p -n -H -o name -r /some/path/with/files give the duplicates correctly /tmp/mappyfile/__init__.pyc /tmp/mappyfile/__init__.pyo /tmp/mappyfile/ordereddict.pyc /tmp/mappyfile/ordereddict.pyo /tmp/mappyfile/tokens.pyc /tmp/mappyfile/tokens.pyo /tmp/mappyfile/utils.pyc /tmp/mappyfile/utils.pyo And the result extracted is right ls -ld __init__.py* -rw-r--r-- 1 root root 147 Apr 5 18:26 __init__.py -rw-r--r-- 2 root root 307 Apr 5 18:26 __init__.pyc -rw-r--r-- 2 root root 307 Apr 5 18:26 __init__.pyo We got the 2 links for the duplicate confirmed by stat stat __init__.py* File: __init__.py Size: 147 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030758 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:26:52.287792333 +0200 Modify: 2017-04-05 18:26:52.287792333 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - File: __init__.pyc Size: 307 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030772 Links: 2 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:28:13.052033536 +0200 Modify: 2017-04-05 18:26:52.291792345 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - File: __init__.pyo Size: 307 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030772 Links: 2 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:28:13.052033536 +0200 Modify: 2017-04-05 18:26:52.291792345 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - So rpmlint is broken :-) -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc
|
|
In reply to this post by Brüns, Stefan
On mercredi, 5 avril 2017 17.35:06 h CEST Brüns, Stefan wrote:
> On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote: > > For any packages build with the new singlespec way there those warnings > > > > [ 21s] RPMLINT report: > > [ 21s] =============== > > [ 21s] python3-mappyfile.noarch: W: files-duplicate > > /usr/lib/python3.6/site- > > packages/mappyfile/__pycache__/tokens.cpython-36.pyc > > /usr/lib/python3.6/site- > > packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc > > [ 21s] python3-mappyfile.noarch: W: files-duplicate > > /usr/lib/python3.6/site- > > packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ > > python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ > > 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- > > packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe > > /usr/lib/python3.6/site- > > packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt > > [ 21s] python3-mappyfile.noarch: W: files-duplicate > > /usr/lib/python3.6/site- > > packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ > > python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ > > > > 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. > > > > Should RPMLINT be fixed to become smarter, and just ignore those, > > or something still need to be fixed in the python part ? > > RPMLINT is right on this, we should not ship duplicate files. The files are > duplicate because the opt-1 optimization level has no effect on the bytecode > files. > > At least run %fdupes -s on the target directory, this will remove (symlink) > the duplicates. > > The next question is, should we even ship both no-optimization and opt-1 > bytecode files? The bytecode itself improves startup time significantly, but > the opt-1 is dubious: > > - it removes any assert(...) statements. Although may improve speed if there > are assertions in a hot path, I doubt this is a good idea in general > > - opt-1 bytecode files will be ignored until we use "python3 -O1" or set > PYTONOPTIMIZE=1 > > - if there are no assertions, both bytecode files will be identical > > To test it yourself, put the following three files in the same directory, > and execute the tests with: > a) python3 ./opttest.py > b) python3 -O ./opttest.py > c) ./opttest.py > d) PYTHONOPTIMIZE=1 ./opttest.py > > --- > # optmodule_noassert.py > def test_no_assert(): > print("no assertion") > > --- > # optmodule.py > def test_assert_good(): > assert(1) > print("assert good") > > def test_assert_fail(): > assert(0) > print("assert fail") > --- > #! /usr/bin/python3 > # opttest.py > import optmodule > import optmodule_noassert > > print("opttest") > > optmodule_noassert.test_no_assert() > optmodule.test_assert_good() > optmodule.test_assert_fail() > --- > > Command a) and c) will throw an assertion and generate __pycache__/ > optmodule{_noassert}.cpython-36.pyc. > > Command b) and d) will throw no assertion and generate __pycache__/ > optmodule{_noassert}.cpython-36.opt-1.pyc. > > Both optmodule_noassert bytecode files are binary identical. > > Kind regards, > > Stefan The fault is due to rpmlint, mostly there's a patch in it but it has 3.5 hardcoded :-) Reported https://bugzilla.opensuse.org/show_bug.cgi?id=1032748 -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
|
|
In reply to this post by Brüns, Stefan
On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan
<[hidden email]> wrote: > On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote: >> For any packages build with the new singlespec way there those warnings >> >> [ 21s] RPMLINT report: >> [ 21s] =============== >> [ 21s] python3-mappyfile.noarch: W: files-duplicate >> /usr/lib/python3.6/site- >> packages/mappyfile/__pycache__/tokens.cpython-36.pyc >> /usr/lib/python3.6/site- >> packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc >> [ 21s] python3-mappyfile.noarch: W: files-duplicate >> /usr/lib/python3.6/site- >> packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ >> python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ >> 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- >> packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe >> /usr/lib/python3.6/site- >> packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt >> [ 21s] python3-mappyfile.noarch: W: files-duplicate >> /usr/lib/python3.6/site- >> packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ >> python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ >> 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. >> >> Should RPMLINT be fixed to become smarter, and just ignore those, >> or something still need to be fixed in the python part ? > > RPMLINT is right on this, we should not ship duplicate files. The files are > duplicate because the opt-1 optimization level has no effect on the bytecode > files. > > At least run %fdupes -s on the target directory, this will remove (symlink) > the duplicates. > > The next question is, should we even ship both no-optimization and opt-1 > bytecode files? The bytecode itself improves startup time significantly, but > the opt-1 is dubious: > > - it removes any assert(...) statements. Although may improve speed if there > are assertions in a hot path, I doubt this is a good idea in general > > - opt-1 bytecode files will be ignored until we use "python3 -O1" or set > PYTONOPTIMIZE=1 > > - if there are no assertions, both bytecode files will be identical > > To test it yourself, put the following three files in the same directory, and > execute the tests with: > a) python3 ./opttest.py > b) python3 -O ./opttest.py > c) ./opttest.py > d) PYTHONOPTIMIZE=1 ./opttest.py > > --- > # optmodule_noassert.py > def test_no_assert(): > print("no assertion") > > --- > # optmodule.py > def test_assert_good(): > assert(1) > print("assert good") > > def test_assert_fail(): > assert(0) > print("assert fail") > --- > #! /usr/bin/python3 > # opttest.py > import optmodule > import optmodule_noassert > > print("opttest") > > optmodule_noassert.test_no_assert() > optmodule.test_assert_good() > optmodule.test_assert_fail() > --- > > Command a) and c) will throw an assertion and generate __pycache__/ > optmodule{_noassert}.cpython-36.pyc. > > Command b) and d) will throw no assertion and generate __pycache__/ > optmodule{_noassert}.cpython-36.opt-1.pyc. > > Both optmodule_noassert bytecode files are binary identical. > > Kind regards, > > Stefan Can the python singlespec macros be updated to automatically hardlink or symlink these files? -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc
|
|
On vendredi, 7 avril 2017 16.25:43 h CEST Todd Rme wrote:
> On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan > > <[hidden email]> wrote: > > On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote: > >> For any packages build with the new singlespec way there those warnings > >> > >> [ 21s] RPMLINT report: > >> [ 21s] =============== > >> [ 21s] python3-mappyfile.noarch: W: files-duplicate > >> /usr/lib/python3.6/site- > >> packages/mappyfile/__pycache__/tokens.cpython-36.pyc > >> /usr/lib/python3.6/site- > >> packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc > >> [ 21s] python3-mappyfile.noarch: W: files-duplicate > >> /usr/lib/python3.6/site- > >> packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ > >> python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ > >> 21s] python3-mappyfile.noarch: W: files-duplicate > >> /usr/lib/python3.6/site- > >> packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe > >> /usr/lib/python3.6/site- > >> packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt > >> [ 21s] python3-mappyfile.noarch: W: files-duplicate > >> /usr/lib/python3.6/site- > >> packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ > >> python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc > >> [ > >> > >> 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. > >> > >> Should RPMLINT be fixed to become smarter, and just ignore those, > >> or something still need to be fixed in the python part ? > > > > RPMLINT is right on this, we should not ship duplicate files. The files > > are > > duplicate because the opt-1 optimization level has no effect on the > > bytecode files. > > > > At least run %fdupes -s on the target directory, this will remove > > (symlink) > > the duplicates. > > > > The next question is, should we even ship both no-optimization and opt-1 > > bytecode files? The bytecode itself improves startup time significantly, > > but the opt-1 is dubious: > > > > - it removes any assert(...) statements. Although may improve speed if > > there are assertions in a hot path, I doubt this is a good idea in > > general > > > > - opt-1 bytecode files will be ignored until we use "python3 -O1" or set > > PYTONOPTIMIZE=1 > > > > - if there are no assertions, both bytecode files will be identical > > > > To test it yourself, put the following three files in the same directory, > > and execute the tests with: > > a) python3 ./opttest.py > > b) python3 -O ./opttest.py > > c) ./opttest.py > > d) PYTHONOPTIMIZE=1 ./opttest.py > > > > --- > > # optmodule_noassert.py > > > > def test_no_assert(): > > print("no assertion") > > > > --- > > # optmodule.py > > > > def test_assert_good(): > > assert(1) > > print("assert good") > > > > def test_assert_fail(): > > assert(0) > > print("assert fail") > > > > --- > > #! /usr/bin/python3 > > # opttest.py > > import optmodule > > import optmodule_noassert > > > > print("opttest") > > > > optmodule_noassert.test_no_assert() > > optmodule.test_assert_good() > > optmodule.test_assert_fail() > > --- > > > > Command a) and c) will throw an assertion and generate __pycache__/ > > optmodule{_noassert}.cpython-36.pyc. > > > > Command b) and d) will throw no assertion and generate __pycache__/ > > optmodule{_noassert}.cpython-36.opt-1.pyc. > > > > Both optmodule_noassert bytecode files are binary identical. > > > > Kind regards, > > > > Stefan > > Can the python singlespec macros be updated to automatically hardlink > or symlink these files? I've found where we need to be really carefull with singlespecs .... The fdupes line HAS to be written like this %python_expand %fdupes %{buildroot}%{$python_sitelib} Then it works -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: [hidden email] To contact the owner, e-mail: [hidden email] |
«
Return to opensuse-packaging
|
1 view|%1 views
| Free forum by Nabble | Edit this page |