reproducible builds

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

reproducible builds

Adam Spiers
Is anyone working on (or thinking of working on) making our build
process reproducible?

    https://reproducible-builds.org/

It seems Debian and Fedora are already part of the project, and the
advantages are quite compelling, not just from a security perspective,
but also due to the potential savings in storage and network
consumption:

    https://hackweek.suse.com/13/projects/131
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Tomas Chvatal
Dne Pá 11. prosince 2015 10:43:51, Adam Spiers napsal(a):

> Is anyone working on (or thinking of working on) making our build
> process reproducible?
>
>     https://reproducible-builds.org/
>
> It seems Debian and Fedora are already part of the project, and the
> advantages are quite compelling, not just from a security perspective,
> but also due to the potential savings in storage and network
> consumption:
>
>     https://hackweek.suse.com/13/projects/131
Even if we are not directly involved in this we actually mandate our packages
to always generate the same output and bugs reported against this are always
treated properly...

The hackweek I see is more about reducing the rebuilds that get tossed away.

Ie. we now rebuild libreoffice with each update of tumbleweed (because it is
beast) and we rarely "update" it simply it is rebuilt, verified it is the same
and the new packages are thrown away.

Sortly put: report bugs where the package does not generate same stuff every
time and it will be fixed for stuff maintained internaly and should be fixed for
community based stuff.

Tom

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Michael Matz
In reply to this post by Adam Spiers
Hi,

On Fri, 11 Dec 2015, Adam Spiers wrote:

> Is anyone working on (or thinking of working on) making our build
> process reproducible?
>
>     https://reproducible-builds.org/

We have that since about forever as far as easily possible.  The hard part
is changing packages to not depend on things like build time (e.g.
encoding build date/time into strings into executables).  That's not
something you can do generally in a build system, but must be changed in
each and every individual package.


Ciao,
Michael.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Stephan Kulow-3
On 11.12.2015 11:51, Michael Matz wrote:

> Hi,
>
> On Fri, 11 Dec 2015, Adam Spiers wrote:
>
>> Is anyone working on (or thinking of working on) making our build
>> process reproducible?
>>
>>     https://reproducible-builds.org/
>
> We have that since about forever as far as easily possible.  The hard part
> is changing packages to not depend on things like build time (e.g.
> encoding build date/time into strings into executables).  That's not
> something you can do generally in a build system, but must be changed in
> each and every individual package.
>
Unfortunately most of my bug reports are gently rotting ;(

(check summaries starting with build-compare)

Greetings, Stephan



--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Ludwig Nussel
In reply to this post by Michael Matz
Michael Matz wrote:

> On Fri, 11 Dec 2015, Adam Spiers wrote:
>
>> Is anyone working on (or thinking of working on) making our build
>> process reproducible?
>>
>>      https://reproducible-builds.org/
>
> We have that since about forever as far as easily possible.  The hard part
> is changing packages to not depend on things like build time (e.g.
> encoding build date/time into strings into executables).  That's not
> something you can do generally in a build system, but must be changed in
> each and every individual package.

It's probably worth to explain the difference between our reproducible
builds and this new interpretation. SUSE and openSUSE distributions
have always had reproducible builds, for something like 20 years now.
Reproducible in the sense that a packager never builds binaries on his
own system in some magic way and then uploads binaries.

We always build sources server side (nowadays OBS, previously
autobuild). How the build environment has to look like is defined via
BuildRequires in the spec file and settings in the project config on
server side. Moreover, we don't allow packagers to directly build
packages in the distribution's project. There's always a review step
(four eyes principle). Some distributions don't have that and only have
reviews when a package is accepted for the first time.

OBS always re-creates the build environment from scratch for each
package and automatically uses other packages in the same project to set
up that build environment. Ie there's no magic base build system, the
distribution bootstraps itself. Not only on request or mass rebuilds but
fully automatic. So even packages that haven't been submitted for
years are rebuilt with current compilers and libraries. Additionally
every binary rpm produced by obs contains a back reference to the
used sources (in the disturl).

IOW our process and infrastructure guarantees that our packages can
reproducibly be built from source. Everyone can reproduce that by
firing up their own build service and linking to OBS. In that sense
_our build process is reproducible_ and has always been. The terrifying
news here is that other distributions still have to do homework to
even get there.

However, that reproducible-builds.org project wants to achieve an
addition goal and that is to have bit identical rpm results of two
build runs. We trust our process and infrastructure so it wasn't
important to us so far.

The mechanism we have in place to detect unchanged binary rpms
(build-compare) only partially gets us there as it solves a
different problem. build-compare is basically a heuristic to avoid
triggering rebuilds of other packages and publishing of packages
with only trivial changes (like time stamps). It does that by
"normalizing" some content and throwing away results with trivial
changes. It does not necessarily prevent trivial changes.
Some of the measurements to please build-compare are also useful for
bit identical results though, like eliminating time stamps in build
results in the first place.

So if anyone wants to pursue that reproducible-builds.org idea,
fixing the issues with build-compare is a good first step.

Also, it needs to be defined what a bit identical rpm result
actually is. Only the rpm payload isn't sufficient as the header
includes e.g. scripts. The full header can't be compared as it
contains time stamps, build host names, signatures etc. So again a
script similar to what build-compare does would be needed to do the
actual comparison, md5sums of rpm files won't suffice.

Another project related to that might be to verify source rpms can
be rebuilt. So far our assumption is that a build service is
available. So everything is centered around build service packages
containers rather than srpms.

cu
Ludwig

--
  (o_   Ludwig Nussel
  //\
  V_/_  http://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Richard Biener
On Wed, 16 Dec 2015, Ludwig Nussel wrote:

> Michael Matz wrote:
> > On Fri, 11 Dec 2015, Adam Spiers wrote:
> >
> > > Is anyone working on (or thinking of working on) making our build
> > > process reproducible?
> > >
> > >      https://reproducible-builds.org/
> >
> > We have that since about forever as far as easily possible.  The hard part
> > is changing packages to not depend on things like build time (e.g.
> > encoding build date/time into strings into executables).  That's not
> > something you can do generally in a build system, but must be changed in
> > each and every individual package.
>
> It's probably worth to explain the difference between our reproducible
> builds and this new interpretation. SUSE and openSUSE distributions
> have always had reproducible builds, for something like 20 years now.
> Reproducible in the sense that a packager never builds binaries on his
> own system in some magic way and then uploads binaries.
>
> We always build sources server side (nowadays OBS, previously
> autobuild). How the build environment has to look like is defined via
> BuildRequires in the spec file and settings in the project config on
> server side. Moreover, we don't allow packagers to directly build
> packages in the distribution's project. There's always a review step
> (four eyes principle). Some distributions don't have that and only have
> reviews when a package is accepted for the first time.
>
> OBS always re-creates the build environment from scratch for each
> package and automatically uses other packages in the same project to set
> up that build environment. Ie there's no magic base build system, the
> distribution bootstraps itself. Not only on request or mass rebuilds but
> fully automatic. So even packages that haven't been submitted for
> years are rebuilt with current compilers and libraries. Additionally
> every binary rpm produced by obs contains a back reference to the
> used sources (in the disturl).
>
> IOW our process and infrastructure guarantees that our packages can
> reproducibly be built from source. Everyone can reproduce that by
> firing up their own build service and linking to OBS. In that sense
> _our build process is reproducible_ and has always been. The terrifying
> news here is that other distributions still have to do homework to
> even get there.

Well, I don't see how you can "easily" reproduce a build of an
"old" version of a package without re-bootstrapping the whole
old state of the repository to build against.  I understand that
we eventually have enough information in the package to actually
re-create that repository "soruce" state and we keep old sources
around but not binaries.

Not sure if reproducible-builds.org intends to make that any easier
than what we have though.

Richard.

--
Richard Biener <[hidden email]>
SUSE LINUX GmbH, GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Ludwig Nussel
Richard Biener wrote:

> On Wed, 16 Dec 2015, Ludwig Nussel wrote:
>> [...]
>> IOW our process and infrastructure guarantees that our packages can
>> reproducibly be built from source. Everyone can reproduce that by
>> firing up their own build service and linking to OBS. In that sense
>> _our build process is reproducible_ and has always been. The terrifying
>> news here is that other distributions still have to do homework to
>> even get there.
>
> Well, I don't see how you can "easily" reproduce a build of an
> "old" version of a package without re-bootstrapping the whole
> old state of the repository to build against.  I understand that
> we eventually have enough information in the package to actually
> re-create that repository "soruce" state and we keep old sources
> around but not binaries.

Depends on whether one wants to reproduce some Tumbleweed package
from a random point in the past or one from a released distribution.
For the latter we do keep binaries of released maintenance updates.
Nevertheless it should be possible to reproduce binaries in any
case, even though it might require some time and computing power.

cu
Ludwig

--
  (o_   Ludwig Nussel
  //\
  V_/_  http://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Johannes Meixner
In reply to this post by Ludwig Nussel

Hello Ludwig,

I think your great explanation about the SUSE and openSUSE
build process should be made obvious for everybody out there.

Perhaps it could be placed on some of our web front pages
at least on
https://build.opensuse.org/
and then linked on other web front pages like
https://en.opensuse.org/Main_Page

I would be a shame if not everybody out there knows about
what On Dec 16 11:18 Ludwig Nussel wrote:

>
> ... SUSE and openSUSE distributions
> have always had reproducible builds, for something like 20 years now.
> Reproducible in the sense that a packager never builds binaries on his
> own system in some magic way and then uploads binaries.
>
> We always build sources server side (nowadays OBS, previously
> autobuild). How the build environment has to look like is defined via
> BuildRequires in the spec file and settings in the project config on
> server side. Moreover, we don't allow packagers to directly build
> packages in the distribution's project. There's always a review step
> (four eyes principle). Some distributions don't have that and only have
> reviews when a package is accepted for the first time.
>
> OBS always re-creates the build environment from scratch for each
> package and automatically uses other packages in the same project to set
> up that build environment. Ie there's no magic base build system, the
> distribution bootstraps itself. Not only on request or mass rebuilds but
> fully automatic. So even packages that haven't been submitted for
> years are rebuilt with current compilers and libraries. Additionally
> every binary rpm produced by obs contains a back reference to the
> used sources (in the disturl).
>
> IOW our process and infrastructure guarantees that our packages can
> reproducibly be built from source. Everyone can reproduce that by
> firing up their own build service and linking to OBS. In that sense
> _our build process is reproducible_ and has always been. The terrifying
> news here is that other distributions still have to do homework to
> even get there.


Kind Regards
Johannes Meixner
--
SUSE LINUX GmbH - GF: Felix Imendoerffer, Jane Smithard,
Graham Norton - HRB 21284 (AG Nuernberg)

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Jérémy Bobbio
In reply to this post by Ludwig Nussel
Hi!

Ludwig Nussel:

> Michael Matz wrote:
> >On Fri, 11 Dec 2015, Adam Spiers wrote:
> >
> >>Is anyone working on (or thinking of working on) making our build
> >>process reproducible?
> >>
> >>     https://reproducible-builds.org/
> >
> >We have that since about forever as far as easily possible.  The hard part
> >is changing packages to not depend on things like build time (e.g.
> >encoding build date/time into strings into executables).  That's not
> >something you can do generally in a build system, but must be changed in
> >each and every individual package.
>
> It's probably worth to explain the difference between our reproducible
> builds and this new interpretation. SUSE and openSUSE distributions
> have always had reproducible builds, for something like 20 years now.
> Reproducible in the sense that a packager never builds binaries on his
> own system in some magic way and then uploads binaries.
> […]
Thanks Ludwig for your explanations as we unfortunately use some common
words to convey different ideas. I want to try to elaborate on them so
we can better colaborate between projects as we—in Tor, Debian and many
other projects—strongly believe that these issues concern all free
software projects.

I believe we have trouble understanding one another because, from my
understanding, we have been working on solving different problems:

The problem openSUSE has solved with OBS is “Can a software be rebuilt
from source?”

The problem Bitcoin and Tor Browser have solved is “Can users verify
that the binary has been built using the source distributed with it?”
That's the problem which prompted efforts in Debian and other projects,
and which lead to the creation of reproducible-builds.org.


From the point of view of an openSUSE developper, it might look the
same. As only source code can be submitted to OBS, as long as one trust
the infrastructure, they can be confident than the binary produced by
OBS is coming from the source code that lands in the archive.

But end users might want to verify by themselves that binaries match a
source code they can audit. When a build process doesn't produce
bit-for-bit identical results, this become much harder.

Having the ability to have third parties assess binary packages also
relieve the infrastructure maintainers from potential pressures. It's
easier to resist by saying “if we change that binary, someone is going
to notice”.

> The mechanism we have in place to detect unchanged binary rpms
> (build-compare) only partially gets us there as it solves a
> different problem. build-compare is basically a heuristic to avoid
> triggering rebuilds of other packages and publishing of packages
> with only trivial changes (like time stamps). It does that by
> "normalizing" some content and throwing away results with trivial
> changes. It does not necessarily prevent trivial changes.
> Some of the measurements to please build-compare are also useful for
> bit identical results though, like eliminating time stamps in build
> results in the first place.
Speaking of build-compare, the tool that we built to debug differences
between two builds, diffoscope, is getting wider exposure and becoming
quite versatile. It's in Python and designed to be as modular as
possible. The hope was to have something more maintainable and
extensible than a single shell script.

One feature diffoscope is missing to become a proper alternative to
build-compare is the ability to filter differences. It's on my list of
things to implement, and I'll eventually get to it, but contributions
would be great if anyone is interested.

In any cases, don't hesitate to get in touch. We, people behind Debian
reproducible builds efforts and reproducible-builds.org, want to help!
:)

--
Lunar                                .''`.
[hidden email]                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'`
                                      `-  

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Olaf Hering-2
On Thu, Dec 17, Jérémy Bobbio wrote:

> Speaking of build-compare, the tool that we built to debug differences
> between two builds, diffoscope, is getting wider exposure and becoming
> quite versatile. It's in Python and designed to be as modular as
> possible. The hope was to have something more maintainable and
> extensible than a single shell script.

The (unwritten) goal of build-compare is that it must be usable in every
released product. One just '_link's that package into his build project
and it will just work. Perhaps a replacement in python may be equally
compatible across the various python variants.

Olaf
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Ludwig Nussel
In reply to this post by Jérémy Bobbio
Jérémy Bobbio wrote:

> Ludwig Nussel:
>> Michael Matz wrote:
>>> On Fri, 11 Dec 2015, Adam Spiers wrote:
>>>
>>>> Is anyone working on (or thinking of working on) making our build
>>>> process reproducible?
>>>>
>>>>      https://reproducible-builds.org/
>>>
>>> We have that since about forever as far as easily possible.  The hard part
>>> is changing packages to not depend on things like build time (e.g.
>>> encoding build date/time into strings into executables).  That's not
>>> something you can do generally in a build system, but must be changed in
>>> each and every individual package.
>>
>> It's probably worth to explain the difference between our reproducible
>> builds and this new interpretation. SUSE and openSUSE distributions
>> have always had reproducible builds, for something like 20 years now.
>> Reproducible in the sense that a packager never builds binaries on his
>> own system in some magic way and then uploads binaries.
>> […]
>
> Thanks Ludwig for your explanations as we unfortunately use some common
> words to convey different ideas.

Exactly. The Maximum RPM book already advertised "Reproducible Builds" in
1997 without promising bit identical results :-)

> I want to try to elaborate on them so
> we can better colaborate between projects as we—in Tor, Debian and many
> other projects—strongly believe that these issues concern all free
> software projects.
>
> I believe we have trouble understanding one another because, from my
> understanding, we have been working on solving different problems:
>
> The problem openSUSE has solved with OBS is “Can a software be rebuilt
> from source?”

Yes, we take that for granted.

> [...]
> But end users might want to verify by themselves that binaries match a
> source code they can audit. When a build process doesn't produce
> bit-for-bit identical results, this become much harder.

No doubt about that.

> Speaking of build-compare, the tool that we built to debug differences
> between two builds, diffoscope, is getting wider exposure and becoming
> quite versatile. It's in Python and designed to be as modular as
> possible. The hope was to have something more maintainable and
> extensible than a single shell script.
>
> One feature diffoscope is missing to become a proper alternative to
> build-compare is the ability to filter differences. It's on my list of
> things to implement, and I'll eventually get to it, but contributions
> would be great if anyone is interested.

To be usable within build roots it would have to be sufficiently
standalone though. Due to the automatic boot strapping and always fresh
build roots we have to be careful to avoid polluting build environments
and causing build cycles.

cu
Ludwig

--
  (o_   Ludwig Nussel
  //\
  V_/_  http://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Bernhard M. Wiedemann-5
In reply to this post by Adam Spiers
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015-12-11 11:43, Adam Spiers wrote:

> Is anyone working on (or thinking of working on) making our build
> process reproducible?
>
> https://reproducible-builds.org/
>
> It seems Debian and Fedora are already part of the project, and
> the advantages are quite compelling, not just from a security
> perspective, but also due to the potential savings in storage and
> network consumption:
>
> https://hackweek.suse.com/13/projects/131
>

now I had some time to look into how reproducible our builds are
and had a VM (with 4 cores) build all Leap 42.1 packages named a*

using some helper scripts, which I uploaded to
https://github.com/bmwiedemann/reproducibleopensuse

with those helpers, I just had to do
rebuildmany a*
comparemany a*

btw: the rebuild of those 217 packages took 382 minutes

of those 217 (a small subset of Leap's 7830),
build-compare reported a diff for 36 packages

and if you wonder, those are
a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy
antlr apache-commons-cli apache-commons-codec
apache-commons-collections apache-commons-email apache-commons-io
apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking
argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el
autogen autoyast2 avfs avogadro avrdude awesome axis


The other thing I did was to look how common the use of the better
SOURCE_DATE_EPOCH is.
For that I used
find openSUSE\:Factory/ -name \*.gz -o -name \*.xz -o -name \*.bz2 |\
  grep -v "\.osc" | xargs zgrep -l SOURCE_DATE_EPOCH

to find that SOURCE_DATE_EPOCH is already used in
doxygen
deja-dup
help2man
u-boot


another thing I found is that when you use
 osc getbinaries
you get a file named _buildenv and that contains unique IDs of all
packages used for building this, so it should be possible to archive
and later fetch the exact versions of everything needed.


Ciao
Bernhard M.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlbEw3IACgkQSTYLOx37oWQycQCgwiuCeD34UFMSyHaC8kr5pmnk
FGwAn0yyO0H+vAch4er3jtU+1XQLJeMQ
=uCT4
-----END PGP SIGNATURE-----
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Christian Boltz-5
Hello,

Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:

> now I had some time to look into how reproducible our builds are
> and had a VM (with 4 cores) build all Leap 42.1 packages named a*
>
> using some helper scripts, which I uploaded to
> https://github.com/bmwiedemann/reproducibleopensuse
>
> with those helpers, I just had to do
> rebuildmany a*
> comparemany a*
>
> btw: the rebuild of those 217 packages took 382 minutes
>
> of those 217 (a small subset of Leap's 7830),
> build-compare reported a diff for 36 packages
>
> and if you wonder, those are
> a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy
> antlr apache-commons-cli apache-commons-codec
> apache-commons-collections apache-commons-email apache-commons-io
> apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking
> argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el
> autogen autoyast2 avfs avogadro avrdude awesome axis

I just checked the Debian result for some of them on
https://tests.reproducible-builds.org/reproducible.html
and interestingly, several of the packages you listed above are
reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl,
aspell, autogen, avogadro, avrdude.

I know the reproducible builds team at Debian is actively pushing
patches to make the build reproducible, but I'm not sure if they were
able to fix that many packages already (10 out of 36 - actually I should
probably say 10 out of ~25 because some of the packages you listed
don't exist in Debian, at least not with this name).

Can you upload the differences for the packages you tested somewhere so
that interested people don't need to rebuild everything just to get the
diff?

Oh, BTW - maybe having some variables in your scripts instead of
hardcoded paths would be nice ;-)



I was curious and tried with aspell. The compare result is:


/dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell
/dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell
Comparing aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm
comparing rpmtags
--- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J  2016-02-17 21:34:31.060962628 +0100
+++ /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48  2016-02-17 21:34:31.068962571 +0100
@@ -9,7 +9,7 @@
 has many other technical enhancements over Ispell, such as using shared
 memory for dictionaries and intelligently handling personal
 dictionaries when more than one Aspell process is open at once.
- openSUSE openSUSE Leap 42.1 obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications
+ openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications
  Productivity/Text/Spell http://aspell.net/ (none) (none) (none)
  (none) 4.11.2 x86_64-suse-linux
  cpio lzma 5
comparing RELEASE
comparing PROVIDES
comparing scripts
comparing filelist
comparing file checksum
creating rename script
RPM meta information is identical
Extracting packages
Package content is identical

[...same for other subpackages...]


It seems the only difference is in the rpmtags - (none) vs.
obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell

Is this really a relevant difference, or should it be whitelisted in
build-compare?

Hmm, actually $?=0, so on my system it _is_ reproducible. What did I do
wrong? ;-)



I also tried aespipe, and get a real difference. Maybe someone should
compare the openSUSE package with the Debian package to find out what
they do different to make the package reproducible ;-)


Regards,

Christian Boltz
--
> 204°C währen jawohl das Todesurteil für meine Platte, oder?
Wenn es keine Kochplatte ist, schon...
[Jürgen Fahnenschreiber und Manfred Tremmel in suse-linux]

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Damian Ivanov
Doing these builds on OBS wouldn't they always differ?
I mean the release tag in .rpm's is always incremented which already
results in a diff.
In my opinion only the actual files contained in the rpm and maybe the
scriptlets contained should be compared
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Bernhard M. Wiedemann-2
In reply to this post by Christian Boltz-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2016-02-17 22:07, Christian Boltz wrote:

> Hello,
>
> Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
>> now I had some time to look into how reproducible our builds are
>> and had a VM (with 4 cores) build all Leap 42.1 packages named
>> a*
>>
>> using some helper scripts, which I uploaded to
>> https://github.com/bmwiedemann/reproducibleopensuse
>>
>> with those helpers, I just had to do rebuildmany a* comparemany
>> a*
>>
>> btw: the rebuild of those 217 packages took 382 minutes
>>
>> of those 217 (a small subset of Leap's 7830), build-compare
>> reported a diff for 36 packages
>>
>> and if you wonder, those are a2ps acct aegisub aespipe allegro
>> alpine amanda amor anjuta anthy antlr apache-commons-cli
>> apache-commons-codec apache-commons-collections
>> apache-commons-email apache-commons-io apache-commons-lang
>> apache-portlet-1_0-api apel appframework aqbanking argyllcms arts
>> asl asm3 aspell aspell-en atmel-firmware autoconf-el autogen
>> autoyast2 avfs avogadro avrdude awesome axis
>
> I just checked the Debian result for some of them on
> https://tests.reproducible-builds.org/reproducible.html and
> interestingly, several of the packages you listed above are
> reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl,
> aspell, autogen, avogadro, avrdude.
>
> I know the reproducible builds team at Debian is actively pushing
> patches to make the build reproducible, but I'm not sure if they
> were able to fix that many packages already (10 out of 36 -
> actually I should probably say 10 out of ~25 because some of the
> packages you listed don't exist in Debian, at least not with this
> name).

I think, they are very active and use latest upstream versions,
possibly with extra patches, while I tested with Leap, which is an
older codebase, but Factory was too fast-moving for me to test with atm.

Some diffs could also be caused by the fact that we do not always
rebuild all dependent packages on changes. That could be avoided by
doing it like the debian guys - rebuilding twice in two very different
contexts.

> Can you upload the differences for the packages you tested
> somewhere so that interested people don't need to rebuild
> everything just to get the diff?

sure. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/

> Oh, BTW - maybe having some variables in your scripts instead of
> hardcoded paths would be nice ;-)

yep, it is an early alpha and certainly on the TODO list.

> I was curious and tried with aspell. The compare result is:
>
>
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell Comparing
> aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm
> comparing rpmtags --- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J
> 2016-02-17 21:34:31.060962628 +0100 +++
> /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48  2016-02-17 21:34:31.068962571
> +0100 @@ -9,7 +9,7 @@ has many other technical enhancements over
> Ispell, such as using shared memory for dictionaries and
> intelligently handling personal dictionaries when more than one
> Aspell process is open at once. - openSUSE openSUSE Leap 42.1
> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell

> GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications
> GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications +
> openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND
> and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and
> SUSE-BSD-Mark-Modifications Productivity/Text/Spell
> http://aspell.net/ (none) (none) (none) (none) 4.11.2
> x86_64-suse-linux cpio lzma 5 comparing RELEASE comparing PROVIDES
> comparing scripts comparing filelist comparing file checksum
> creating rename script RPM meta information is identical Extracting
> packages Package content is identical
>
> [...same for other subpackages...]
>
>
> It seems the only difference is in the rpmtags - (none) vs.
> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
>
>  Is this really a relevant difference, or should it be whitelisted
> in build-compare?

that is what I did with
https://github.com/bmwiedemann/reproducibleopensuse/blob/master/build-co
mpare.diff
but not yet in a nice upstreamable way.

for aspell (and only this 1 of 36) I found, that is is reproducible
when building with
osc build --vm-type=kvm
but when using the default chroot, it shows

comparing filelist
@@ -1,8 +1,8 @@
 /usr/bin/aspell 0 (none) 100755 root root 0 4294967295
 /usr/bin/aspell-import 0 (none) 100755 root root 0 4294967295
- -/usr/bin/precat 0 (none) 100755 root root 0 4294967295
- -/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 precat
- -/usr/bin/prezip 0 (none) 120777 root root 0 4294967295 precat
+/usr/bin/precat 0 (none) 120777 root root 0 4294967295 prezip
+/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 prezip
+/usr/bin/prezip 0 (none) 100755 root root 0 4294967295
 /usr/bin/prezip-bin 0 (none) 100755 root root 0 4294967295
 /usr/bin/run-with-aspell 0 (none) 100755 root root 0 4294967295
 /usr/bin/word-list-compress 0 (none) 100755 root root 0 4294967295
comparing file checksum
creating rename script
RPM file checksum differs.
Extracting packages
@@ -1 +0,0 @@
- -precat
symlink target for /usr/bin/prezip differs

probably from the brp-symlink doing things differently in that context.

> Hmm, actually $?=0, so on my system it _is_ reproducible. What did
> I do wrong? ;-)
>
>
>
> I also tried aespipe, and get a real difference. Maybe someone
> should compare the openSUSE package with the Debian package to find
> out what they do different to make the package reproducible ;-)

e.g.
https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/alpine-c
ompare.out
shows an embedded ASCII date in 2 binaries.


Ciao
Bernhard M.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlbFe7AACgkQSTYLOx37oWQJ8gCeM7uFpKuf77sb7iVHc7q8DMw6
BWQAmwYevLzk4Tu169Np3JFVUu5oDdHS
=A5R4
-----END PGP SIGNATURE-----
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Andreas Schwab-2
In reply to this post by Christian Boltz-5
Christian Boltz <[hidden email]> writes:

> It seems the only difference is in the rpmtags - (none) vs.
> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell

This is the the DISTURL, if that changes you have different sources.
But since you didn't build in OBS you have no DISTURL.

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Stephan Kulow-3
On 18.02.2016 09:33, Andreas Schwab wrote:
> Christian Boltz <[hidden email]> writes:
>
>> It seems the only difference is in the rpmtags - (none) vs.
>> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
>
> This is the the DISTURL, if that changes you have different sources.
> But since you didn't build in OBS you have no DISTURL.
>
I guess we need to special case (none) then for use outside of OBS.

Greetings, Stephan



--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Bernhard M. Wiedemann-5
In reply to this post by Christian Boltz-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2016-02-17 22:07, Christian Boltz wrote:

> Hello,
>
> Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
>> now I had some time to look into how reproducible our builds are
>> and had a VM (with 4 cores) build all Leap 42.1 packages named
>> a*
>>
>> using some helper scripts, which I uploaded to
>> https://github.com/bmwiedemann/reproducibleopensuse
>>
>> with those helpers, I just had to do rebuildmany a* comparemany
>> a*
>>
>> btw: the rebuild of those 217 packages took 382 minutes
>>
>> of those 217 (a small subset of Leap's 7830), build-compare
>> reported a diff for 36 packages
>>
>> and if you wonder, those are a2ps acct aegisub aespipe allegro
>> alpine amanda amor anjuta anthy antlr apache-commons-cli
>> apache-commons-codec apache-commons-collections
>> apache-commons-email apache-commons-io apache-commons-lang
>> apache-portlet-1_0-api apel appframework aqbanking argyllcms
>> arts asl asm3 aspell aspell-en atmel-firmware autoconf-el
>> autogen autoyast2 avfs avogadro avrdude awesome axis
>
> I just checked the Debian result for some of them on
> https://tests.reproducible-builds.org/reproducible.html and
> interestingly, several of the packages you listed above are
> reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl,
> aspell, autogen, avogadro, avrdude.
>
> I know the reproducible builds team at Debian is actively pushing
> patches to make the build reproducible, but I'm not sure if they
> were able to fix that many packages already (10 out of 36 -
> actually I should probably say 10 out of ~25 because some of the
> packages you listed don't exist in Debian, at least not with this
> name).

I think, they are very active and use latest upstream versions,
possibly with extra patches, while I tested with Leap, which is an
older codebase, but Factory was too fast-moving for me to test with atm.

Some diffs could also be caused by the fact that we do not always
rebuild all dependent packages on changes. That could be avoided by
doing it like the debian guys - rebuilding twice in two very different
contexts.

> Can you upload the differences for the packages you tested
> somewhere so that interested people don't need to rebuild
> everything just to get the diff?

sure. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/

> Oh, BTW - maybe having some variables in your scripts instead of
> hardcoded paths would be nice ;-)

yep, it is an early alpha and certainly on the TODO list.

> I was curious and tried with aspell. The compare result is:
>
>
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell
> /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell Comparing
> aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm
> comparing rpmtags --- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J 2016-02-17
> 21:34:31.060962628 +0100 +++ /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48
> 2016-02-17 21:34:31.068962571 +0100 @@ -9,7 +9,7 @@ has many other
> technical enhancements over Ispell, such as using shared memory for
> dictionaries and intelligently handling personal dictionaries when
> more than one Aspell process is open at once. - openSUSE openSUSE
> Leap 42.1
> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
>
>
GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications

> GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications +
> openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND
> and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND
> and SUSE-BSD-Mark-Modifications Productivity/Text/Spell
> http://aspell.net/ (none) (none) (none) (none) 4.11.2
> x86_64-suse-linux cpio lzma 5 comparing RELEASE comparing PROVIDES
> comparing scripts comparing filelist comparing file checksum
> creating rename script RPM meta information is identical
> Extracting packages Package content is identical
>
> [...same for other subpackages...]
>
>
> It seems the only difference is in the rpmtags - (none) vs.
> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
>
>
>
Is this really a relevant difference, or should it be whitelisted
> in build-compare?

that is what I did with
https://github.com/bmwiedemann/reproducibleopensuse/blob/master/build-co
mpare.diff
but not yet in a nice upstreamable way.

for aspell (and only this 1 of 36) I found, that is is reproducible
when building with
osc build --vm-type=kvm
but when using the default chroot, it shows

comparing filelist
@@ -1,8 +1,8 @@
 /usr/bin/aspell 0 (none) 100755 root root 0 4294967295
 /usr/bin/aspell-import 0 (none) 100755 root root 0 4294967295
- -/usr/bin/precat 0 (none) 100755 root root 0 4294967295
- -/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 precat
- -/usr/bin/prezip 0 (none) 120777 root root 0 4294967295 precat
+/usr/bin/precat 0 (none) 120777 root root 0 4294967295 prezip
+/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 prezip
+/usr/bin/prezip 0 (none) 100755 root root 0 4294967295
 /usr/bin/prezip-bin 0 (none) 100755 root root 0 4294967295
 /usr/bin/run-with-aspell 0 (none) 100755 root root 0 4294967295
 /usr/bin/word-list-compress 0 (none) 100755 root root 0 4294967295
comparing file checksum
creating rename script
RPM file checksum differs.
Extracting packages
@@ -1 +0,0 @@
- -precat
symlink target for /usr/bin/prezip differs

probably from the brp-symlink doing things differently in that context.

> Hmm, actually $?=0, so on my system it _is_ reproducible. What did
> I do wrong? ;-)
>
>
>
> I also tried aespipe, and get a real difference. Maybe someone
> should compare the openSUSE package with the Debian package to
> find out what they do different to make the package reproducible
> ;-)

e.g.
https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/alpine-c
ompare.out
(and some others)
shows an embedded ASCII date in 2 binaries.


Ciao
Bernhard M.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlbFi4UACgkQSTYLOx37oWQh8wCgnJuISPwhGh7mS+uv8q1KKSY4
du4AoOWdV4d+5nRSLUuaQXsqBWAe81/d
=yMBl
-----END PGP SIGNATURE-----
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Marcus Meissner
In reply to this post by Stephan Kulow-3
Hi,

One thing, do we keep a list of packages which do not reproducible build(-compare)
that can be processed manually at leisure?

Because I think there is some manual work to do.

Ciao, Marcus

On Thu, Feb 18, 2016 at 09:57:46AM +0100, Stephan Kulow wrote:

> On 18.02.2016 09:33, Andreas Schwab wrote:
> > Christian Boltz <[hidden email]> writes:
> >
> >> It seems the only difference is in the rpmtags - (none) vs.
> >> obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
> >
> > This is the the DISTURL, if that changes you have different sources.
> > But since you didn't build in OBS you have no DISTURL.
> >
> I guess we need to special case (none) then for use outside of OBS.
>
> Greetings, Stephan
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: reproducible builds

Andreas Schwab-2
In reply to this post by Bernhard M. Wiedemann-5
"Bernhard M. Wiedemann" <[hidden email]> writes:

> probably from the brp-symlink doing things differently in that context.

It probably depends on how files are ordered in the directory, which
differs between filesystems.

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

123