Tumbleweed - Review of the week 2018/03

classic Classic list List threaded Threaded
104 messages Options
123456
Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Hadrien Grasland
Le 22/01/2018 à 18:16, John Paul Adrian Glaubitz a écrit :

> On 01/22/2018 05:38 PM, Hadrien Grasland wrote:
>> ...that is, until you end up on a codebase which:
>>
>>   * Has -Werror hardcoded deep into its build system.
>
> That doesn't make any sense. You can always override/amend
> CFLAGS/CXXFLAGS. There
> is no such thing as "deeply hardcoded".
>
>>   * Relies on code which does not follow the C/++ standard to get
>>     compiled by accident.
>
> Never seen that.
>
>>   * Relies on undefined behaviour, or another codegen characteristic
>>     that is considered an unimportant detail by GCC's optimizer.
>
> Never seen that.
>
>>   * Uses compiler version detection macros which have not been adapted
>>     to the new GCC release.
>
> That would speak of a very poor build system. Yet, I don't think I have
> run into such a problem.
>
>>   * Causes an ICE in that specific release of GCC, which was introduced
>>     by accident in what was supposed to be a simple bugfix.
>
> Very rare. So far I have only seen such problems on less common
> architectures
> and it was always a breeze to get these things fixed with upstream.

It looks like you have enjoyed pretty well-written and unambitious C/++
code so far, then. Lucky you! Where I work, broken build systems, code
and compilers are a relatively common sight, I'd say we deal with them
every other month or so, and that is with a package base is much smaller
than the repos of SuSE or Debian!


>> Relying on UB especially happens more often than one would think, and
>> is basically the kernel of truth behind the old "-O3 breaks code" meme.
>>
>> No matter which way one looks at it, compiler updates are
>> unfortunately always a bit of a risky business from a Linux
>> distribution maintainer's point of view.
>
> Thanks, but I have helped with several gcc transitions in Debian. I
> never saw anything
> there as close as with Rust. The changes in gcc actually made sense to
> me, as I said,
> I was always able to address them with either very simple patches or
> by just disabling
> a certain warning.
>
> What about the fact that Rust only considers x86/x86_64 to be a tier 1
> architecture?

In Mozilla's terminology, "tier 2" means "guaranteed to build" and "tier
1" means "and in addition, all automated tests were run". The reason why
you would want to only run the build is that running tests is much more
trouble than building, because you can build for any architecture from
x86 using a cross-compiler, whereas you need real hardware on the target
architecture in order to perform serious testing (as emulators are
usually too slow to be practical in intensive testing scenarios, and too
"clean" to expose real hardware quirks).

Assuming you wanted to build yourself a cross-architecture test farm,
capable of withstanding the full traffic of Rust's high-volume CI
system, what you would soon discover is that most hardware architectures
do not address this need very well. It is trivial to find a hardware
reseller who will build you a good x86-based rack at a fair price,
whereas other architectures often do not provide hardware in a standard
rack form factor at all, or only sell hardware at a crazy premium like
IBM does with Power. Moreover, embedded architectures also often
restrict themselves to cheaper and slower hardware which is not powerful
enough for intensive continuous testing, meaning that you need to pile
up tons of un-rackable junk before you get enough processing power for
this kind of use case...

Add to this that keeping a highly heterogeneous hardware base running is
very cumbersome, and that some of Rust's tier 2 architectures do not
even provide the required capabilities for running a test server (e.g.
asmjs/wasm is too limited, Fuschia is too immature, and iOS is too much
locked down), and hopefully you will get a fair picture of how much of
an undertaking this all really is.

Now, this isn't to say that it cannot be done, of course. Nor that it
would not be very worthwhile. There are some awesome multi-architecture
test beds out there, like Debian's package QA test bed or Microsoft's
driver compatibility torture test farm, and I'm pretty sure Novell also
have some cool stuff around for testing SuSE too. But I think that level
of QA sophistication may be a bit much to expect from a relatively small
team inside of a money-starved nonprofit organization. If someone is
ready to donate or lend Mozilla the required infrastructure, great, but
if not, I would not expect them to build it on their own...


> We have just recently seen with Spectre and Meltdown how bad it is to
> merely focus
> on x86.

I think you may want to check the latest developments of the
Meltdown/Spectre saga here. Meltdown, it turns out, goes beyond Intel
processors (AMD remaining unaffected) and also hits some high-end ARM
processors. And Spectre attacks have been demonstrated on pretty much
every modern CPU which has a cache and speculative execution features.
It is not an x86 versus the rest of the world thing, almost every
popular high-performance CPU architecture has been demonstrated to be
vulnerable to these attacks in some way, and all high-performance CPU
manufacturers now needs to reflect upon these events and figure out how
to build a more secure product next time...

Cheers,
Hadrien

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Jan Engelhardt-4

On Monday 2018-01-22 21:30, Hadrien Grasland wrote:

> Le 22/01/2018 à 18:16, John Paul Adrian Glaubitz a écrit :
>>
>>>   * Causes an ICE in that specific release of GCC, which was introduced
>>>     by accident in what was supposed to be a simple bugfix.
>>
>> Very rare. So far I have only seen such problems on less common architectures
>> and it was always a breeze to get these things fixed with upstream.
>
> It looks like you have enjoyed pretty well-written and unambitious C/++ code so
> far, then. Lucky you! Where I work, broken build systems, code and compilers
> are a relatively common sight, I'd say we deal with them every other month or
> so, and that is with a package base is much smaller than the repos of SuSE or
> Debian!

Are you working in academia/science? There's quite a bit of bad code coming out
of that field. But, a distribution is comprised of less than 20% (number pulled
out of thin air) packages from obs://science, so indeed we don't experience as
many WPM[1] as you may be ;-)


[1] http://www.osnews.com/story/19266/WTFs_m
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Hadrien Grasland
Le 22/01/2018 à 23:28, Jan Engelhardt a écrit :

> On Monday 2018-01-22 21:30, Hadrien Grasland wrote:
>> Le 22/01/2018 à 18:16, John Paul Adrian Glaubitz a écrit :
>>>>    * Causes an ICE in that specific release of GCC, which was introduced
>>>>      by accident in what was supposed to be a simple bugfix.
>>> Very rare. So far I have only seen such problems on less common architectures
>>> and it was always a breeze to get these things fixed with upstream.
>> It looks like you have enjoyed pretty well-written and unambitious C/++ code so
>> far, then. Lucky you! Where I work, broken build systems, code and compilers
>> are a relatively common sight, I'd say we deal with them every other month or
>> so, and that is with a package base is much smaller than the repos of SuSE or
>> Debian!
> Are you working in academia/science? There's quite a bit of bad code coming out
> of that field. But, a distribution is comprised of less than 20% (number pulled
> out of thin air) packages from obs://science, so indeed we don't experience as
> many WPM[1] as you may be ;-)
>
>
> [1] http://www.osnews.com/story/19266/WTFs_m

You're right, it's probably that :) My largest package management
experience comes from HEP experiments, where we try to run quite poorly
written and bleeding edge (C++14/17) software on very... ahem...
*stable* RedHat releases, and it doesn't end very well. We basically end
up maintaining everything but the kernel and libc ourselves in order to
get sufficiently recent compilers and libs.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Hadrien Grasland
In reply to this post by John Paul Adrian Glaubitz
Le 22/01/2018 à 18:29, John Paul Adrian Glaubitz a écrit :

> On 01/22/2018 05:12 PM, Hadrien Grasland wrote:
>> The Rust compiler uses unstable internal interfaces, which are not
>> exposed to code which builds on stable releases. The closest
>> equivalent which I can think of in the C/++ world is the GCC/binutils
>> duo: to build and use GCC, you need a matching release of binutils,
>> which maps to a relatively narrow time window. Use too new or too old
>> a release of binutils, and your GCC build will fail with weird
>> assembler and linker errors. And conversely, like any C/++ program,
>> binutils itself has some compiler version requirements.
>
> I'm pretty confident there is never a problem when binutils is too new,
> at least I haven't run into such a problem with my porting work within
> Debian.

I remember reading that too new a binutils could also be a problem back
in the days where I was playing with OSdeving (which requires a custom
cross-compiler configuration), but you are right that I never
experienced it firsthand nor have heared about it in a long while. Maybe
it's just an old meme fueled by a couple of backwards-incompatible
binutils breakages that happened a long time ago...


> And, furthermore, the key point here again is the speed of change: gcc
> doesn't introduce breaking changes every six weeks, Rust does.

I can certainly understand that fast-changing software can be difficult
to deal with. I for one never understood how the people packaging
rolling release distros manage to keep up so well with the rapid rate of
kernel releases (and the periodical NVidia driver breakages that ensue),
or with the rapid update frequency of anything GCC-related (where you
basically have to rebuild everything every time).

At the same time, one should not shoot the messenger. Fresh software
with fast feature and bugfix turnaround is also a good thing from the
end user point of view, so long as the project can provide the quality
assurance guarantees that come with that. And for this, Rust comes much
better equipped than many other projects, as you can read up on
https://brson.github.io/2017/07/10/how-rust-is-tested .

Now, you claim that this is not enough, and that you have observed
breakages. But when I have requested you to provide details and
evidence, you have not been able to (and have in fact ignored that
question altogether). All you have precisely explained in your previous
e-mail is that you had issues bootstrapping the compiler using an older
release of itself, a special case which the Rust team is well aware of,
and provides special support for through pre-built binaries. You have
also briefly mentioned something about a disappearing keyword, without
mentioning if that keyword was part of a stable Rust release (which is
where Rust's stability guarantees apply) or not. If you are not going to
provide further details, I will have to assume that it wasn't.


>>> The problem with Rust is simply the lack of stabilization. It's
>>> absolutely
>>> insane that they think it's ok to break compatibility in minor versions
>>> and it blows my mind that so many people find that acceptable.
>>
>> Adding features in a minor software release is considered okay in any
>> modern software versioning scheme. It is only when existing features
>> are changed or removed that compatibility is considered to be broken.
>
> I wouldn't consider a toolchain a normal piece of software. A
> toolchain is
> one of the basic building blocks of your whole distribution. It shouldn't
> change in crazy ways when you just perform a minor update.

Tell that to the kernel maintainers next time they will break my video
driver or send someone's production system in an infinite bootloop in
what was supposed to be a security update. And yet, for some reason, we
in the Linux world never had much issue building on top of that. In
fact, I would argue that one of Tumbleweed's strength is that is the
first Linux distribution which I have used so far which provides
concrete answers to this problem (via OpenQA and Btrfs snapshots)
without forcing its users into software stagnation along the way.

Compared to what major Linux infrastructure projects like the kernel,
Mesa, or KDE will still periodically send people through, I would say
that Rust did pretty well so far. It has managed to iteratively add many
features on a rapid release cycle, without breaking the code of anyone
who builds on stable releases, as evidenced by the team regularly
re-building and re-testing the entire crates.io package library in their
routine procedure.

You claim that you found holes in this procedure, but so far you have
not provided evidence. And even if you had some, all that would mean is
that your discovery would serve to improve the testing procedure and
make it better. I, for one, am very happy that some software projects
are finally taking steps to provide alternatives to the breakage versus
stagnation false dichotomy, that has been with us for way too long in
the Linux world.


>>> Rust upstream lives in a universe where they think that distributions
>>> are an outdated concept. This is why they are shipping their own
>>> package manager and consider such breaking changes in minor releases
>>> acceptable.
>>
>> You must understand where they are coming from. Most Linux
>> distributions consider it okay to ship software which lags 5+ years
>> behind official upstream releases, which is not acceptable for a
>> fast-moving software project like Rust (or even to any software
>> project where new releases matter, such as hardware drivers, web
>> browsers, and office suites). And some of the platforms that they
>> target do not ship a standard package management mechanism at all.
>> The rolling release users among us are sadly the minority here.
>
> Well, your perspective would change if you're responsible for maintaining
> several hundreds desktop machines with several hundred users. Installing
> a rolling release distribution in such setups would be a nightmare
> because
> you would be busy all day long to fix all kinds of regressions.
>
> And I'm not necessarily talking about regressions in the form of bugs. It
> can already be a regression if feature X behaves differently or extension
> Y doesn't work anymore.
>
> It's really frustrating how many upstream projects are refusing to
> understand
> this. So many just say "Awww, just go ahead and update to the latest
> upstream
> version, no big deal. I've been running Arch on my single-user,
> single-machine
> setup for years without problems." It simply doesn't work that way in the
> enterprise world.

Again, there are two sides to this story. Here, you are taking the side
of someone who needs to keep a large production system alive, which I
agree is very important and must be respected. At the same time, if you
put yourselves in a developer's shoes, it is also extremely frustrating
to process bug reports or feature requests about problems which you
resolved on the master branch months ago, and to be requested to keep
alive old software releases which no one even really wants to be using
anymore. Surely, there has to be a way to do better here on both accounts.

I am glad to see that some software projects are taking steps to resolve
this longstanding issue more cleanly. To see better testing and
deployment infrastructures which shrink the risk window, and reduce the
need for costly backports. Things like more extensive test suites,
better continuous integration, containers and staged feature roll-out
are all great news for the software world, which will ultimately all
help us leave more of the legacy baggage behind, and stop saying to
people "Well, you *could* run Linux, open newer DOCX documents, or write
C++17 code on that freshly bought laptop, but for that you will need to
take some risks...".


>> Rust's distribution tools cater to the vast majority of users who are
>> stuck with obsolete operating system packages and want to get modern
>> work done nonetheless. To do this, they sometimes need to bypass the
>> standard distribution package management scheme. But this need not
>> concern you as a distribution maintainer, much like you need not be
>> concerned about users who build and install more recent software
>> releases from source: what users do with their machine is solely
>> their business, so long as they don't come complain when their
>> personal fiddling breaks the system.
>
> It very much becomes concern if a new version of application X requires
> an additional of 250 packages to be updated. It becomes a nightmare from
> a security point of view. Who is going to review all these additional
> updated packages?
>
> What's the point of all these fancy security features Rust have when you
> end up having 25 different versions of libfoo installed on your system?
>
> You might as well then just stop installing security updates.

For any software stabilization and testing process, there is a point of
diminishing returns. No matter how much energy you expend at reviewing
the package base, at some point, you will still need to bite the bullet
and push the thing to the users, being fully aware that this is where
the bulk of the bugs and security holes will be found just by virtue of
users being much more numerous than testers.

For this reason, I've grown increasingly skeptical of stable Linux
distribution release processes over time. They have never been effective
at killing the most annoying bugs for me (like broken hardware drivers),
all the while forcing me into ancient software whose problems have been
fixed upstream months to years ago. Their upgrade procedures are
stressful, time-consuming and fragile.

I am aware that there is a place for such elaborate release schemes, but
personally I would rather see all that QA effort being expended directly
on maintenance and continuous improvement of the relevant software
projects, rather than on making late software later.

Cheers,
Hadrien

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

John Paul Adrian Glaubitz
In reply to this post by Alberto Planas Dominguez
On 01/22/2018 07:02 PM, Alberto Planas Dominguez wrote:
> As a mater of fact there are some changes that were not backward
> compatible, but those were bugs that affected the soundness of the
> compiler, and the usage was not very common. In any case I am not aware
> of any project that was hit by those changes.

No, I have been there, done that. It's currently too frustrating.

>> Different speed is a huge understatement, a seriously huge
>> understatement.
>
> Well C++03, C++11, C++14, C++17. You are right, but is not fair to
> compare a language that is 35 y.o (from 1983) with one that is from
> 2015.

And that's why I think that fundamental packages like librsvg shouldn't
be ported to a language from 2015 which is still subject to quick
movement. We are not talking about some random leaf package here.

I don't know how to run this check in openSUSE, but look at the packages
whose build dependencies would become uninstallable on most architectures
in Debian if we were to upgrade librsvg to the Rust version:

> https://people.debian.org/~glaubitz/librsvg.txt

You don't think this is a problem?

>> And you didn't even address the problem that Rust upstream
>> effectively
>> doesn't care about anything besides x86/x86_64.
>
> Sure. Clearly FF is not running on ARM on Android. I think that you
> have another missunderstanding on Tier 2 concept here [1]
>
> [1] https://forge.rust-lang.org/platform-support.html
Thanks for trying to paint me as an uneducated person. But just in case
you didn't read the link you posted:

> Tier 2 platforms can be thought of as “guaranteed to build”. **Automated tests are not run so it’s not guaranteed to produce a working build**,
> but platforms often work to quite a good degree and patches are always welcome! Specifically, these platforms are required to have each of the following:
>
>    Official binary releases are provided for the platform.
>    **Automated building is set up, but may not be running tests.**
>    Landing changes to the rust-lang/rust repository’s master branch is gated on platforms building. For some platforms only the standard library is compiled, but for others rustc and cargo are too.

I don't think we would accept gcc to be not passing it's testsuite on any
of the platforms openSUSE supports. Apparently, it's acceptable for Rust.

And before you want to accuse me of more incompetence: In Debian, I'm a porter
for most for most of the unofficial architectures. I think I can say I have
somewhat an experience in this field.

PS: I didn't know that SLE only supports x86_64 and ARM :).

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

John Paul Adrian Glaubitz
In reply to this post by Hadrien Grasland
On 01/22/2018 09:30 PM, Hadrien Grasland wrote:
> In Mozilla's terminology, "tier 2" means "guaranteed to build" and "tier 1" means "and in addition, all automated tests were run". The reason why you would want to only run the build is that running tests is much more trouble than building, because you can build for any architecture from x86 using a cross-compiler, whereas you need real hardware on the target architecture in order to perform serious testing (as emulators are usually too slow to be practical in intensive testing scenarios, and too "clean" to expose real hardware quirks).

See, and that is the difference with downstream distributions. They use real
hardware to build **and** run the testsuite.. In Debian, cross-building, for
example, cross-building is not allowed for the supported release architectures
for very good reasons.

What you are saying is not a justification, it's merely an explanation. Go
upstream, for example, runs the builds and testsuites on all their supported
platforms natively. They do not accept ports for which this criteria cannot
be met:

https://build.golang.org/

Any supported target which is unable to pass the testsuites for a longer
period is dropped from Golang.

They take testing and integration much more seriously than Rust does.

> Assuming you wanted to build yourself a cross-architecture test farm, capable of withstanding the full traffic of Rust's high-volume CI system, what you would soon discover is that most hardware architectures do not address this need very well. It is trivial to find a hardware reseller who will build you a good x86-based rack at a fair price, whereas other architectures often do not provide hardware in a standard rack form factor at all, or only sell hardware at a crazy premium like IBM does with Power. Moreover, embedded architectures also often restrict themselves to cheaper and slower hardware which is not powerful enough for intensive continuous testing, meaning that you need to pile up tons of un-rackable junk before you get enough processing power for this kind of use case...

We have that in Debian. I'm not sure why you are trying to educate me here.

> Add to this that keeping a highly heterogeneous hardware base running is very cumbersome, and that some of Rust's tier 2 architectures do not even provide the required capabilities for running a test server (e.g. asmjs/wasm is too limited, Fuschia is too immature, and iOS is too much locked down), and hopefully you will get a fair picture of how much of an undertaking this all really is.

Again, look at this:

> https://build.golang.org/#
> https://jenkins.debian.net/view/rebootstrap/
> https://buildd.debian.org/status/package.php?p=gcc-7&suite=sid

You are talking to someone who is working as a build engineer at Debian.

> Now, this isn't to say that it cannot be done, of course. Nor that it would not be very worthwhile. There are some awesome multi-architecture test beds out there, like Debian's package QA test bed or Microsoft's driver compatibility torture test farm, and I'm pretty sure Novell also have some cool stuff around for testing SuSE too. But I think that level of QA sophistication may be a bit much to expect from a relatively small team inside of a money-starved nonprofit organization. If someone is ready to donate or lend Mozilla the required infrastructure, great, but if not, I would not expect them to build it on their own...

Maybe Mozilla should understand at some point then that they're not Google.

>> We have just recently seen with Spectre and Meltdown how bad it is to merely focus
>> on x86.
>
> I think you may want to check the latest developments of the Meltdown/Spectre saga here. Meltdown, it turns out, goes beyond Intel processors (AMD remaining unaffected) and also hits some high-end ARM processors. And Spectre attacks have been demonstrated on pretty much every modern CPU which has a cache and speculative execution features. It is not an x86 versus the rest of the world thing, almost every popular high-performance CPU architecture has been demonstrated to be vulnerable to these attacks in some way, and all high-performance CPU manufacturers now needs to reflect upon these events and figure out how to build a more secure product next time...

> https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/

PS: I have patches both in Rust upstream and Mozilla (around 30 patches) upstream before
     you again are trying to paint me as someone uneducated on the subject.

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

John Paul Adrian Glaubitz
In reply to this post by Hadrien Grasland
On 01/23/2018 08:46 AM, Hadrien Grasland wrote:
>> And, furthermore, the key point here again is the speed of change: gcc
>> doesn't introduce breaking changes every six weeks, Rust does.
>
> I can certainly understand that fast-changing software can be difficult to deal with. I for one never understood how the people packaging rolling release distros manage to keep up so well with the rapid rate of kernel releases (and the periodical NVidia driver breakages that ensue), or with the rapid update frequency of anything GCC-related (where you basically have to rebuild everything every time).
>
> At the same time, one should not shoot the messenger. Fresh software with fast feature and bugfix turnaround is also a good thing from the end user point of view, so long as the project can provide the quality assurance guarantees that come with that. And for this, Rust comes much better equipped than many other projects, as you can read up on https://brson.github.io/2017/07/10/how-rust-is-tested .

I am not shooting the messenger. I am criticizing the person who thought that
writing a core package with a large number of reverse dependencies [1] in a
language where upstream can't even be bothered to run the testsuite on anything
beyond x86 is a good idea. And, no, not having the resources for that is not
the right justification. If you don't have the resources, either a) don't try
to push your language for core packages, or b), ask projects like Debian which do
have a large test infrastructure with all kinds of architectures

>> I wouldn't consider a toolchain a normal piece of software. A toolchain is
>> one of the basic building blocks of your whole distribution. It shouldn't
>> change in crazy ways when you just perform a minor update.
>
> Tell that to the kernel maintainers next time they will break my video driver or send someone's production system in an infinite bootloop in what was supposed to be a security update. And yet, for some reason, we in the Linux world never had much issue building on top of that. In fact, I would argue that one of Tumbleweed's strength is that is the first Linux distribution which I have used so far which provides concrete answers to this problem (via OpenQA and Btrfs snapshots) without forcing its users into software stagnation along the way.

If the kernel breaks, I can just switch to a different kernel at the boot
prompt. For this very reason, Debian puts every minor kernel release
into a separate package.

Furthermore, the distribution kernels don't bring such breaking changes,
plus the upstream kernel also NEVER breaks any userland.

> Again, there are two sides to this story. Here, you are taking the side of someone who needs to keep a large production system alive, which I agree is very important and must be respected. At the same time, if you put yourselves in a developer's shoes, it is also extremely frustrating to process bug reports or feature requests about problems which you resolved on the master branch months ago, and to be requested to keep alive old software releases which no one even really wants to be using anymore. Surely, there has to be a way to do better here on both accounts.

You need to understand that the people who are paying everyone's bills at
the end of the month are the one that are using the stable releases.

In our other mail, you were saying that Mozilla is a starving organization,
maybe you should try to make the connection between these two statements.

> For any software stabilization and testing process, there is a point of diminishing returns. No matter how much energy you expend at reviewing the package base, at some point, you will still need to bite the bullet and push the thing to the users, being fully aware that this is where the bulk of the bugs and security holes will be found just by virtue of users being much more numerous than testers.

I think you don't have the slightest clue how QA in enterprise distributions
works or how much QA and testing there is before Debian pushes a stable
release.

This isn't about biting a bullet and pushing something out untested, there is
A LOT of testing behind. This is why companies pay very good money for it.

> For this reason, I've grown increasingly skeptical of stable Linux distribution release processes over time. They have never been effective at killing the most annoying bugs for me (like broken hardware drivers), all the while forcing me into ancient software whose problems have been fixed upstream months to years ago. Their upgrade procedures are stressful, time-consuming and fragile.

The key point about stable distributions is not that they are bug-free, the key
point is that the bugs and problems are well documented. A rapid release will
always bring new regressions.

Adrian

> [1] https://people.debian.org/~glaubitz/librsvg.txt
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

John Paul Adrian Glaubitz
In reply to this post by Hadrien Grasland
On 01/23/2018 07:08 AM, Hadrien Grasland wrote:
> You're right, it's probably that :) My largest package management experience comes from HEP experiments, where we try to run quite poorly written and bleeding edge (C++14/17) software on very... ahem... *stable* RedHat releases, and it doesn't end very well. We basically end up maintaining everything but the kernel and libc ourselves in order to get sufficiently recent compilers and libs.
>

And that's not the background experience you should base your statements regarding distribution maintenance on.
I am a physicist myself and I have worked with scientific code. Lots of that stuff causes heart attacks when
reading through the code or build system. Using such packages for reference to argue that C/C++ is not a stable
language and eco system, is dishonest, to say the least.

C/C++, on the other hand, is still much better supported across the open source ecosystem. The Linux kernel
supports more than 30 architectures, all of them are supported by gcc as well. And this portability was
one of the key features why Linux is so successful these days.

If we start porting many core packages to Rust, we are severely limiting the portability of Linux which
is a very bad decision, in my opinion. Linux has effectively zero relevance on the desktop, but it's
the dominating platform in embedded systems. Pushing a language which has no or limited support for
the majority of embedded architectures (x86 is basically not existing in this market) is not a very
good idea.

And who tells me that the "starving organization" Mozilla is going to support Linux in the
future? The vast majority of Firefox users are on Windows. Mozilla supporting Linux is not
very viable from an economic point of view, so I would not be surprised if they dropped
Linux support for this very reason.

To quote long-time Linux kernel developer Geert Uytterhoeven:

  "There's lots of Linux beyond ia32"

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

Alberto Planas Dominguez
In reply to this post by John Paul Adrian Glaubitz
On Tue, 2018-01-23 at 10:37 +0100, John Paul Adrian Glaubitz wrote:
> On 01/22/2018 07:02 PM, Alberto Planas Dominguez wrote:
> > As a mater of fact there are some changes that were not backward
> > compatible, but those were bugs that affected the soundness of the
> > compiler, and the usage was not very common. In any case I am not
> > aware
> > of any project that was hit by those changes.
>
> No, I have been there, done that. It's currently too frustrating.

In that case this is anothera different issue. Can I assume by the
content that is librsvg the one that breaks from one stable version to
another?

We are diverging for the main topic. Your assert is that Rust is
willing to break code that target stable-Rust in almost each release,
and I think that this is a missunderstanding about the expectations
that are fair to use agains the compiler. That is all.

> I don't know how to run this check in openSUSE, but look at the
> packages
> whose build dependencies would become uninstallable on most
> architectures
> in Debian if we were to upgrade librsvg to the Rust version:
>
> > https://people.debian.org/~glaubitz/librsvg.txt
>

`zypper info` can help here.

> You don't think this is a problem?
>
> > > And you didn't even address the problem that Rust upstream
> > > effectively
> > > doesn't care about anything besides x86/x86_64.
> >
> > Sure. Clearly FF is not running on ARM on Android. I think that you
> > have another missunderstanding on Tier 2 concept here [1]
> >
> > [1] https://forge.rust-lang.org/platform-support.html
>
> Thanks for trying to paint me as an uneducated person. But just in
> case
> you didn't read the link you posted:

Sorry if I make this impression. I didn't choose the right words. What
I try to say is that tier 2 is not exacly "doesn't care about anything besides x86/x86_64". Is simply reflecting that some automatic tests are not running for each commit to the compiler, but is expected to work. And in the case of Android and ARM is clearly working.

> And before you want to accuse me of more incompetence: In Debian, I'm
> a porter
> for most for most of the unofficial architectures.

Did I? Sorry if I made this impression!

--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
Maxfeldstraße 5, 90409 Nürnberg, Germany
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

John Paul Adrian Glaubitz
In reply to this post by John Paul Adrian Glaubitz


On 01/23/2018 10:37 AM, John Paul Adrian Glaubitz wrote:
>> Tier 2 platforms can be thought of as “guaranteed to build”. **Automated tests are not run so it’s not guaranteed to produce a working build**,
>> but platforms often work to quite a good degree and patches are always welcome! Specifically, these platforms are required to have each of the following:
>>
>>    Official binary releases are provided for the platform.
>>    **Automated building is set up, but may not be running tests.**
>>    Landing changes to the rust-lang/rust repository’s master branch is gated on platforms building. For some platforms only the standard library is compiled, but for others rustc and cargo are too.

Oh, and to add to that: In Debian, Firefox is now breaking more often
on platforms which are not x86/x86_64 as you can see here with armhf
currently broken:

> https://buildd.debian.org/status/package.php?p=firefox&suite=sid

But I really don't know what people expect from a compiler whose
testsuite is only ever run on x86/x86_64 by upstream - unlike virtually
any other compiler project on the planet. Heck, even Free Pascal
runs tests on all platforms - natively. I know that because I am providing
those guys hardware for testing.

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

Hadrien Grasland


Le 23/01/2018 à 11:33, John Paul Adrian Glaubitz a écrit :

>
>
> On 01/23/2018 10:37 AM, John Paul Adrian Glaubitz wrote:
>>> Tier 2 platforms can be thought of as “guaranteed to build”.
>>> **Automated tests are not run so it’s not guaranteed to produce a
>>> working build**,
>>> but platforms often work to quite a good degree and patches are
>>> always welcome! Specifically, these platforms are required to have
>>> each of the following:
>>>
>>>    Official binary releases are provided for the platform.
>>>    **Automated building is set up, but may not be running tests.**
>>>    Landing changes to the rust-lang/rust repository’s master branch
>>> is gated on platforms building. For some platforms only the standard
>>> library is compiled, but for others rustc and cargo are too.
>
> Oh, and to add to that: In Debian, Firefox is now breaking more often
> on platforms which are not x86/x86_64 as you can see here with armhf
> currently broken:
>
>> https://buildd.debian.org/status/package.php?p=firefox&suite=sid
>
> But I really don't know what people expect from a compiler whose
> testsuite is only ever run on x86/x86_64 by upstream - unlike virtually
> any other compiler project on the planet. Heck, even Free Pascal
> runs tests on all platforms - natively. I know that because I am
> providing
> those guys hardware for testing.
>
> Adrian

Then maybe your time would be better spent working together with the
Rust team to ensure that they can and do run their tests on armhf as well?

Hadrien

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

Andreas Schwab-2
In reply to this post by John Paul Adrian Glaubitz
On Jan 23 2018, John Paul Adrian Glaubitz <[hidden email]> wrote:

> I don't know how to run this check in openSUSE, but look at the packages
> whose build dependencies would become uninstallable on most architectures
> in Debian if we were to upgrade librsvg to the Rust version:

These are the direct dependencies:
osc whatdependson openSUSE:Factory librsvg standard x86_64

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

Hadrien Grasland
In reply to this post by Hadrien Grasland


Le 23/01/2018 à 11:42, Hadrien Grasland a écrit :

>
>
> Le 23/01/2018 à 11:33, John Paul Adrian Glaubitz a écrit :
>>
>>
>> On 01/23/2018 10:37 AM, John Paul Adrian Glaubitz wrote:
>>>> Tier 2 platforms can be thought of as “guaranteed to build”.
>>>> **Automated tests are not run so it’s not guaranteed to produce a
>>>> working build**,
>>>> but platforms often work to quite a good degree and patches are
>>>> always welcome! Specifically, these platforms are required to have
>>>> each of the following:
>>>>
>>>>    Official binary releases are provided for the platform.
>>>>    **Automated building is set up, but may not be running tests.**
>>>>    Landing changes to the rust-lang/rust repository’s master branch
>>>> is gated on platforms building. For some platforms only the
>>>> standard library is compiled, but for others rustc and cargo are too.
>>
>> Oh, and to add to that: In Debian, Firefox is now breaking more often
>> on platforms which are not x86/x86_64 as you can see here with armhf
>> currently broken:
>>
>>> https://buildd.debian.org/status/package.php?p=firefox&suite=sid
>>
>> But I really don't know what people expect from a compiler whose
>> testsuite is only ever run on x86/x86_64 by upstream - unlike virtually
>> any other compiler project on the planet. Heck, even Free Pascal
>> runs tests on all platforms - natively. I know that because I am
>> providing
>> those guys hardware for testing.
>>
>> Adrian
>
> Then maybe your time would be better spent working together with the
> Rust team to ensure that they can and do run their tests on armhf as
> well?
>
> Hadrien
>
PS: By the way, if you go and have a look at the build log for the
firefox package on armhf that you mention, you will find that the
problem is that the build machine ran out of RAM ("terminate called
after throwing an instance of 'std::bad_alloc'"). This has nothing to do
with Rust, it is a common problem with large C++ projects too. You may
want to beef up the corresponding build node or to reduce the amount of
concurrent build processes.
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Aleksa Sarai
In reply to this post by John Paul Adrian Glaubitz
On 2018-01-22, John Paul Adrian Glaubitz <[hidden email]> wrote:

> > I'm not sure we're understanding each other here -- my point was that
> > the *only* Rust project which has this policy for compiling new versions
> > is the Rust compiler. No other Rust project requires this. That's what I
> > meant by "exception, not the rule". So I agree with what you wrote, but
> > it doesn't have much to do with what I was trying to say, which is that
> > the following quote ...
>
> So, you say it's guaranteed that only the Rust compiler will only ever use
> particular code that will be deprecated in release N+1 or only available
> in release N-1?
I'll be honest that I am not sure of the reason why the Rust compiler is
special in this sense. However, from the discussions I've had with folks
from the Rust community, it's related to the fact that the standard
library is "special" (in the same way that Go's "runtime" and "internal"
libraries are quite special) and it should not be possible for a crate
(that is built on stable) to have the same properties as the Rust
compiler.

> I did build test it myself. I tried building Rust 1.22 with Rust 1.20
> which failed with actual compiler errors, not just a warning that I have
> to use the proper version. And I think it's absolutely not unlikely that
> Rust project X will run into such a problem as well. What keeps Rust
> project X from using certain language features that were only recently
> added or removed?
>
> The problem with Rust is simply the lack of stabilization. It's absolutely
> insane that they think it's ok to break compatibility in minor versions
> and it blows my mind that so many people find that acceptable.
As has been discussed in the rest of this thread, Rust has a very
specific stability guarantee which is effectively following:

  "Any Rust crate which builds on a stable version of Rust today is
  guaranteed to build on a future stable version of Rust tomorrow. If it
  does not, it's a bug."

You have given two examples where this was not true:

* The Rust compiler, which as we've discussed is not a "Rust crate".

* A Firefox version which was broken by the removal of some keyword.
  From what you describe this sounds like a bug, and a violation of
  their stability guarantee. Mistakes happen -- though this one seems to
  be quite large to not have been mentioned anywhere else -- and I hope
  that you submitted a bug report, because I'm sure that they would've
  resolved the problem.

You mention that Go does a lot of testing to avoid regressions, *so does
the Rust community*. They do a "crater run" (rebuild and unit-test of
all crates on crates.io) on a regular cadence during development, when
large features are being considered for merging, and for gating
releases.

If they find an issue they either fix it in the compiler (if it's a
regression) or they go to the project itself and submit a patch (if it's
actually a bug in the project). This is one of the things that is
mentioned in literally every talk about the Rust development process.

If they really did break Firefox, then I would like to see a post-mortem
for why their regular crater runs didn't pick it up. But the right thing
to do in this situation would've been to report a bug, rather than to
conclude that Rust is unstable and move on without telling anyone.

> Rust upstream lives in a universe where they think that distributions
> are an outdated concept. This is why they are shipping their own
> package manager and consider such breaking changes in minor releases
> acceptable.

I agree that their attitude toward distributions is a problem (for us
obviously, but also for users IMO). But this is also an attitude that is
shared by a very large number of languages and projects these days.

I cannot help but think that the root cause of this problem is that we
have not done a good job (as distribution developers in general) of
convincing people of the benefits of having a distribution that has a
global view of packaging of a system.

> > > > Finally, one problem with Rust that I ran into when working on the
> > > > Rust compiler code itself is the high volatility of the language and the
> > > > code. Things that used to build fine with Rust 1.19 would break with
> > > > 1.20 and so on. From a distribution maintainer's point of view, this
> > > > can be very tedious and annoying.
> >
> > ... is not accurate for any project other than the Rust compiler (and
> > the reason for the Rust compiler having this requirement is so that they
> > can use new language features in the compiler itself, not because of
> > stability issues with the language). Any other Rust project must be able
> > to build on 1.x and 1.(x+1) with no changes (and the Rust compiler team
> > tests this quite heavily).
>
> What keeps project X from using certain features of Rust? I have seen
> projects which would only build with Rust Nightly.
I think us trying to ship projects that use Rust Nightly would be
absolute madness (I also think it's mad to depend on Rust Nightly
features for a "stable" project, but that's a separate topic). However,
we can avoid the nightly problem by not shipping things that depend on
nightly (neither Firefox nor the library that spawned this discussion
depend on Nightly).

However, Rust Nightly != Rust Stable. The "certain features" you refer
to are unstable features that cannot be used if you compile with the
stable compiler.

> > > So, if SLE wants to update to the next ESR release of Firefox, it
> > > will also have to include Rust in the same maintenance request.
> >
> > We ship a new Go package with most new Docker releases as well (as they
> > usually update the compiler they use every couple of releases, and in
> > the past there were bugs where Docker would break if it was built with
> > the wrong compiler version). This is not really a new thing.
>
> I don't think you always need the latest version of Go for updating
> Docker. I have worked with both codebases myself and never ran into
> this issue.
Not always, but there have been cases (we're talking ~2 years ago) where
a Go upgrade has broken Docker in subtle ways. This was before runc was
a separate project, and they were still using "os/exec" for parts of
container setup (which obviously proved to be a horrible idea, and now
we have the whole nsexec magic that I'm tasked with maintaining upstream).

These days most new Docker releases use a new Go version, and so we
bundle a Go update "just in case" (and usually they do actually use
newer language features).

> > > Upstream Go always ensures that golang-go can be built with gcc-go. Rust
> > > has mrustc for that, but that one isn't supporting anything beyond
> > > x86 at the moment.
> >
> > Hmmm, this is a policy that appears to have changed after the compiler
> > rewrite to Go. I distinctly remember watching a talk where rsc described
> > that they would require you to have version (n-1) of the compiler in
> > order to build version n -- so that the compiler could take advantage of
> > new language features -- and you would start the bootstrapping at go1.4.
>
> Huh? If it changed after the compiler rewrite in Go, wouldn't that mean
> that before that the compiler wasn't written in Go which means that you
> didn't have that problem in the first place?
I meant that the policy they *planned* to have was the whole (n-1)
bootstrap thing, and the *policy* appears to have changed after I looked
into it. If you'd like to see what I mean, you can Google rsc's talks
about the Go compiler rewrite from a few years ago.

--
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Aleksa Sarai
In reply to this post by John Paul Adrian Glaubitz
On 2018-01-23, John Paul Adrian Glaubitz <[hidden email]> wrote:
> Furthermore, the distribution kernels don't bring such breaking changes,
> plus the upstream kernel also NEVER breaks any userland.

Except when they do -- in which case we chalk it up as a mistake, Linus
gets angry at someone, and we all move along with our day. We don't
suddenly start shouting that "Linux is unstable!".

--
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Jan Engelhardt-4
On Tuesday 2018-01-23 12:18, Aleksa Sarai wrote:

>On 2018-01-23, John Paul Adrian Glaubitz <[hidden email]> wrote:
>> Furthermore, the distribution kernels don't bring such breaking changes,
>> plus the upstream kernel also NEVER breaks any userland.
>
>Except when they do -- in which case we chalk it up as a mistake, Linus
>gets angry at someone, and we all move along with our day. We don't
>suddenly start shouting that "Linux is unstable!".

But the thing is - why do we always need to get angry first before
problems (perceived or real) like rsvg get fixed, whatever the fix may
be?
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

John Paul Adrian Glaubitz
In reply to this post by Alberto Planas Dominguez
On 01/23/2018 11:26 AM, Alberto Planas Dominguez wrote:
> In that case this is anothera different issue. Can I assume by the
> content that is librsvg the one that breaks from one stable version to
> another?
>
> We are diverging for the main topic. Your assert is that Rust is
> willing to break code that target stable-Rust in almost each release,
> and I think that this is a missunderstanding about the expectations
> that are fair to use agains the compiler. That is all.

No, we are not.

Two main points are still valid:

1) Rust is not as stable as it should be for core packages.
2) Rust doesn't consider architectures which SUSE considers
    supported as supported. This is definitely a problem.

>> I don't know how to run this check in openSUSE, but look at the
>> packages
>> whose build dependencies would become uninstallable on most
>> architectures
>> in Debian if we were to upgrade librsvg to the Rust version:
>>
>>> https://people.debian.org/~glaubitz/librsvg.txt
>>
>
> `zypper info` can help here.

I don't think you can use zypper to determine the build dependencies
for OBS, can you? The list I obtained for Debian came through the
build system, not anything local on my disk.

> https://wiki.debian.org/ftpmaster_Removals#Reverse_Dependencies

> Sorry if I make this impression. I didn't choose the right words. What
> I try to say is that tier 2 is not exacly "doesn't care about anything besides x86/x86_64". Is simply reflecting that some automatic tests are not running for each commit to the compiler, but is expected to work. And in the case of Android and ARM is clearly working.

And you are missing the point. Working now doesn't mean it's not going
to break for the next version.

Again, a compiler which is not verified to pass it's testsuite should
not be used for production code in my experience. I have seen too many
things break on the buildds in Debian when the testsuites for gcc were
disabled.

Not having Rust upstream build the compiler natively and run the testsuite
on architectures that SUSE considers to be supported is a problem in
my experience.

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tumbleweed - Review of the week 2018/03

Michal Kubecek
In reply to this post by Aleksa Sarai
On Tuesday, 23 January 2018 12:18 Aleksa Sarai wrote:
> On 2018-01-23, John Paul Adrian Glaubitz <[hidden email]>
wrote:
> > Furthermore, the distribution kernels don't bring such breaking
> > changes, plus the upstream kernel also NEVER breaks any userland.
>
> Except when they do -- in which case we chalk it up as a mistake,
> Linus gets angry at someone, and we all move along with our day. We
> don't suddenly start shouting that "Linux is unstable!".

Not exactly. Linus rarely does get (really) angry just because someone
made a mistake and "broke userspace". But when that someone - or someone
else - refuses to fix it and starts to argue that it's actually the
right thing to do, that's when you can expect one of his famous rants.

And this difference is important. If you break users' usecase by
accident, it's a mistake and we all make mistakes. You shouldn't make
serious ones too often, sure, but nobody is going to rip your head for
that. But you shouldn't insist that it's perfectly fine and refuse to
fix it.

I'm not sure if you meant it that way but "chalk it up as a mistake ...
and move on" sounds as if you forgot the "we fix it" part which is quite
important.

Michal Kubeček


--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

John Paul Adrian Glaubitz
In reply to this post by Hadrien Grasland
On 01/23/2018 11:42 AM, Hadrien Grasland wrote:
> Then maybe your time would be better spent working together with the
> Rust team to ensure that they can and do run their tests on armhf as well?

Am I the person who is trying to push Rust everywhere?

Rust upstream wants Rust to succeed as a systems programming language. It's
therefore their responsibility to make sure the compiler works properly on
the supported targets.

If they cannot achieve that, they should stop trying to claim how superior
Rust is over other languages and stop trying to push it as a systems
programming language.

I am already doing lots of upstream work in various projects. But this
isn't my main job, so you can't expect me to do the homework of the
Rust developers.

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Rust

John Paul Adrian Glaubitz
In reply to this post by Andreas Schwab-2
On 01/23/2018 11:46 AM, Andreas Schwab wrote:
>> I don't know how to run this check in openSUSE, but look at the packages
>> whose build dependencies would become uninstallable on most architectures
>> in Debian if we were to upgrade librsvg to the Rust version:
>
> These are the direct dependencies:
> osc whatdependson openSUSE:Factory librsvg standard x86_64

Awesome, thank you.

That's what I like about these discussions, they will almost always result
in learning something new :).

Adrian
--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]

123456