[Roadmap 2017] Needs of no-std / embedded developers

[nikomatsakis]

Continuing the discussion from Setting our vision for the 2017 cycle:

I’d like to try something. There were a lot of posts that touched on topics relevant to embedded development. I’d like to get a better understanding of the feature sets that are involved here. I have this impression that embedded development is a kind of chimera: it depends very much on the particulars of your target. For example, some people are working on a Raspberry Pi running node.js, and others are in an environment where there is basically no heap and no memory allocation. Naturally the needs and constraints here will be very different.

Skimming the original roadmap thread I saw a number of related areas that I would like to understand better. I’ll just drop some notes here to get things started. I’m going to include a quote or two from the original thread but by no means am I able to be comprehensive – sorry if I miss your quote. =)

Stabilizing more of the non-std crates and APIs

Setting our vision for the 2017 cycle

One thing that could be improved with #[no_std] is getting some of the core crates (like alloc) stabilized.

I’m a little confused about that one in particular – are we likely to be changing alloc::heap::allocate in the future? It seems pretty basic to me.

Right now, the attribute #[no_std] is stable, but just about anything you can do outside of that is not. It’s true that stabilizing these low-level details has been progressing slowly. One complication is that things which seem pretty basic typically are not – for example, we were holding off on allocate because we thought we might want to supply metadata for tracing GC purposes. However, the latest design that @Manishearth, @pnkfelix, and I have been pursuing has abandoned that direction – and I feel pretty confident we are not going back. It is worth doing a review of some of these APIs and trying to move forward where possible.

Support for a wider range of CPUs, toolchains

Setting our vision for the 2017 cycle

I would love to see firmer support for the existing non-x86 architectures. I feel a bit guilty that I’m only building i686 and x86_64 for Fedora so far, but these are the only Tier 1 targets. Perhaps I should bite the Tier 2 bullet for aarch64 and armv7, but the rest don’t have bootstrap binaries for rustc and cargo, especially ppc64 and ppc64le I’d hope for next.

I often hear requests for making ARM support easier. This is sort of outside my wheelhouse so I’m not an expert on what is required, but I understand that there are many, many ARM families, so even this request is really a myriad of requests. Frankly, making things into a Tier 1 platform (meaning, fully tested) puts a big strain on our testing resources (we have many, many buildbots going 24x7 as it is), so I think we have to be selective here, and from what I understand many ARM architectures are already Tier 2 (meaning we distribute binaries).

Are there other ARM families we should support? Other steps needed beyond building binaries? I saw some complaints about opaque linker errors (I hear you! My eyes glaze over immediately).

Probably @alexcrichton or @brson should really be writing this paragraph, because they know what they are talking about here and I don’t, but I’m the one doing it so oh well.

Handling allocation failure

Currently Rust (often?) aborts on OOM. @lilith gave a great summary of their concerns here and on issue #27700. I have heard this from others as well. My general feeling is that Rust should almost never abort but rather attempt to panic whenever possible, but there are some places that we do abort now that (e.g., double unwind, stack overflow) – and some of them may be hard to change.

I have some questions of my own:

• Would it be sufficient if OOM panics instead of simply aborting?
  • vs, say, having collection methods that return a Result?
• How much do we worry about backwards compatibility here?
  • my feeling is that all unsafe code ought to be exception safe anyhow, but having easy ways to test seems important
  • certainly we can get libstd up to speed
• How important is the notion of “auditing for memory allocation”?
  • I would definitely not be ready to add anything like this to the standard language or distribution – lots of complexity and big design space here – but I think that it is something that could be done externally with relative ease. If anyone wants to chat about this particular project in more detail, they can contact me, but here are some rough notes:
    • This seems very doable today as a kind of lint, though it would probably rely on unstable compiler internals.
    • This sort of analysis is often called an “effect system”.
    • The usual shortfalls are virtual dispatch – e.g., calling a trait object or fn pointer – since it’s hard to know what code will run. Sometimes you may be able to use alias analysis to figure it out, but otherwise you can either be conservative, optimistic, or else start annotating traits and so forth.
• Does it matter if stack overflow aborts?

What’d I miss?

There are like 250 posts in the thread. I must have missed something related to embedded development.

[steveklabnik]

@japaric took the code from intermezzOS/“building an OS in Rust” and asked “why does everyone say that OS dev in Rust still requires a Makefile?” The answer is https://github.com/japaric/eighty-six

One thing that comes up continually for in this space is “float-free libcore”.

inline asm is neglected, and very important for x86 platforms at least.

const fn is very very useful, lots of static muts in OS code, and making Mutex::new const helps keep it safe.

[sfackler]

my feeling is that all unsafe code ought to be exception safe anyhow, but having easy ways to test seems important

I’m a bit confused by this. If it is unreasonable for code to assume that allocation doesn’t panic, is it also unreasonable to assume that a = b; doesn’t panic? Imagine an (absurd) universe where we decided to add Thread::stop and panics could be injected at arbitrary locations. Would we really tell people “well, your code wasn’t exception safe in the first place” then?

Isn’t exception safety all about knowing where you do and don’t need to be careful to checkpoint? If the standard library is broken in a world where allocation can panic, I would be shocked if significant portions of third party code isn’t broken as well.

[nikomatsakis]

If it is unreasonable for code to assume that allocation doesn’t panic, is it also unreasonable to assume that a = b; doesn’t panic?

You’re right of course. What I said was too broad. What I meant was that I think unsafe code should generally assume that most function calls can panic unless that function is under direct control of the unsafe code or there are firm commitments in the documentation – and the latter should be viewed with suspicion. =)

If the standard library is broken in a world where allocation can panic, I would be shocked if significant portions of third party code isn’t broken as well.

Indeed, they may well be. They may also be broken in other ways. Trying to get more hard data here seems important. That said, all these unstable APIs are unstable for a reason – because we are still working out the invariants for them!

But this is why I mentioned having easy ways to test – I would very much like to get to a point with the unsafe code guidelines and other work where you can do “cargo test --sanitize” or something like that and get automated fuzz testing focusing on these kinds of edge cases that can easily be overlooked. This seems to fall into that category.

[jethrogb]

Stabilizing more of the non-std crates and APIs

Yes, and also moving some things that are currently in std into core or into seperate crates, such as io and HashMap (in collections).

[japaric]

Support for a wider range of CPUs, toolchains

@nikomatsakis

I’m not an expert on what is required, but I understand that there are many, many ARM families

Yes, there are a lot.

We currently support, as tier 2, the ARMv6, ARMv7-A and ARMv8-A architecture. These are used in application processors, have CPU frequencies in the GHz range and usually run an OS, e.g. Linux. We provide std, rustc and cargo binaries for these architectures.

Then there is the ARMv6-M and ARMv7-M architectures. These are used in microcontrollers, which have CPU frequency in the low range of MHz (usually less than 100 MHz) and low amount of RAM (usually in the order of dozens of KBs). Right now, we don’t “officially” support these at all but you can write Rust programs for these architecture if you implement your own target definition and build core yourself. There’s a RFC to add target definitions for these architecture to the compiler, as well as start providing binary releases of (a smaller subset of) std for these targets.

There is also the new ARMv8-M architecture, which is an upgrade of the ARMv7-M architecture. AFAIK, there are no implementations of this architecture in the market yet, so there are no reports of anyone using Rust on these devices. However, LLVM already supports the architecture.

There are also the ARMv7-R and ARMv8-R architectures. These are used in “real time” processors but I don’t know anything else about them …

Frankly, making things into a Tier 1 platform (meaning, fully tested) puts a big strain on our testing resources

To add some numbers. Running the unit tests of the “standard” crates (std and all its dependencies) under QEMU (because we don’t have hardware for every single architecture) takes about 15 minutes (on Travis; it would probably take a little less on the buildbots). That’s 15 minutes more PER target on every tested PR. And that’s not enough to elevate the target to tier 1 because we would still have to run the test suite (run-make, compile-fail, etc) of their compilers – that would probably take hours under QEMU.

If there are not enough resources to test these targets on each PR, perhaps we could tests these targets (at least std unit tests) on a “nightly” basis. Would that elevate them to tier … 1.5?

@cuviper

the rest don’t have bootstrap binaries for rustc and cargo, especially ppc64 and ppc64le I’d hope for next.

Sofware-wise, we should already have all the requirements to start producing rustc and cargo binaries for these targets. The issue is again: resources.

What’d I miss?

puts ARM microcontroller hacker hat on

I’d like to see:

• On the fly compilation of standard crates for custom targets using Cargo. Basically, this RFC accepted and implemented. One shouldn’t need to manually compile core and manage their sysroot or, worse, reach out to Makefiles to build a no_std executable for a custom target.

• Either the target definitions for the ARMv*-M architecture built into the compiler or officially distributed as .json files. One shouldn’t need to figure out how to write these and the definition shouldn’t break on a nightly update.

With that, I’ll be happier on nightly. Moving to stable would, at a minimum, require fixing these issues somehow:

• asm! is nightly forever.
• I need lang = "panic_fmt" to build no_std executables.

@steveklabnik

One thing that comes up continually for in this space is “float-free libcore”.

I thought this was fixed with a custom target or with a bunch of codegen options. I wish more x86 kernel/OS devs answered this question I asked a while ago.

[jethrogb]

Would it be sufficient if OOM panics instead of simply aborting?

• vs, say, having collection methods that return a Result?

I think this is a pretty big move away from Rust’s mantra of not using panics for error handling. In addition, unwinding support for many embedded platforms is a big unknown at this time.

[japaric]

Needs of no-std / embedded developers

ccing @alevy. We’ll love to hear the needs of the Tock(OS) project.

[cbiffle]

Yes, and also moving some things that are currently in std into core or into seperate crates, such as io and HashMap.

Separate crates, please — it seems difficult to implement HashMap without making assumptions about the memory allocation model (though correct me if I’m wrong on that), and core is useful because it makes few such assumptions.

@japaric took the code from intermezzOS/“building an OS in Rust” and asked “why does everyone say that OS dev in Rust still requires a Makefile?” The answer is https://github.com/japaric/eighty-six

I hit the same question yesterday and reached the same conclusion as @japaric – I think people tend to hit underdocumented behaviors in the tools and fall back to a Makefile as a well-understood solution. A HOWTO might solve the problem. I’m working on documenting this in my minimal example repo.

I was able to ditch the Makefile in this commit by declaring a bogus start item to mollify the compiler, thus making the program conform to Cargo’s idea of what a binary looks like; I later found #[no_main] and was able to eliminate that too.

I’m happy to help document all this more, I found it pretty hard to discover.

[cbiffle]

There are also the ARMv7-R and ARMv8-R architectures. These are used in “real time” processors but I don’t know anything else about them …

For the purposes of a compiler and language runtime, they are very similar to big-ARM. They are dual-instruction-set parts implementing a large subset of the ARMvx-A ISA. Think of them as spiritual successors to ARM7 (ARMv4T) parts but with more powerful memory protection facilities (that are probably not relevant to this discussion).

Part of the reason I’m focusing my Rust embedded work on ARMv7-M is to avoid some of the thornier questions posed by ARMv7-R/A. For example,

• Interrupt service routines on A/R use a different calling convention; on M they use the C ABI. Different toolchains for A/R may provide a function attribute to control this, or expect all ISRs to be called through assembly veneers.
• Writing very high performance code on A/R requires control over whether a function is generated in ARM or Thumb/Thumb-2 and interworking support. On M, only Thumb-2 is supported, simplifying things considerably.

I use parts from all these families regularly and am happy to answer any questions that arise.

[nikomatsakis]

I think this is a pretty big move away from Rust’s mantra of not using panics for error handling.

This is not the way I view Rust’s error handling strategy. The way that I see the strategy is:

• result for something locally recoverable
• panic for non-local recovery
  • this would ideally occur at some place where the vast majority of memory is isolated, rather like a process
  • for example, at thread boundary is a common choice
  • with -C panic=abort, this boundary is the process
• abort => basically irrecoverable, but certainly process boundary is an operation

The question is “who gets to define the point of non-local recovery”? In general, I’d say the answer is “not you”. It’s typically the outermost app layer. If you are a library, and you want to recover from an error, for maximum compatibility you should use Result.

Anyway, the point of my asking whether it’d be ok to use panic is basically this: how often do we want to locally recover from malloc failures? If the answer is “always”, then I would think we would want Result-based APIs, but that is a heavy cost (lots of duplicate API surface). I’m not sure what’s the best way to handle that scenario.

[jethrogb]

Anyway, the point of my asking whether it’d be ok to use panic is basically this: how often do we want to locally recover from malloc failures? If the answer is “always”, then I would think we would want Result-based APIs, but that is a heavy cost (lots of duplicate API surface). I’m not sure what’s the best way to handle that scenario.

I’ve not given this much thought, but placement-new might be able to support allocation failure much better than the current alloc/collections APIs.

Edit: I’ve given it some more thought now. Placer::make_place could return Result<Place,AllocationError>. Then you could write e.g.

in HEAP? { 5 };

Issue: Tracking issue for placement new

[cuviper]

For my comment on expanding platform support, I’m coming from the distro perspective, just hoping to get Rust going on all platforms we generally support. That’s only slightly related to “embedded” for things like Raspberry Pi, as you mention, so I don’t want to derail the topic here.

It would be a great step just to have more bootstrap-capable platforms in Tier 2, and there’s movement on that front in #36006 and #36015 - thanks @japaric for your efforts there! I understand build and test resources are an issue - I don’t have an answer for that right now.

[withoutboats]

It seems like in the context of the road map, “supporting OOM recovery” is enough without getting into the nitty gritty of how Rust could solve this problem.

However if we want to expose “local recovery from OOM,” I think if heap::allocate or alike is every stabilized in a way that returns a Result or an analogous type, OOM recoverable collections could be constructed outside of std, rather than duplicating the API surface of std.

[Ericson2314]

Ok a lot to say here but most import is that I do not want to rush at all stabilizing things. I don’t think any hobbyist has a problem with nightly ATM, and unlike Rust-in-userspace I think there is far less potential commercial use being put on hold because of stability.

[newpavlov]

Separate crates, please — it seems difficult to implement HashMap without making assumptions about the memory allocation model (though correct me if I’m wrong on that), and core is useful because it makes few such assumptions.

But what assumptions do we need for implementing core::io? It was part of the pre-1.0 IO reform RFC but was not implemented for some reason. Yes, currently we have core_io crate, but it’s not the easiest one to use, especially on the nightly compiler.

[jethrogb]

it’s not the easiest one to use, especially on the nightly compiler.

Sorry, I really need to update it for newer nightly, but it’s not super high on my priority list right now.

[Ericson2314]

As to allocation, I think issue #1 is that allocators take III hasn’t been implemented.

As to results vs panicking, I’d like to remind you @nikomatsakis of our conversation ending in https://github.com/rust-lang/rfcs/pull/1398#issuecomment-209079205 which, now that ! is a real type, means all collections can immediately become allocator-generic with an A: Allocator<Error = !> constraint.

Also, I pointed out in https://github.com/rust-lang/rfcs/pull/1398#issuecomment-163438980 which IIRC might have been lost in the comments that since allocators themselves need to be created in systems level code, it would be nicest if liballoc did not contain any notion of a static singleton always-working allocator.

---

Then common thread here is I’d like to ignore std and what not for the time being, and do a bottom-up approach where we first do the maximally flexible thing (custom associate errors, Result-based apis, let users define system singleton alloc only if that makes sense), and only then worry about ergonomics/overkill.

[Ericson2314]

And +1000 every @japaric said and did on toolchains. We should hold ourselves to a really high standard here because we can, and because that will give us a huge advantage over everything else.

[farcaller]

I want to see a cargo support for cross-compiling, so that there would be no need to carry complex build rules all around.

This could include things like cargo cross-building crates (but still being able to run cargo tests on native arch), better support for linker scripts, better support for defining the truly external symbols. In a perfect scenario, cargo build with a bunch of feature flags could be able to provide a flashable binary object.