I’ve made a list of existing tools for Rust development and brain dumped a whole bunch of stuff about how to encourage a great tools ecosystem for Rust (including some tool-ish aspects of the compiler). Please let me know if there are any existing tools I’ve missed, or if you have comments or questions about the other stuff.
3 Likes
If you want to add a compiler plugin section: spellck would be a nice addition.
rustc depends on ar and gcc (for linking).
It shouldn’t be so difficult to copy minimal functions of ar into rustc . (We may also switch to ZIP or TAR format for rlib.)
As for gcc for linking, we may replace it with LLVM linker in the future.
This is a very important topic. It’s great that you took the time to write about it, Nick!
Apart from debuginfo stuff, I’ve been mostly thinking about what it would take to have full-featured IDE support with realtime-error highlighting, debugging, code completion, navigation, refactoring support, etc. Many of these interactive things need to be really fast in order to be useful and feel stable. I’m not sure how far we could get towards that end by adapting a batch-mode compiler like rustc is.
Some things that I discovered during my limited spare-time experiments of implementing a fully incremental, soft-realtime “program knowledge base” server for supporting IDEs:
- If you take response times really seriously (wanting them to be in the millisecond range) then I think the compiler has to treat things incrementally at a low granularity, e.g. the “item” level. Reparsing a file and finding out which items have changed is not that hard but one has to be careful to really update only necessary parts of the compiler output (resolve map, type checking information, etc). A careful dependency analysis is needed in order to implement this correctly.
- Error tolerance becomes an important topic, something can be pretty much ignored by a batch compiler. You’d still want code completion to work even if there are mild syntax errors in the codebase.
- Memory management becomes much more important. In a long-running program (potentially days or weeks) you can’t get away with not deallocating stuff properly and in a timely fashion.
- The codemap and spans as they are implemented in libsyntax are not well-suited for incremental changes, since the codemap stores all files in the same “address space”. It’s basically an arena, always pushing new things to the end and never deallocating. This is a good solution for a batch compiler but unsuited when having to deal with many small incremental changes.
- Error messages and other diagnostics also have to be treated as output data instead of just being a side-effect of compilation.
In my opinion some of the above things suggest that it might be problematic to “just extend” rustc and use it as a helper program. But it would be highly desirable to re-use common compiler components/passes in a tool like the above as much as possible. You’d want error messages to match up between IDE and batch compilation output, you’d want custom compiler plugins and syntax extensions to be usable in both scenarios and so on.
I agree with Nick that having stable interfaces for tools (like DWARF debuginfo or a separate AST for libsyntax) is very important and could be something that would help with implementing something like the above by splitting up rustc into separate components that only rely on clearly defined interfaces, i.e. take their data in a well specified format and produce data in a well-specified format. This also helps with keeping compiler structure and pass responsibilities clear (although rustc does not have a problem with this as far as I can tell). However, this also conflicts a bit with allowing for incrementality, since then you’d need slightly different interfaces if you want to make a performant implementation possible.
Needless to say all of this would take a lot more research and quite some effort to get started. But it’s also super-interesting to think about 
Thanks for the comments! I have been updating the doc on the gist with the comments
I now get a 404:
Whoops.
We seem to have missed the gist of that gist you were looking for.
I fixed the link (https://gist.github.com/nrc/a3bbf6dd1b14ce57f18c#file-tools-md)
By the way, in the Java world, NetBeans goes without incremental parsing and just relies on periodically calling javac. Granted, it’s not as fast (or as resource-saving) as an incremental parser like IntelliJ’s or Eclipse’s, but it works for many people nonetheless.
That said, incremental parsing is a must-have for a decent rust IDE. Error tolerance can be implemented by backing off token-wise until the token stream makes sense again (on error).
As for memory management, I’d suggest keeping the code inline the AST nodes instead of paging out (and inserting “unknown” nodes for anything newly typed) could help keep memory management under control.
Otherwise, a rope that somehow maintains addresses of its parts could be the solution (this would also make it easier to reuse the AST API between rustc and our not-yet existing incremental parser, it would just need to be parametric over the code data).
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.