Making the compiler interface on-demand driven


#1

Currently users of the compiler (rustdoc, miri, the rustc executable itself, etc.) manually set up the compiler state, spawn suitable threads for the compiler and call passes. This leads to a lot of code duplication and subtle differences betweeen each use of the compiler, leaking implementation details into the users of the compiler. This makes it hard to make changes to the high-level structure of the compiler. Here is an example invocation from rustdoc with some rustdoc specific parts removed:

driver::spawn_thread_pool(sessopts, move |sessopts| {
    let source_map = Lrc::new(source_map::SourceMap::new(sessopts.file_path_mapping()));
    let diagnostic_handler = new_handler(error_format,
                                         Some(source_map.clone()),
                                         debugging_options.treat_err_as_bug,
                                         debugging_options.ui_testing);

    let mut sess = session::build_session_(
        sessopts, cpath, diagnostic_handler, source_map,
    );

    ...

    let codegen_backend = rustc_driver::get_codegen_backend(&sess);
    let cstore = Rc::new(CStore::new(codegen_backend.metadata_loader()));
    rustc_lint::register_builtins(&mut sess.lint_store.borrow_mut(), Some(&sess));

    let mut cfg = config::build_configuration(&sess, config::parse_cfgspecs(cfgs));
    target_features::add_configuration(&mut cfg, &sess, &*codegen_backend);
    sess.parse_sess.config = cfg;

    let control = &driver::CompileController::basic();

    let krate = panictry!(driver::phase_1_parse_input(control, &sess, &input));

    let name = match crate_name {
        Some(ref crate_name) => crate_name.clone(),
        None => ::rustc_codegen_utils::link::find_crate_name(Some(&sess), &krate.attrs, &input),
    };

    let mut crate_loader = CrateLoader::new(&sess, &cstore, &name);

    let resolver_arenas = resolve::Resolver::arenas();
    let result = driver::phase_2_configure_and_expand_inner(&sess,
                                                    &cstore,
                                                    krate,
                                                    None,
                                                    &name,
                                                    None,
                                                    resolve::MakeGlobMap::No,
                                                    &resolver_arenas,
                                                    &mut crate_loader,
                                                    |_| Ok(()));
    let driver::InnerExpansionResult {
        mut hir_forest,
        resolver,
        ..
    } = abort_on_err(result, &sess);

    // We need to hold on to the complete resolver, so we clone everything
    // for the analysis passes to use. Suboptimal, but necessary in the
    // current architecture.
    let defs = resolver.definitions.clone();
    let resolutions = ty::Resolutions {
        freevars: resolver.freevars.clone(),
        export_map: resolver.export_map.clone(),
        trait_map: resolver.trait_map.clone(),
        maybe_unused_trait_imports: resolver.maybe_unused_trait_imports.clone(),
        maybe_unused_extern_crates: resolver.maybe_unused_extern_crates.clone(),
        extern_prelude: resolver.extern_prelude.iter().map(|(ident, entry)| {
            (ident.name, entry.introduced_by_item)
        }).collect(),
    };
    let analysis = ty::CrateAnalysis {
        access_levels: Lrc::new(AccessLevels::default()),
        name: name.to_string(),
        glob_map: if resolver.make_glob_map { Some(resolver.glob_map.clone()) } else { None },
    };

    let mut arenas = AllArenas::new();
    let hir_map = hir_map::map_crate(&sess, &*cstore, &mut hir_forest, &defs);
    let output_filenames = driver::build_output_filenames(&input,
                                                        &None,
                                                        &None,
                                                        &[],
                                                        &sess);

    let resolver = RefCell::new(resolver);
    abort_on_err(driver::phase_3_run_analysis_passes(&*codegen_backend,
                                                    control,
                                                    &sess,
                                                    &*cstore,
                                                    hir_map,
                                                    analysis,
                                                    resolutions,
                                                    &mut arenas,
                                                    &name,
                                                    &output_filenames,
                                                    |tcx, analysis, _, result| {
        ...
    }))

})

In order to improve on this situation I propose a new compiler interface which should live in a rustc_interface crate. In order to create an compiler context, represented by an instance of the Compiler type, you’d call run_compiler passing in the desired configuration. It would set up the compiler and create a suitable thread pool for it to run in and then call the passed closure giving control of the compiler back to the caller.

pub struct Config {
    pub opts: config::Options,
    pub crate_cfg: HashSet<(String, Option<String>)>,
    pub input: Input,
    pub input_path: Option<PathBuf>,
    pub output_dir: Option<PathBuf>,
    pub output_file: Option<PathBuf>,
    pub file_loader: Option<Box<dyn FileLoader + Send + Sync>>,
    pub emitter: Option<Box<dyn Write + Send>>,
    pub stderr: Option<Arc<Mutex<Vec<u8>>>>,
    pub crate_name: Option<String>,
}

pub fn run_compiler<F, R>(mut config: Config, f: F) -> R
where
    F: FnOnce(Compiler) -> R + Send,
    R: Send;

Once you have access to a Compiler you can call methods on it to compute just the required information on-demand. Here is a list of methods my prototype current has:

impl Compiler {
    pub fn dep_graph_future(&self) -> Result<&Query<Option<DepGraphFuture>>>;
    pub fn parse(&self) -> Result<&Query<ast::Crate>>;
    pub fn crate_name(&self) -> Result<&Query<String>>;
    pub fn expansion(&self) -> Result<&Query<(ast::Crate, Rc<Option<RefCell<BoxedResolver>>>)>>;
    pub fn dep_graph(&self) -> Result<&Query<DepGraph>>;
    pub fn lower_to_hir(&self) -> Result<&Query<(Steal<hir::map::Forest>, ExpansionResult)>>;
    pub fn prepare_outputs(&self) -> Result<&Query<OutputFilenames>>;
    pub fn codegen_channel(&self) -> Result<&Query<(Steal<mpsc::Sender<Box<dyn Any + Send>>>,
                                                    Steal<mpsc::Receiver<Box<dyn Any + Send>>>)>>;
    pub fn hir_map(&self) -> Result<&Query<BoxedHirMap>>;
    pub fn global_ctxt(&self) -> Result<&Query<BoxedGlobalCtxt>>;
    pub fn analysis(&self) -> Result<&Query<ty::CrateAnalysis>>;
    pub fn ongoing_codegen(&self) -> Result<&Query<Box<dyn Any>>>;
    pub fn link(&self) -> Result<&Query<()>>;
}

These methods will themselves call the other methods they depend upon. For example expansion will call the parse method causing the crate to be parsed before macro expansion occurs. Note that expansion will also take ownership of the result of parse making its result unavailable once expansion is called. Here is the expansion method for an example of what these look like:

pub fn expansion(&self) -> Result<&Query<(ast::Crate, Rc<Option<RefCell<BoxedResolver>>>)>> {
    self.queries.expansion.compute(|| {
        let crate_name = self.crate_name()?.peek().clone();
        let krate = self.parse()?.take();
        passes::configure_and_expand(
            self.sess.clone(),
            self.cstore().clone(),
            krate,
            &crate_name,
            MakeGlobMap::No,
        ).map(|(krate, resolver)| (krate, Rc::new(Some(RefCell::new(resolver)))))
    })
}

You can find the implementation of all these methods in my prototype here.

Query is a type which allows look or modify existing computed information, it has the following interface:


impl<T> Query<T> {
    pub fn take(&self) -> T;
    pub fn give(&self, value: T);
    pub fn peek(&self) -> Ref<'_, T>;
    pub fn peek_mut(&self) -> RefMut<'_, T>;
}

In the above list of methods, there are 3 interesting types, BoxedResolver, BoxedHirMap and BoxedGlobalCtxt. These have internal self-references. For example, BoxedGlobalCtxt stores a GlobalCtxt<'gcx> inside that can be accessed through the enter method on BoxedGlobalCtxt which gives you a `TyCtxt:

impl BoxedGlobalCtxt {
    pub fn enter<F, R>(&mut self, f: F) -> R
    where
        F: for<'tcx> FnOnce(TyCtxt<'_, 'tcx, 'tcx>) -> R;
}

This gives access to a proper TyCtxt. This trick is enabled by allocating the GlobalCtxt and the arenas inside an immovable generator.

With these changes rustdoc's compiler setup should look like this:

let config = Config {
    opts: sessopts,
    crate_cfg: config::parse_cfgspecs(cfgs),
    input,
    input_path: cpath,
    output_file: None,
    output_dir: None,
    file_loader: None,
    emitter: None,
    stderr: None,
    crate_name: crate_name.clone(),
};

run_compiler(config, |compiler| {
    // We need to hold on to the complete resolver, so we cause everything to be
    // cloned for the analysis passes to use. Suboptimal, but necessary in the
    // current architecture.
    let resolver = compiler.expansion()?.peek().1.clone();

    let ty::CrateAnalysis { access_levels, .. } = compiler.analysis()?.take();

    compiler.global_ctxt()?.take().enter(|tcx| {
        ...
    })
})

I’d expect Compiler's methods to gradually turn into proper TyCtxt queries and I think this is a helpful incremental step towards that goal. Eventually only Compiler::global_ctxt and Compiler::link should remain.


Proposal: Borrow #run idea from JAI
#2

Are you planning on making a PR for this?


#3

I understand that the already submitted PR #56732 is related.


#4

@Zoxc thanks for posting this! Sorry for not responding more quickly. Holidays. :christmas_tree:

This isn’t quite the interface that I had imagined, but then I’m not entirely sure what I was imagining anyway. I’ll have to browse your PR to help form my opinion.

First off, though, a few questions and clarifications, especially as I haven’t had a chance to read the PR yet.

Are we able to implement this interface without unsafe code (especially the points regarding internal references)?

One could also imagine passing in the arena as an argument when creating the Compiler, and having the Compiler interface be parameterized by a lifetime ('gcx, basically).

I was also a bit surprised to see that the “stealing” concept is so central in this interface. Perhaps that makes sense given how the code works.


#5

I’d think of this as a temporary interface until we get end-to-end incremental queries. It should probably be revisited then.

No. However, this would be possible to do if we had generators with arguments. Currently my branch uses a safe abstraction using macros. It would be a bit cleaner to just yield raw pointers here. I structured the code the way I’d write it in safe code given generator arguments.

We want to destroy the arenas while we have a Compiler type around. The resolver has it’s owner arena (in BoxedResolver) that is freed and BoxedGlobalCtxt is freed in the compile method to minimize memory usage.

We want to avoid cloning large structures like the AST, HIR, etc. It also allows code to easily free results of queries by just stealing them.


#6

OK. Given that this is not aiming to be a “final” interface, I feel reasonably good about landing it, but I would like us to think over and design what that final interface ought to be.