Currently users of the compiler (rustdoc, miri, the rustc executable itself, etc.) manually set up the compiler state, spawn suitable threads for the compiler and call passes. This leads to a lot of code duplication and subtle differences betweeen each use of the compiler, leaking implementation details into the users of the compiler. This makes it hard to make changes to the high-level structure of the compiler. Here is an example invocation from rustdoc with some rustdoc specific parts removed:
driver::spawn_thread_pool(sessopts, move |sessopts| {
let source_map = Lrc::new(source_map::SourceMap::new(sessopts.file_path_mapping()));
let diagnostic_handler = new_handler(error_format,
Some(source_map.clone()),
debugging_options.treat_err_as_bug,
debugging_options.ui_testing);
let mut sess = session::build_session_(
sessopts, cpath, diagnostic_handler, source_map,
);
...
let codegen_backend = rustc_driver::get_codegen_backend(&sess);
let cstore = Rc::new(CStore::new(codegen_backend.metadata_loader()));
rustc_lint::register_builtins(&mut sess.lint_store.borrow_mut(), Some(&sess));
let mut cfg = config::build_configuration(&sess, config::parse_cfgspecs(cfgs));
target_features::add_configuration(&mut cfg, &sess, &*codegen_backend);
sess.parse_sess.config = cfg;
let control = &driver::CompileController::basic();
let krate = panictry!(driver::phase_1_parse_input(control, &sess, &input));
let name = match crate_name {
Some(ref crate_name) => crate_name.clone(),
None => ::rustc_codegen_utils::link::find_crate_name(Some(&sess), &krate.attrs, &input),
};
let mut crate_loader = CrateLoader::new(&sess, &cstore, &name);
let resolver_arenas = resolve::Resolver::arenas();
let result = driver::phase_2_configure_and_expand_inner(&sess,
&cstore,
krate,
None,
&name,
None,
resolve::MakeGlobMap::No,
&resolver_arenas,
&mut crate_loader,
|_| Ok(()));
let driver::InnerExpansionResult {
mut hir_forest,
resolver,
..
} = abort_on_err(result, &sess);
// We need to hold on to the complete resolver, so we clone everything
// for the analysis passes to use. Suboptimal, but necessary in the
// current architecture.
let defs = resolver.definitions.clone();
let resolutions = ty::Resolutions {
freevars: resolver.freevars.clone(),
export_map: resolver.export_map.clone(),
trait_map: resolver.trait_map.clone(),
maybe_unused_trait_imports: resolver.maybe_unused_trait_imports.clone(),
maybe_unused_extern_crates: resolver.maybe_unused_extern_crates.clone(),
extern_prelude: resolver.extern_prelude.iter().map(|(ident, entry)| {
(ident.name, entry.introduced_by_item)
}).collect(),
};
let analysis = ty::CrateAnalysis {
access_levels: Lrc::new(AccessLevels::default()),
name: name.to_string(),
glob_map: if resolver.make_glob_map { Some(resolver.glob_map.clone()) } else { None },
};
let mut arenas = AllArenas::new();
let hir_map = hir_map::map_crate(&sess, &*cstore, &mut hir_forest, &defs);
let output_filenames = driver::build_output_filenames(&input,
&None,
&None,
&[],
&sess);
let resolver = RefCell::new(resolver);
abort_on_err(driver::phase_3_run_analysis_passes(&*codegen_backend,
control,
&sess,
&*cstore,
hir_map,
analysis,
resolutions,
&mut arenas,
&name,
&output_filenames,
|tcx, analysis, _, result| {
...
}))
})
In order to improve on this situation I propose a new compiler interface which should live in a rustc_interface crate. In order to create an compiler context, represented by an instance of the Compiler type, you’d call run_compiler passing in the desired configuration. It would set up the compiler and create a suitable thread pool for it to run in and then call the passed closure giving control of the compiler back to the caller.
pub struct Config {
pub opts: config::Options,
pub crate_cfg: HashSet<(String, Option<String>)>,
pub input: Input,
pub input_path: Option<PathBuf>,
pub output_dir: Option<PathBuf>,
pub output_file: Option<PathBuf>,
pub file_loader: Option<Box<dyn FileLoader + Send + Sync>>,
pub emitter: Option<Box<dyn Write + Send>>,
pub stderr: Option<Arc<Mutex<Vec<u8>>>>,
pub crate_name: Option<String>,
}
pub fn run_compiler<F, R>(mut config: Config, f: F) -> R
where
F: FnOnce(Compiler) -> R + Send,
R: Send;
Once you have access to a Compiler you can call methods on it to compute just the required information on-demand. Here is a list of methods my prototype current has:
impl Compiler {
pub fn dep_graph_future(&self) -> Result<&Query<Option<DepGraphFuture>>>;
pub fn parse(&self) -> Result<&Query<ast::Crate>>;
pub fn crate_name(&self) -> Result<&Query<String>>;
pub fn expansion(&self) -> Result<&Query<(ast::Crate, Rc<Option<RefCell<BoxedResolver>>>)>>;
pub fn dep_graph(&self) -> Result<&Query<DepGraph>>;
pub fn lower_to_hir(&self) -> Result<&Query<(Steal<hir::map::Forest>, ExpansionResult)>>;
pub fn prepare_outputs(&self) -> Result<&Query<OutputFilenames>>;
pub fn codegen_channel(&self) -> Result<&Query<(Steal<mpsc::Sender<Box<dyn Any + Send>>>,
Steal<mpsc::Receiver<Box<dyn Any + Send>>>)>>;
pub fn hir_map(&self) -> Result<&Query<BoxedHirMap>>;
pub fn global_ctxt(&self) -> Result<&Query<BoxedGlobalCtxt>>;
pub fn analysis(&self) -> Result<&Query<ty::CrateAnalysis>>;
pub fn ongoing_codegen(&self) -> Result<&Query<Box<dyn Any>>>;
pub fn link(&self) -> Result<&Query<()>>;
}
These methods will themselves call the other methods they depend upon. For example expansion will call the parse method causing the crate to be parsed before macro expansion occurs. Note that expansion will also take ownership of the result of parse making its result unavailable once expansion is called. Here is the expansion method for an example of what these look like:
pub fn expansion(&self) -> Result<&Query<(ast::Crate, Rc<Option<RefCell<BoxedResolver>>>)>> {
self.queries.expansion.compute(|| {
let crate_name = self.crate_name()?.peek().clone();
let krate = self.parse()?.take();
passes::configure_and_expand(
self.sess.clone(),
self.cstore().clone(),
krate,
&crate_name,
MakeGlobMap::No,
).map(|(krate, resolver)| (krate, Rc::new(Some(RefCell::new(resolver)))))
})
}
You can find the implementation of all these methods in my prototype here.
Query is a type which allows look or modify existing computed information, it has the following interface:
impl<T> Query<T> {
pub fn take(&self) -> T;
pub fn give(&self, value: T);
pub fn peek(&self) -> Ref<'_, T>;
pub fn peek_mut(&self) -> RefMut<'_, T>;
}
In the above list of methods, there are 3 interesting types, BoxedResolver, BoxedHirMap and BoxedGlobalCtxt.
These have internal self-references. For example, BoxedGlobalCtxt stores a GlobalCtxt<'gcx> inside that can be accessed through the enter method on BoxedGlobalCtxt which gives you a `TyCtxt:
impl BoxedGlobalCtxt {
pub fn enter<F, R>(&mut self, f: F) -> R
where
F: for<'tcx> FnOnce(TyCtxt<'_, 'tcx, 'tcx>) -> R;
}
This gives access to a proper TyCtxt. This trick is enabled by allocating the GlobalCtxt and the arenas inside an immovable generator.
With these changes rustdoc's compiler setup should look like this:
let config = Config {
opts: sessopts,
crate_cfg: config::parse_cfgspecs(cfgs),
input,
input_path: cpath,
output_file: None,
output_dir: None,
file_loader: None,
emitter: None,
stderr: None,
crate_name: crate_name.clone(),
};
run_compiler(config, |compiler| {
// We need to hold on to the complete resolver, so we cause everything to be
// cloned for the analysis passes to use. Suboptimal, but necessary in the
// current architecture.
let resolver = compiler.expansion()?.peek().1.clone();
let ty::CrateAnalysis { access_levels, .. } = compiler.analysis()?.take();
compiler.global_ctxt()?.take().enter(|tcx| {
...
})
})
I’d expect Compiler's methods to gradually turn into proper TyCtxt queries and I think this is a helpful incremental step towards that goal. Eventually only Compiler::global_ctxt and Compiler::link should remain.
