Rust Debugging Quest

[tromey]

I’ve recently started working full time to improve Rust debugging. I thought I’d post a bit about what I’m planning to do. I’m open to additions and commentary; and also I have some questions.

The main thing I’m doing first is writing a Rust plugin for lldb. This is needed in order to enable future improvements to debuginfo (see below for a case in point). I’m currently doing this work here, but at some point I suppose this should be moved to rust-lang-nursery. (Sooner or later? The first question for you.)

The plugin is maybe halfway done, with the remaining piece being expression parsing. For this I’m currently planning write an external parser using the syn crate (or perhaps libsyntax 2.0), and have lldb call into it.

For a while, I think the Rust community will have to ship a fork of lldb. This message on lldb-dev explains why. Even once the Rust plugin lands upstream, we’ll still need to ship the external parser. I talked a bit with acrichto about this, see the bug he filed.

An lldb plugin will let us evolve the debuginfo output of the compiler in parallel with fixes to the debuggers. I can write up a full list if it’s needed (many of the to-do items are in the github issues already), but some highlights are:

• Correctly represent optimized enums. See this issue. I’ve actually implemented all of this for llvm (patches landed), rustc, and gdb – but I am having second thoughts about landing it. On the one hand, the enum optimization improvements regressed debugger support for this in some cases. On the other hand, these patches will make enums basically incomprehensible to lldb. So: land them anyway? Or hold them until the lldb plugin is done?

• Emit debuginfo for traits. This will enable method calls via traits, and also operator overloading support.

• Emit more debuginfo for virtual tables, allowing calls on trait objects.

• Macro stepping. (This one doesn’t actually need the lldb port, it’s just something that keeps coming up…)

I’ll also be filing bugs against DWARF to try to get Rust bindings into the standard. There are already a couple of Rust-inspired extensions in LLVM’s DWARF output which should probably be standardized; and we’ll need some new tags as well.

One unknown is how to deal with Windows. There’s some PDB support in LLVM but it’s unclear to me how much of Rust can be expressed there.

[Manishearth]

I say land them anyway. We’re going to have some churn either way, it’s okay if stuff is unstable for this year as long as we eventually settle down.

[michaelwoerister]

Great work, @tromey! I think this will be much appreciated once the community can get their hands on it.

Some questions:

• You say you have implemented the changes around optimized enums in LLVM, rustc, and gdb. Am I correct in assuming that only the LLVM part has landed upstream while the rustc and gdb changes are on still on personal branches that you maintain?
• Is there a way to make gdb support both, the current format and the new one?

Depending on the answers to the question above I have two suggestions on how to move forward:

1. If gdb cannot support both formats, we could:
   • keep the GDB changes in a fork while we still need to support the old format for LLDB,
   • make the compiler support both formats, with the new format only being emitted when a -Zdebuginfo-2018 flag is specified.
   • add a job to our CI that uses the GDB fork and tests the new format,
   • while regular users we keep using the current format.
2. If we can make GDB support both formats,
   • we’d just land the changes in GDB,
   • land the changes in the compiler, but still behind a nightly-only flag,
   • update our tests to test both the new and the old format.

I’d like to not break current LLDB, especially as long as we don’t have an ETA for the proper Rust plugin. People are using it as far as I can tell. However, I would be fine with starting out with a minimal LLDB plugin that just maintains the level of support that we have now (i.e. no parser, no expression evaluation). If we ship that then we could just switch the compiler over to the format.

One question though is: What do we do about older versions of GDB out there? Would we just tell people “download a recent version of GDB”?

[tromey]

@michaelwoerister

Am I correct in assuming that only the LLVM part has landed upstream while the rustc and gdb changes are on still on personal branches that you maintain?

Yes. The rust compiler patch is here and the gdb patch is here. I'm going to submit the gdb patch upstream soon, like probably today.

@michaelwoerister

Is there a way to make gdb support both, the current format and the new one?

Yes, that's what i implemented -- the patch moves the current enum-handling bits into the DWARF reader, and then adds support for the new format there. So now the rust code in gdb only sees a single representation.

@michaelwoerister

However, I would be fine with starting out with a minimal LLDB plugin that just maintains the level of support that we have now (i.e. no parser, no expression evaluation).

This is interesting. I was under the impression that you could currently evaluate expressions in LLDB, but you had to use C++ syntax. Is that not the case? Because my current Rust plugin for LLDB is basically DWARF reading without expressions -- not 100% done, but close to that milestone.

@michaelwoerister

What do we do about older versions of GDB out there? Would we just tell people “download a recent version of GDB”?

Yes, I think that's the only way. But also see this issue about shipping lldb. One idea would be to also sometimes ship our own gdb. I've argued against this in the past, but maybe it really is the way to go.

[michaelwoerister]

@tromey

Yes, that’s what i implemented – the patch moves the current enum-handling bits into the DWARF reader, and then adds support for the new format there. So now the rust code in gdb only sees a single representation.

OK, that sounds great!

@tromey

I was under the impression that you could currently evaluate expressions in LLDB, but you had to use C++ syntax. Is that not the case?

That's probably the case, yes. However, I don't think one can call functions in a reliable way. The most important thing, I think, is that one can inspect the contents of variables. I guess that goes through expression evaluation as soon as you follow a pointer...

@tromey

Would we just tell people “download a recent version of GDB”?

Yes, I think that’s the only way.

I think that's acceptable.

So, do you agree that we should make the compiler support both the old and the new format, keep emitting the old format for now, but start testing the new format on our infrastructure? People on nightly Rust and with a recent GDB version could already start benefiting from it.

[gilescope]

Excellent news - this really really needed doing but I wasn’t the right guy to do it. Happy to help test out custom lldb builds asap. Anything I can do to help please let me know.

[tromey]

It’s too soon to try it yet but I will definitely post here when it is ready. And, thank you.

[tromey]

@michaelwoerister

I’d like to not break current LLDB, especially as long as we don’t have an ETA for the proper Rust plugin

I recently discovered that the debug info for enums is already pretty broken in 1.24, see this LLVM bug. Maybe this can be fixed in some minimal way; but overall I'd rather just try to land my real enum patch; I'm wondering what you think about this.

[tromey]

I think the Rust plugin is complete enough for adventurous users to try it out. It can parse and evaluate many Rust expressions. Function calls work. There are a few known bugs, a few things that could be polished, and of course probably a bundle of undiscovered bugs.

Scared off yet? If not, please give it a try. You can find build instructions here.

PRs and bug reports are gratefully accepted.

[gilescope]

Superstar! This is most excellent news.

[mrborrowck]

@tromey I wonder if it s possible to upstream the enum-debuginfo branch very soon? It somehow boggles me how people are actually debugging their rust binaries without proper debugging information, how are they able to inspect the enum values? Right now it’s nearly impossible (rust-nightly, gdb, lldb).

[tromey]

Yeah, I am fixing up this branch now. Today I rebased it and fixed the problems. You can follow the progress in the bug: https://github.com/rust-lang/rust/issues/32920#issuecomment-412296203

[mrborrowck]

@tromey Thanks for the update! It seems that https://github.com/rust-lang/rust/pull/52716 has been merged.

But I am unable to install it with rustup component add lldb-preview --toolchain nightly although it seems that the package available in the toolchain: https://static.rust-lang.org/dist/2018-08-17/index.html (lldb-nightly-x86_64-apple-darwin.tar.gz)

Is this intended behaviour?

[tromey]

No, it’s not. I don’t know what is missing yet (I didn’t look today) but you can follow along in this bug: https://github.com/rust-lang/rust/issues/48168

[tromey]

Debugging Quest Update

There have been a few changes in the Rust Debugging Quest and it’s finally time for an update. I’ll also talk a bit about planned changes.

GDB 8.2

A few Rust-related bug fixes are in GDB 8.2, the newest release. Nothing earth-shattering here, as the Rust support in GDB is rather stable; but ptype/o (a pahole-like command) now works for Rust. You can see the list of bugs here.

One thing I’ve been contemplating is shipping GDB via rustup. This would let us roll out Rust-related fixes more quickly. There is a rust bug tracking this.

Rust-enabled LLDB

Primary development on the Rust plugin for LLDB is done, and the rust-enabled LLDB is now shipping via rustup.

Currently LLDB is a preview and only in nightly, and only on macOS. In order for this to work, you must be using rustup 1.14.0 (or later, presumably), which was released yesterday. Earlier versions of rustup did not preserve symlinks at install time, causing lldb crashes. You can update rustup and install lldb with

$ rustup self update
$ rustup component add lldb-preview --toolchain nightly

Note that this does not put an lldb executable in your PATH. This was deemed too risky. Instead, the rust-lldb wrapper was modified to run the rust-enabled lldb if it is available. So, use rust-lldb to start it.

If you find bugs – and you will – in the rust-enabled lldb, please report them to the rust lldb github issue tracker.

The rust plugin for lldb understands the rustc-generated DWARF a bit better than the C++ plugin, and this will only become more pronounced as more debuginfo changes go into rustc. It also uses Rust syntax for expressions. It is roughly on par with GDB, but less tested.

Perhaps in the future we’ll ship LLDB on more platforms, not just macOS. There is some discussion in this rust issue.

Debuginfo Changes

No actual debuginfo change have landed yet. Better support for optimized enums is in the works, but as you can see that’s been a long slog.

However, once that goes in, it will be time to prioritize the to-do list. I’ll write that up soon, but meanwhile one possibility for the next change would be representing traits in DWARF. This would enable method calls via traits, which can’t be done today.

If you have specific things that are painful, I’d like to hear about them. You can just reply to this note. No promises of course, but this sort of information helps with sorting the to-do list.

[vramana]

@tromey I wrote some of my experiences debugging rust code with GDB here https://vramana.github.io/blog/rust-debugging/

[tromey]

Thanks. I didn't see a way to reply there, so I thought I'd reply here:

More generally it would be nice to have command that will omit stepping into a few selected crates

Take a look at help skip.

Enable gdb history by default for rust-gdb

For historical reasons and/or inertia gdb has a few bad defaults. I recommend:

set breakpoint pending on
set print pretty on
set print object on
set history filename ~/.gdb_history
set history save on

Though set print object on doesn't actually affect Rust, only C++.

The try operator dance

This one is worth a bug report, perhaps it's something that can be fixed. Skipping to the } and back seems odd to me. It could be due to drops, but I suppose then it would be nicer to jump back to the declaration, not the }.

[masonk]

Hi. Thanks for working on the debugger!

What’s the current state of Rust debugging on Linux? I’m confused about what the debugging story is today, as well as what the intended debugging story is in the long term. Is lldb meant for Mac and gdb meant for linux? Or is it that gdb works better on linux right now, but in the future, both lldb and gdb are meant to be supported?

Right now, I have lldb set up on Linux with the rust-lldb formatter installed, but not (I think) the custom fork of lldb. Plain lldb is letting me set breakpoints and inspect some variables, which is already more than nothing. However, evaluating expressions (& macros) would be really handy, but right now that doesn’t seem to work. Also, in complicated structs, some of the data in them is not coming out. I think it’s not showing me data behind pointers. Does that sound right?

Also, the names of functions in other rust crates are mangled. (Or, possibly, it’s that the names of generic functions are mangled).

[gilescope]

I think the current state is a new effort is needed:

How to support Rust debugging post-Tromey?

So, I’m sorry to say it but @tromey will be started a new job come next Thursday and likely won’t have time to help with Rust debugging anymore. We need to discuss how we are going to maintain the current Rust debuginfo support. I had a few thoughts: Would there be interest in @tromey doing a kind of “intro to debugging support” talk? I was thinking we could do a video call where @tromey walks through how debuginfo works, where rust-lldb lives, etc. We could record it, too, for others to watch…

[tromey]

Sorry about the delay in my reply.

The state of debugging Rust is that gdb and lldb are roughly at feature parity, though gdb is probably a bit better for most things.

If you want the lldb features you mentioned (evaluating Rust expressions) to work, then you need the Rust-enabled lldb. Those features were one of the main pieces of work done there.

Neither debugger supports macro expansion. This information doesn’t appear in the debug info, so either a compiler change or a debugger change (say, to reuse the compiler itself) would be needed.

In the long term, I don’t know. I’m working on gdb and may still land the occasional Rust patch there (if bugs come to my attention); but I don’t plan to work on lldb. So, if you’re interested in lldb, it’s an opportunity to volunteer.