Rust-lang/rust LLM policy

internals policy
1

This is a discussion thread for rust-lang/rust's new LLM policy, which has been merged in rust-forge#1040. You can find more information about that policy in the blog post. Please comment here, not on the Forge or blog PRs.

Please remember to respect the Code of Conduct at all times.

11 Likes
2

You disclosed the secret of git diff --check! Now we'll have to find another signal :rofl:

3

So does this ban writing code (or other text) using speech to text with an LLM to help correct misheard (which is a thing these days)? If so that is highly problematic from an accessibility point of view.

Similarly, what about "smarter tab complete" style AI, which helps reduce the number of keystrokes, but doesn't really go off and do it's own slop thing (unlike agentic AI)? Again, this is an accessibility concern for me as I have RSI in both wrists.

4

The policy does not directly say anything about speech-to-text. I would consider that to be allowed without disclosure.

what about "smarter tab complete" style AI, which helps reduce the number of keystrokes, but doesn't really go off and do it's own slop thing (unlike agentic AI)

the policy says this:

This policy makes no distinction between LLM output that comes from a chat interface and output that comes from editor auto-completion. In most cases the output is “trivial” (see above under :warning:), but regardless, it is not treated specially by this policy.

and this:

:warning: Allowed with caveats

...
All uses under “:warning: Allowed with caveats” must disclose that an LLM was used.
...
“Trivial” code or prose changes.

  • :information_source: Changes are trivial if there is no other way to write them, or the other ways to write them are nearly identical. For example, the following are all trivial:
    • Typo fixes
    • Markdown links
    • Changing a word to a synonym
    • Type signatures for a trait implementation

I think the disclosure requirement here is probably overboard and would be open to making that less restrictive.

4 Likes
5

The way I use tab completion AI is very similar to old style intellisense: I know exactly what I want to write, the computer shows some suggestion, and sometimes I can press tab instead of writing the words by myself. There is really no room for slop there, but it saves my wrists.

On the other hand, I have seen even tab completion style AI able to answer some things I don't know myself, such as writing a bash style comment describing what I want and getting an obscure ffmpg command line on the next line.[1]

  1. Interestingly, AI tends to be far worse at generating working sed expressions, those I have to write myself, but I also know sed syntax by heart. ↩︎

6 Likes
6

Something entirely different, as a Rust user who only watched this: Huge thanks to @jyn for writing the policy and blog post, going through the long-winded review process, moderating a tight corridor between frustration and hostility, coming to the merge, then thinking of and creating dedicated avenues like this one, while being transparent and open during the entire period. That is highly impressive and commendable!

A modefully similar thank-you extends to other contributors on rust-forge#1040! I truly believe that the policy you all came out with is a solid transient optimum between {actionability,clarity,complexity,length,malleability} until we can come up with a project-scoped policy. :3

6 Likes
7

the main issue with discussing "smarter tab complete" is that it is very broad (and i don't use LLMs at all so idk what the current trend among implementations as tab completion is), it could mean

  1. the IDE literally doing the same thing as the web interface (and maybe using a different system prompt) and just copy-pasting the result in
  2. the entire project getting modified by a LLM whenever you press tab, making changes anymore
  3. giving a list of all theoretically valid methods and variables in scope to the LLM and asking it for a single method and its arguments
  4. a LLM to identify patterns in your codebases, automatically turning them into standard LSP snippets and adding caret-jump-points (without allowing any other modifications) and then suggesting the completion whenever you type a related keyword
  5. adding a much larger set of completions and using LLMs to filter out irrelevant completions and/or ranking the existing completions in a better way
  6. an infinite amount of interpretations i did not even consider

LLM companies generally want 2 or 1.

1 depends very heavily on context (e.g. is the output an entire function from 5 words or if let Some(x) = foo() { return x } in response to foo().ret_some)
2 sounds like a nigthmare
3 would likely be allowed as its trivial,
4 technically doesn't generate new code, so it'd likely be allowed
5 would also not be generating any code, so it'd likely be allowed

8

thank you so much <3 i’m still kind of amazed this actually happened

9

GH copilot's tab completion does a lot of different things.

  • detects typos, suggesting fixes
  • after a change (like names or types) suggests followups of near things that need to be adjusted too
  • completing a few words ahead
  • suggesting entire blocks
  • suggesting edits across multiple functions

My impression is (purely based on use) is that it is stateful - it sees what you just edited - and essentially suggesting diffs within the current file and the size and scope of those diffs varies a lot. It also reads adjacent comments. It might also be looking at LSP errors and trying to offer fixes for that, not sure.

Personally I find that its nonsense output rate to be fairly high, both on simple and complex changes. It's also using an older model compared to the chat stuff. Often its changes are trivial, but sometimes when it vomits out larger blocks those can be almost right yet subtly wrong.

So unfortunately its range is wide enough that I wouldn't write it off as always-insignificant.