History of shadowing in Rust


#1

Unlike most languages, Rust allows and actively encourages use of shadowing, which is one of my most favorite lesser features of the language (I’ve spend hours debugging problems caused by the use of wrong foo in languages which forbid/lint against shadowing).

What was the history of this feature? Was it considered good style from the start? Were there any mailing-lists fights over it? An RFC perhaps? I am just curious about the past age :slight_smile:


#2

That seems to be from “the early days” of Rust - it’s likely taken directly from OCaml, which also allows and encourages-ish shadowing, and which was the language that Rust was bootstrapped in (i.e., the first compiler for Rust was written in OCaml).


#3

IIRC @pcwalton implemented it one day, and we’ve had it every since.


#4

It’s more like we never forbade shadowing, since it just fell out of the implementation of the compiler.

As I recall, Graydon floated the idea of forbidding shadowing, but I stuck up for the feature, nobody else really cared, and so it stayed.


#5

A better question, imo, is why other languages do forbid shadowing.


#6

From my experience it’s a double-edged sword. I have had bugs happen because of unintended shadowing ("Oh, I meant to use the other x"). At the same time, shadowing can also be used to preclude certain kinds of bugs caused by unintended reuse of variables.

I personally think shadowing is nice to have: it’s convenient and has the advantage that an outer scope can introduce variables without conflicting with inner scopes.


#7

I think every language allows nested shadowing that (except for like, “everything is global” languages), but Rust’s difference is that it allows things to shadow at the same “level”. i.e., the following is valid C++:

static int const foo = 0;
int main() {
  double foo = 1;
  auto x = [] {
    float foo = 2;
    return foo;
  }();
  for (size_t foo = 0; foo < 10; ++foo) {
    { // note the extra brackets are necessary
      // the scope of for loop variables is the same scope as the stmt
      auto foo = std::vector<int>();
    }
    std::cout << foo; // 0, 1, 2, ...
  }
  std::cout << foo; // 1
  std::cout << x; // 2
}

void print_foo() {
  std::cout << foo; // 0
}

#8

In most languages, I’d argue against shadowing, since my experiences with it are riddled with bugs caused by unintended shadowing, and no memory of it ever being helpful. In Rust, however, the ability to redefine an immutable variable, possibly with a different type, is too damn convenient.


#9

There’s definitely an important difference here between static and dynamic languages.

In a dynamic language, there’s much less of a need since something like x = parse_int(x) doesn’t need a new variable, and any shadowing that did happen would be that much more surprising as there’s no type system to help notice it.

In a static language, shadowing helps avoid forcing hungarian-like x = parse(x_str), and types (plus ownership, in rust) help notice using the wrong one.

Probably weaker type systems, like those with object all over, are more likely to avoid shadowing as they’re closer to the dynamic side of things…


#10

PyCharm has inspections to warn against shadowing in Python. One that I find particularly useful is to warn if you shadow a built-in like list, str, or int, which is legal but can get confusing very quickly.


#11

No matter how I look at it I just cannot imagine how anybody ever accidentally uses a local binding where they intended to use a binding of larger scope, in any language. Well… okay, maybe in languages that have truly dynamic scope, by which I mean languages where if f calls g, then g can see variables defined in f. (these would be emacs 23, or Perl’s local…) But lo and behold, those are also the cases of shadowing which are undecidable for a static lint! (I think)

I code in python all the time, for practically everything. Even there, I cannot see how anyone makes this mistake. Especially for the builtin functions; I think it takes a special sort of skill to accidentally shadow something like dir or type or next or file, and not get the world’s most obvious type error when you try to use the corresponding builtin function.

Meanwhile, what does happen to me is accidentally using something with a larger scope. I’ve wasted hours debugging pointless issues like this in jupyter notebook, where at some point I had a cell that looked like

for n in range(N):
    print(f'Item {n}: ')
    do = a(data[n])
    bunch = of()
    nontrivial(data[n])
    stuff()

which I would at some point clean up and refactor because I needed it elsewhere:

def do_a_bunch_of_stuff(dataset):
    do = a(dataset)
    bunch = of()
    nontrivial(data[n]) # <---- uh oh; this should be 'dataset', but there's no error because it
    stuff()             #        "successfully" resolves to the global 'n' and 'data'

for (n, dataset) in enumerate(data):
    print(f'Item {n}: ')
    do_a_bunch_of_stuff(dataset) # worse yet, it still appears to work when I call it here!

# (NOTE: from this point onwards, 'n' is equal to 'len(data) - 1', causing
#        the function to behave strangely)

It has gotten to the point where basically all global level code in my notebooks are now wrapped in dummy functions just to prevent the proliferation of globals.

@iife   #  defined in the first cell as:    iife = lambda f: f()
def _():
    for (n, dataset) in enumerate(data):
        print(f'Item {n}: ')
        do_a_bunch_of_stuff(dataset)

And now for something completely different

Little-known fact: Python supports some pretty aggressive forms of shadowing in list comprehensions.

def frobnicate(value):

    print(                           # Equivalent to:
        [ value for value in value   #  [ x for plane in value
                if len(value) == 1   #      if len(plane) == 1
                for value in value   #      for row in plane
                for value in value   #      for x in row
        ]                            #  ]
    ) # [5, 6]

    print(value) # [[[1, 2], [3, 4]], [[5, 6]]]


frobnicate([[[1, 2], [3, 4]], [[5, 6]]])

Even the authors of pylint don’t seem to be aware of this.

$ python3 -m pylint --disable=bad-whitespace,bad-continuation,missing-docstring travesty.py
No config file found, using default configuration

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)