# Tensors static typing

**URL:** https://internals.rust-lang.org/t/tensors-static-typing/13114
**Category:** language design
**Created:** [September 22, 2020, 12:18pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114 "2020-09-22T12:18:11Z")
**Posts on this page:** 13
**Page:** 2

<div class="post-metadata">

### Author: ![elidupree](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/elidupree/32/4304_2.png) [@elidupree](https://internals.rust-lang.org/u/elidupree)
#### Post date: [September 23, 2020, 2:56pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/21 "2020-09-23T14:56:41Z")

</div>

[EDIT: somehow, the thread hadn't showed me a lot of the more recent posts when I made this post, so apologies for repeating some things others have said]

Interesting example… We have both conciseness issues and performance issues to think about here.

I had [a look at this in Godbolt](https://rust.godbolt.org/z/WYTe7T) [Edit: forgot to include Godbolt link]. Note that the assembly for `foo` and `foo2` are identical (i.e. the runtime check for the try\_into was already optimized out), but both versions still have a runtime check for the _ordering_ of the 2 indices, which... hmm. Which _can't_ be optimized out in the language as-is, because `i + 4` can do wrapping overflow in release builds.

Maybe what you want is a function like (don't mind the awkward implementation):

```rust
impl<T> [T] {
  pub fn get_array_mut<const LEN: usize>(&mut self, i: usize) -> &mut [T; LEN] {
    if i.checked_add(LEN).unwrap() < self.len() {panic!()}
    unsafe{
      match std::slice::from_raw_parts_mut(self.as_mut_ptr().add(i), LEN).try_into() {
        Ok(b) => b,
        Err(_) => std::hint::unreachable_unchecked()
      }
    }
  }
}

```

I've used this to implement `foo3` in my godbolt link, and it indeed appears to save one `cmp` instruction, even despite the awkward double-check on the first line.

Once the `min_const_generics` feature stabilizes, it should be straightforward to implement a clean version of this in the standard library.

---

<div class="post-metadata">

### Author: ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)
#### Post date: [September 23, 2020, 10:46pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/22 "2020-09-23T22:46:13Z")

</div>

> [@197g](#):
>
> `.array_chunks_mut().next().unwrap()`

I also have [PR #76635](https://github.com/rust-lang/rust/pull/76635) open which would allow that as `.as_chunks().0[0]` which is shorter, if not really more obvious.

You could consider a PR to make it just `a[i+1..].prefix_array_mut().unwrap()` or whatever.

That said, I'm not sure that any of these are much better than the `[..3].try_into().unwrap()` versions -- they're just moving around the bounds checks and panics.

---

<div class="post-metadata">

### Author: ![sighoya](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@sighoya](https://internals.rust-lang.org/u/sighoya)
#### Post date: [September 28, 2020, 6:07am UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/23 "2020-09-28T06:07:50Z")

</div>

> [@leonardo](#):
>
> I am not an expert about type systems, but I think dependent types are types that could depend on the run-time value of some variable. That's not happing above.

Yes, but it is compile dependent typing something akin to Dlang.

Compile time dependent typing is unfortunately not easier to decide than runtime dependent typing when tied with specialization. It doesn't suffice to evaluate the bounds of each trait method when more than one method applies to the input. In this case you need to solve a constraint system which is maybe of exponential complexity and the user doesn't always see this immediately and wonders why the compiler is so slow.

For the above reason, Dlang doesn't support specialization over value expressions not equal to simple membership expressions (like trait bounds for instance).

The other point is what did you do if your vector may be of length 3 or 4, or it may depends non-deterministally on foreign events requiring compile time parametrized vector lengths.

---

<div class="post-metadata">

### Author: ![atagunov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/atagunov/32/5877_2.png) [@atagunov](https://internals.rust-lang.org/u/atagunov)
#### Post date: [September 30, 2020, 9:31pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/24 "2020-09-30T21:31:11Z")

</div>

Hi, wouldn't const-generics allow you to write function signatures you desired?  
Provided all the sizes are known at compile time..

---

<div class="post-metadata">

### Author: ![leonardo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/leonardo/32/1344_2.png) [@leonardo](https://internals.rust-lang.org/u/leonardo)
#### Post date: [October 1, 2020, 8:40pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/25 "2020-10-01T20:40:38Z")

</div>

> [@sighoya](#):
>
> In this case you need to solve a constraint system which is maybe of exponential complexity and the user doesn't always see this immediately and wonders why the compiler is so slow.

I don't know how much fast this is in practice.

---

<div class="post-metadata">

### Author: ![ckaran](https://avatars.discourse-cdn.com/v4/letter/c/f475e1/32.png) [@ckaran](https://internals.rust-lang.org/u/ckaran)
#### Post date: [October 2, 2020, 1:19pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/26 "2020-10-02T13:19:31Z")

</div>

I already came up with an idea to solve the speed problem [here](https://internals.rust-lang.org/t/idea-make-assert-a-keyword/13066/60). Put in an extra compiler flag that specifies how long the compiler can take to execute the SAT solver; once the time is up, no more SAT solving. Instead, you emit less optimized code that is known to be correct, but possibly slower than fully optimized code.

---

<div class="post-metadata">

### Author: ![sighoya](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@sighoya](https://internals.rust-lang.org/u/sighoya)
#### Post date: [October 2, 2020, 2:16pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/27 "2020-10-02T14:16:10Z")

</div>

> [@leonardo](#):
>
> I don't know how much fast this is in practice.

Another problem we could face is operator overloading, e.g. a heavy product computation of two mathematic structures implemented with `while` unintended including the option to loop forever prolonging compilation time infinitely.

> [@ckaran](#):
>
> I already came up with an idea to solve the speed problem [here](https://internals.rust-lang.org/t/idea-make-assert-a-keyword/13066/60). Put in an extra compiler flag that specifies how long the compiler can take to execute the SAT solver; once the time is up, no more SAT solving. Instead, you emit less optimized code that is known to be correct, but possibly slower than fully optimized code.

The main problem is, even if you can see your own macros, others cannot view your macros. You compile your lib with calls to a const generic constrained function which are all finitely solvable. Another Guy calls your lib with a different const value breaking the compilation. This Guy may not understand why it doesn't work and may can't view your macro as it is enclosed in a private module.

Turning to runtime checking doesn't help either in case of specialization unless you change the dispatch strategy in this case.

---

<div class="post-metadata">

### Author: ![ckaran](https://avatars.discourse-cdn.com/v4/letter/c/f475e1/32.png) [@ckaran](https://internals.rust-lang.org/u/ckaran)
#### Post date: [October 2, 2020, 2:46pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/28 "2020-10-02T14:46:00Z")

</div>

> [@sighoya](#):
>
> The main problem is, even if you can see your own macros, others cannot view your macros. You compile your lib with calls to a const generic constrained function which are all finitely solvable. Another Guy calls your lib with a different const value breaking the compilation. This Guy may not understand why it doesn't work and may can't view your macro as it is enclosed in a private module.

But the compiler still has to execute my macros, even if they are private to my module, correct? That means that the compiler can still emit warnings or errors when the time limit is exceeded, correct? I agree though that even in that case trying to figure out why your own code is causing the compiler to fail is still going to be a real headache...

> [@sighoya](#):
>
> Turning to runtime checking doesn't help either in case of specialization unless you change the dispatch strategy in this case.

I'll have to trust you on that; I don't know enough to comment on this one way or another.

---

<div class="post-metadata">

### Author: ![jerry73204](https://avatars.discourse-cdn.com/v4/letter/j/e9bcb4/32.png) [@jerry73204](https://internals.rust-lang.org/u/jerry73204)
#### Post date: [October 2, 2020, 9:24pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/29 "2020-10-02T21:24:56Z")

</div>

Knowing all sizes is too strict in practice. In most cases, we desire our algorithm can process images with adaptive sizes and with fixed number of channels. Also it's not possible to define compile-time restrictions like `IsOdd` or `IsPrime` for const generics. It is still far away from building a truely static API.

---

<div class="post-metadata">

### Author: ![jerry73204](https://avatars.discourse-cdn.com/v4/letter/j/e9bcb4/32.png) [@jerry73204](https://internals.rust-lang.org/u/jerry73204)
#### Post date: [October 2, 2020, 10:17pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/30 "2020-10-02T22:17:54Z")

</div>

In contrast to adding static checkers in dynamic typed language, Rust goes on the reverse direction of gradual typing. It _relaxes_ type checks when some information is missing in compile time and leave it to runtime checkers. Though dependent typing can somewhat a solution, we cannot have it because it violates the Rust's design principals, and we don't want type checkers in runtime. Instead, we can combine a `Dyn` type with Rust's generics. That's the way we can achieve a weak form of gradual typing. The code below illustrates the basic idea.

```rust
use typenum::UTerm; // a number type standing for zero
struct Dyn(usize);

impl Add<UTerm> for Dyn {
    type Output = Dyn;
    fn add(self, _rhs: UTerm) -> Self::Output { /* runtime checker goes here */ }
}

```

---

<div class="post-metadata">

### Author: ![jerry73204](https://avatars.discourse-cdn.com/v4/letter/j/e9bcb4/32.png) [@jerry73204](https://internals.rust-lang.org/u/jerry73204)
#### Post date: [October 2, 2020, 10:54pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/31 "2020-10-02T22:54:47Z")

</div>

Towards building the static tensor API, the core issues can be arranged into two threads.

- How to encode complex logic in types. For example, can we statically compute the dimensions of tensor product of two tensors?
- Dispatch b/w static checkers and runtime checkers (at best, without runtime overheads).

The first issue can be somewhat solved by type operators. Those interested can look at my [typ](https://github.com/jerry73204/typ) to see how it's possible in Rust.

For the second issue, we could interpolate runtime checkers by dispatching `UTerm + UTerm` and `UTerm + Dyn` into different `impl` blocks, and insert checkers accordingly. I did an attempt in my [type-vec](https://github.com/jerry73204/rust-type-vec). However, once the types become more complex, writing in this way is in fact a tedious process. I would rather think of an approach that we insert _runtime provers_ only when needed in a middle of complex type operations.

Also, there is a thread in tch-rs talking about static tensor types.

> <https://github.com/LaurentMazare/tch-rs/issues/112>
>
> Inspired by Tensor Considered Harmful, I'm wondering if it's possible to build compile-time checked tensor type. There prior works to make...

---

<div class="post-metadata">

### Author: ![sighoya](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@sighoya](https://internals.rust-lang.org/u/sighoya)
#### Post date: [October 28, 2020, 4:07pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/32 "2020-10-28T16:07:19Z")

</div>

> [@jerry73204](#):
>
> Those interested can look at my [typ](https://github.com/jerry73204/typ) to see how it's possible in Rust.

Interesting

> [@jerry73204](#):
>
> How to encode complex logic in types. For example, can we statically compute the dimensions of tensor product of two tensors?

I think that this will be possible someday with const generics, when the logic only involves addition and multiplication.

> [@jerry73204](#):
>
> For the second issue, we could interpolate runtime checkers by dispatching `UTerm + UTerm` and `UTerm + Dyn` into different `impl` blocks, and insert checkers accordingly

I think too, that this will happen someday with const generics but there are other problems incured by const generics and dispatching, see [this thread](https://internals.rust-lang.org/t/future-direction-const-generics-and-dispatching/13283).

---

<div class="post-metadata">

### Author: ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)
#### Post date: [January 26, 2021, 4:07pm UTC](https://internals.rust-lang.org/t/tensors-static-typing/13114/33 "2021-01-26T16:07:27Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

[Previous page](https://internals.rust-lang.org/t/tensors-static-typing/13114.md?page=1)
