Working from memory here, so this is not an official statement from the team. The Cargo team has discussed this several times. So Yes we agree that the current private registries is not sufficient for all users. As I recall the reason there has not been work done on this in the past is that we don’t have the bandwidth to find the relevant stakeholders and come up with a shared design. ( not to mention the lack of bandwidth to do the implementation. ) If you want to spearhead that design / implementation work we are open to working with you!
Now for some details:
Configuration a way to send some authorization with more endpoints is defintly needed. Open questions include how should it be configured, what forms of authorizations should it support, does this leave the api open to MITM or Credential stuffing attacks, what threat model do we need to be robust to… All theas need to be answered for the current users and the foreseeable upcoming users.
As for non-Git indexes, this may be more complicated than it first appears. One piece of background is that many different people (think thay) want this, ech for different reasons, and they end up talking past each other. There are some important design constraints on how the index works given the design of Cargo. For example the first interaction of Cargo with an index is to do a git clone; backwards compatibility means that it needs to work with indexes that don’t know about the new api. An additional example is that Cargos resolver is assumed to be fast ( at least for the happy path ) and gets called quite a lot, at least once per build, so a new api will need to have a persistent on disk copy and a reliable way to know if it is still up to date.
One thing that makes the Git index less onerous than if may at first appear is that Cargo always does a pull from master. So the returned commit does not need to be from the same tree as anything that was previously sent. (see Cargo's crate index: upcoming squash into one commit) If for example the url has a username in it and the server needs to figure out what that user is allowed to see. It is entirely correct for a custom registry to return a totally new git repo with one commit on each request.