Call for help: Cargo "airplane" mode

21

Can this be avoided by using explicitly versioned dependencies?

Welllll... It could, were it not for one little hiccup. (to be discussed)

What about when there are no dependencies at all?

Rest assured, cargo does not use the internet if you have no dependencies.

In reality, the way to avoid it is:

  • Have a valid Cargo.lock containing at least entries for all of the crates you depend on (and their recursive dependencies).
  • Each dependency required by your crate (at the versions listed in Cargo.lock) must have been downloaded at least once in the past, so that it can be found in ~/.cargo/registry/src.

This means that the times you are at biggest risk of requiring a network connection are when you are adding a dependency to a brand new crate.

$ cargo new lol
$ cd lol
$ echo 'clap = "2.0"' >>Cargo.toml
$ cargo build
    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): curl error: Couldn't resolve host 'github.com'
; class=Net (12)
warning: spurious network error (1 tries remaining): curl error: Couldn't resolve host 'github.com'
; class=Net (12)
error: failed to load source for a dependency on `clap`                         

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  curl error: Couldn't resolve host 'github.com'
; class=Net (12)

The crates.io index

Here is kinda the funny bit. If you haven't yet, go take a look inside ~/.cargo/registry/index. You'll see:

$ ls ~/.cargo/registry/index/github.com-1ecc6299db9ec823/
1   ap  bf  c-  config.json  dd  dv  el  fe  ga  gs  hj  ic  iv  jv  ks  lo  me  mv  no  oh  oy  pn  qm  rg  ry  sm  tb  tu  ul  vg  wd  wy  xn  yu  zs
2   aq  bg  c3  cp           de  dw  em  ff  gb  gt  hk  id  iw  jw  kt  lp  mf  mx  np  oi  oz  po  qn  rh  rz  sn  tc  tv  um  vi  we  x-  xo  yy  zw
3   ar  bi  c_  cq           df  dx  en  fg  gc  gu  hl  ie  ix  k2  ku  lq  mg  my  nq  oj  p-  pp  qo  ri  s-  so  td  tw  un  vk  wf  x1  xp  z3  zx
aa  as  bk  ca  cr           dg  dy  ep  fh  gd  gv  hm  if  j4  k8  kv  lr  mh  n-  nr  ok  p0  pq  qp  rj  s2  sp  te  tx  up  vl  wg  x2  xr  za  zy
ab  at  bl  cb  cs           dh  e1  er  fi  ge  gw  ho  ig  ja  ka  kw  ls  mi  n2  ns  ol  p2  pr  qr  rl  s3  sq  tf  ty  ur  vm  wh  x5  xs  zb  
ac  au  bm  cc  ct           di  e2  es  fl  gf  gy  hp  ih  jb  kb  ky  lt  mj  n_  nt  om  pa  ps  qs  rm  s_  sr  tg  tz  us  vn  wi  x8  xt  zc  
ad  av  bn  cd  cu           dj  e3  et  fm  gg  gz  hs  ik  jc  kc  l2  lu  mk  na  nu  on  pb  pt  qt  rn  sa  ss  th  u2  ut  vo  wk  xa  xv  zd  
ae  aw  bo  ce  cv           dk  ea  eu  fn  gh  h2  ht  il  je  kd  l3  lv  ml  nb  nv  oo  pc  pu  qu  ro  sb  st  ti  ua  uu  vp  wl  xb  xx  ze  
af  ax  bp  cf  cx           dl  eb  ev  fo  gi  ha  hu  im  jh  ke  la  lw  mm  nc  ny  op  pd  pv  qw  rp  sc  su  tj  ub  uv  vr  wm  xc  xy  zf  
ag  ay  bq  cg  cy           dm  ec  ex  fp  gj  hb  hw  in  ji  kh  lb  ly  mn  nd  o2  oq  pe  pw  r1  rq  sd  sv  tk  uc  ux  vs  wn  xd  xz  zh  
ah  az  br  ch  cz           dn  ed  ey  fr  gl  hc  hy  io  jl  ki  lc  lz  mo  ne  oa  or  pf  px  r2  rr  se  sw  tl  ud  v4  vt  wo  xe  ya  zi  
ai  b2  bs  ci  d-           do  ee  ez  fs  gm  hd  i1  ip  jm  kl  ld  m3  mp  nf  ob  os  pg  py  ra  rs  sf  sx  tm  ue  v8  vu  wp  xf  yd  zk  
ak  ba  bt  cj  d2           dp  ef  f1  ft  gn  he  i2  iq  jn  km  le  m4  mq  ng  oc  ot  ph  qa  rb  rt  sg  sy  to  uf  va  w5  wr  xh  ye  zl  
al  bb  bu  cl  d3           dr  eg  fa  fu  go  hf  i3  ir  jo  kn  lf  ma  mr  ni  od  ou  pi  qc  rc  ru  sh  sz  tq  ug  vc  w_  ws  xi  yi  zm  
am  bc  bw  cm  da           ds  eh  fb  fw  gp  hg  i8  is  jp  ko  li  mb  ms  nl  oe  ov  pk  qd  rd  rv  si  t2  tr  uh  vd  wa  wt  xk  yk  zo  
an  bd  by  cn  db           dt  ei  fc  fx  gq  hh  ia  it  js  kq  ll  mc  mt  nm  of  ow  pl  qe  re  rw  sk  t_  ts  ui  ve  wb  wu  xl  yo  zp  
ao  be  bz  co  dc           du  ej  fd  g-  gr  hi  ib  iu  ju  kr  lm  md  mu  nn  og  ox  pm  qi  rf  rx  sl  ta  tt  uk  vf  wc  ww  xm  yt  zq

This is a complete, local copy of the crates.io index, used to resolve compatible versions of crates and their dependencies. Needless to say, this is what cargo uses when Cargo.lock is missing something and you're offline, right?

...if you thought that, you'd be wrong. What actually happens is that cargo wants to make really really sure the registry is up to date before using it. So it tries to connect to the network, and dies if it cannot.

"well gee that sounds like a simple fix"

well gee there's also more to it than that

Suppose that the above "silly error" was fixed, so that cargo simply continued onward and used the local copy of the crates.io index in whatever state it currently is when connecting to the network fails. This, unfortunately, would very likely cause it to pick versions of crates that you have never downloaded before!

  • The last time I added clap as a dependency to a crate was months ago. cargo picked the latest version at the time, clap-2.27.1, and downloaded it.
  • Later, when adding e.g. pretty-assertions as a dependency to some other crate, cargo updated my local copy of the crates.io index, causing it to learn about many new versions of clap.
  • Now, when cargo looks in my local offline copy of the index for a version of clap to add to lol, it will pick the latest version it sees---say, 2.31.0.
  • It looks in the download cache, but clap-2.31.0 is nowhere to be found! Uh oh, time to start downloading!

...ideally, when we are in offline mode, we would want cargo to choose clap-2.27.1, the one that exists in the download cache, so that downloading is not necessary. From reading the mentoring instructions linked in the first post, it appears that a good deal of the work necessary goes into fixing paper cuts like this.

Working around the limitations today

Original content

If you really need to get something up and running while you're offline, here's what you do.

  • Find the biggest crate you've ever built on your local machine. Something with at least all of the dependencies you need.
  • Copy its Cargo.lock over into the new crate.
  • Edit Cargo.toml if necessary to make sure the versions it lists in there are semver compatible with the ones available in the lockfile.

That's it. As long as all of the dependencies have an entry in there, you don't need to remove anything from the lockfile. When you do cargo build it will add the missing entry for the current crate, and start building dependencies from the cached downloads.

(if you need to splice together multiple Cargo.locks, be careful; you'll need to edit it to make sure it does not contain multiple semver-compatible versions of any crate!)

Edit:

Offline mode has been implemented!

Thanks to @ehuss for correcting me on this: Offline mode was in fact implemented not too long after the posting of this thread, and is currently available through the flag -Z offline on the nightly channel. Please give it a try!

5 Likes
22

You could also install cargo-local-registry, copy ~/.cargo/registry/cache/*/*.crate to a separate directory, set it up as a local registry, and use ~/.cargo/config to refer to that instead of crates.io.

23

Interesting –

Could this problem be solved by adding a new cargo subcommand, then, that would simply explicitly request a new cargo.lock file be generated? I’m thinking something like this:

cargo lock-deps [--update-reg] [--update-toml]

     --update-reg    -     Attempt to connect to crates.io (or another online registry)
                           to update the local registry cache entries. Without this command,
                           the operation will be entirely offline.
     --update-toml  -      Automatically update entries in cargo.toml to the latest dependency
                           versions available. (Does there already exist a subcommand to
                           do this?)

An error would be raised if a dependency required by cargo.toml isn’t already cached and either --update-reg has not been supplied or the online registry can’t be reached.

Can cargo.lock contain superfluous dependencies? If so, the generated cargo.lock file should probably contain the latest available versions of any dependencies not already listed in cargo.toml just to facilitate further online development of the crate in question.

It might even make sense to provide an option to regress dependencies in cargo.toml (perhaps with an interactive confirmation prompt) if the requested version isn’t available in the cache but an earlier version is.

24

That's kind of putting the cart before the horse; the greater issue is that cargo does not currently have the ability to resolve a working dependency tree that only picks versions with cached downloads. (and once it does, it might as well just do that when you cargo build while offline).

     --update-toml  -      Automatically update entries in cargo.toml to the latest dependency
                          versions available. (Does there already exist a subcommand to
                          do this?)

I think there are user-contributed cargo subcommands out there that do stuff like this. Last time I checked, cargo itself does not ever touch Cargo.toml, and I'd be kind of surprised if it ever began doing so in the future, because I think modifying a config file (with comments and such) is a difficult problem. (are there any roundtrip Toml parsers out there for rust yet?)

Yes and no. To be precise, it can contain superfluous dependencies, as well as unused versions of dependencies... but it cannot contain two semver-compatible versions of the same dependency. E.g. it could have serde-0.7.1, serde-0.9.0, and serde-1.0.2, but it could not have serde-0.9.1 in addition to these (it would "conflict" with 0.9.0).

That can pose challenges to naive attempts to "put everything in there", as some crates might put constraints on their dependencies which are stronger than semver (such as ~ or = dependencies).

(that said, now that I'm reading your suggestion more closely, I'm not 100% sure what problem it is trying to solve... but it sounds like it would lead to unwieldy and large lock files)

25

There exist two style-preserving toml parsing libraries I know of - Molten and toml_edit. The current version of cargo-edit uses toml_edit for (mostly) style preserving edits to Cargo.toml.

26

I'm a little confused. Offline mode (-Z offline) has been implemented and does exactly this. If it is not working correctly or not meeting people's expectations, that would be good to know. But these recent messages don't sound like they are discussing how it currently works.

1 Like
27

Ah. No, I was simply not aware of it. :slight_smile:

(D’oh! And you even mentioned it in another post just a few posts before mine!)

I edited my earlier post to reflect this in place of the hacky Cargo.lock workaround.

28

So does -Z offline work for the initial build? I got the impression from the last few posts before my first post that it does not. The point of my suggestion was to provide a way to prepare for the initial offline build, utilizing the -Z offline logic.

29

Of course it does, otherwise it’d be useless! (The initial build was the only one that ever required an internet connection in the first place)

$ rm Cargo.lock
$
$ # You can see I am offline
$ cargo +nightly build
    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): curl error: Couldn't resolve host 'github.com'
; class=Net (12)
warning: spurious network error (1 tries remaining): curl error: Couldn't resolve host 'github.com'
; class=Net (12)
error: failed to load source for a dependency on `ansi_term`                    
...

$ # -Z offline has no problem generating Cargo.lock and building
$ cargo +nightly build -Z offline
   Compiling cc v1.0.17
   Compiling unicode-xid v0.0.4
   Compiling void v1.0.2
   Compiling libc v0.2.42
   Compiling libloading v0.4.3
   Compiling lazy_static v1.0.1
   Compiling cfg-if v0.1.3
   Compiling glob v0.2.11
^C
$
$ # now that Cargo.lock exists, we no longer even require -Z offline
$ cargo +nightly build
   Compiling libc v0.2.42
   Compiling cc v1.0.17
   Compiling libloading v0.4.3
   Compiling ucd-util v0.1.1
   ...
30

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.