The Rust Project needs much better visibility into important metrics

[dikaiosune]

Thanks!

Any chance you could post a script to scrape the data ourselves, or upload the database somewhere. I'd love to tinker with the data!

Yep, all the code is on the repo, and there's a bootstrap.sql which will build a PostgreSQL DB with the GitHub data (it doesn't include CREATE DATABASE but if you point it at an empty database it will init a schema and insert all the scraped values). I'm using pg 9.5 on my machine, but I don't think I'm relying on any behavior not available in 9.3/9.4 (I could be wrong though). The code is still super rough as is the database schema, but it mostly works for now! I also uploaded a file (queries.sql) that has some test queries for most of these metrics that you can try out and mess with. Issue reports are more than welcome if you find something wrong while poking around.

Apart from point estimates like mean/median. You could try some graphs. In this case, I would plot some sort of histogram where the X axis are categories like: "closed within one hour", "one day", "one week", etc and the Y axis are the number of issues/PR in that category.

I agree that graphs are useful! I've been thinking that most of the metrics discussed here should be presented in a time series of some sort (in addition to histograms/pie charts/etc. which I agree would be super useful). I think time series are useful for dashboards because they allow one to see incremental creep that might not be noticeable otherwise. Only problem is that to have a good time series, one needs to have a decent metric for each point (thus my suggestion of a scalar metric for open time).

Yeah, you could get a percentage out of this. I think "over the past 365 days we've uploaded 360 nightly releases" sounds nicer though.

In terms of visualizations, I was picturing something like a 15x30 (or whatever) grid with red and green squares to give a quick at a glance representation (eyes are good at telling proportions, IME, and it'd be easy to see "streaks" that way too).

AFAIK, all the builds (each and every platform) have to succeed to get a nightly release. If you go ahead and parse the manifests, you'll find out when the first binary release for each platform was produced though.

I'm just finishing up right now a scraper that says "if we hit the manifest URL and get a 404, it failed, if we get a 200 the nightly build released." If it'd be useful to have a per-platform/per-binary breakdown (maybe for tier-2/3 builds?), I can add that in, but I imagine there are more important data points hiding in TravisCI and the buildbots.

[brson]

Mean and median age of PRs/issues which were closed in the last 7 days (perhaps as a time series for the last year?)
Mean and median age of PRs/issues which are still open (again with some historical data presented)

How do those sound?

This is perfect. We definitely want to see a graph over time. As you say, seeing the last year of data in 1 week chunks should be good. Though I'm definitely curious how it's looked over the entire life of the project too.

For the currently open PR's just the number representing the current average age is fine, though if possible (I'm guessing not) it would also be great to see that over time.

These two data points can probably be presented together - show the graph of average age of closed PR's then next to it show the average age of all currently-open PRs.

Personally, just the mean is fine by me.

Also, for "number of @bors retries per PR" -- based on data from 2015-05-15 to 2016-04-19 (last night), I count 416 PRs which included a bors retry, so probably too many to just present as a list. I'm curious what sort of aggregate metric would be useful to the Rust teams, as I don't think a simple average would.

I'm again interested in changes over time, so I'd suggest quantifying these as retries per PR per week (or day maybe?), where the time is the PR closing time. The actual PRs don't matter that much, though for investigative purposes it could be very nice to click through to the worst ones.

An amazing version of this might let you click on the data point in the time series, then next to the chart show the top ten PRs for that time slice.

Another question, although this time about release manifests. It looks like the historical releases (the YYYY-MM-DD folders) are just omitted if there wasn't a release for that day. Is that true? If so, is that sufficient for seeing whether the overall nightly build was broken?

Yes that is true and it is sufficient. Every day in the archives without a nightly manifest can be considered a broken nightly.

You could try some graphs. In this case, I would plot some sort of histogram where the X axis are categories like: "closed within one hour", "one day", "one week", etc and the Y axis are the number of issues/PR in that category.

A histogram does sound useful. Again it might be good to link two visualizations together. One one side you have a chart of mean performance over time, quantized to the week or day; when you click on a week it then adjusts the histogram to be for that time slice.

Yeah, you could get a percentage out of this. I think "over the past 365 days we've uploaded 360 nightly releases" sounds nicer though.

@alexcrichton and my main concern with nightly bustage is that we just never know it's happening until somebody happens to point it out, so just a big label saying either "It's been X days since nightly broke" or "Nightly has been broken for X days" would be a big improvement.

AFAIK, all the builds (each and every platform) have to succeed to get a nightly release. If you go ahead and parse the manifests, you'll find out when the first binary release for each platform was produced though.

This is not quite true. There are a few platforms that are allowed to fail. I don't have any immediate need to know this information though.

I'm so excited for this.

[dikaiosune]

I have some other questions about the discussion here, but I just want to quickly throw this one out there: should the dashboard care about the duration of failed CI builds? I would assume that the times would vary heavily depending on the stage of the failure, and so wouldn't be useful, but I wanted to check before I throw away times for failed builds.

[dikaiosune]

A not-so-quick update:

I've been busy wrapping up my semester, so I'm not as far along as I'd hoped, but this is getting close to an end-to-end prototype.

My big question right now is what to use for the presentation layer. After a conversation on IRC with @brson, I think the dashboard would work best as a JS SPA in front of the Rust API server, rather than dealing with the deployment complexity of a separate server-side application like Shiny. But there remains a panoply of routes that direction. Should I expect this to grow into a larger interactive application which needs something like Angular or Ember? Should I worry about that later and just build a dashboard MVP with jQuery? Not sure myself and I'd be eager to hear any input (if anyone is still reading this thread, that is ).

Working:

• Bootstrapping the database with data from GitHub, nightly releases, and BuildBot
• A scraper daemon which scrapes those three data sources at configurable intervals
• A JSON API server with one working endpoint (/summary/) which serves a summary of the last 120 days of activity
• A bunch of queries written mostly in Diesel (many many many thanks to @sgrif for his help), yielding pretty good performance. After parallelization, the full summary endpoint (15-20 queries + JSON marshalling) takes ~80ms per request on my desktop with an SSD, down from ~150ms when done serially (which is still pretty good though).

TODO before a deployment:

• Clean up the code for the summary endpoint -- parallelizing it has improved the speed, but it's ugly and I'm sure there's a better way. Would love to hear if anyone has better ideas about how to elegantly farm out the queries to threads.
• Extend the summary endpoint to have a configurable date range
• Write a simple JavaScript page/app/thing to grab the JSON, parse it, and graph it (still need to decide how to do this -- the KISS in me says jQuery + Highcharts/Highstock)
• Split the scraper daemon and web server out into separate executables with separate configuration (to keep separate API credentials from being known to the API server and allow for easily using a read-only DB account for the API server)
• Some other stuff I'm surely forgetting

Down the road:

• Everything else listed in this thread and in the issues on the repository, including but not limited to:
  • Convert some of the scalar metrics to time series (not sure how best to do this efficiently because SQL is a bit complex for windowing over dates)
  • Scrape many more data sources and add them to the summary
  • Include medians for the average scalar metrics
  • Many other things I'm forgetting

[matklad]

Should I expect this to grow into a larger interactive application which needs something like Angular or Ember? Should I worry about that later and just build a dashboard MVP with jQuery?

I would go with Ember. I think you would be able to prototype faster with Ember than with plain jQuery (not very reliable data point, I've touched Ember last time about two years ago). Also, crates.io is implemented in Ember.

[brson]

Hi @dikaiosune!

I have some other questions about the discussion here, but I just want to quickly throw this one out there: should the dashboard care about the duration of failed CI builds? I would assume that the times would vary heavily depending on the stage of the failure, and so wouldn't be useful, but I wanted to check before I throw away times for failed builds.

That number doesn't interest me right now.

Should I expect this to grow into a larger interactive application which needs something like Angular or Ember? Should I worry about that later and just build a dashboard MVP with jQuery?

I don't know enough about Ember to say. Personally I'd first figure out how to display the data graphically to prove that you can get the data all the way to the screen. If Ember helps with that then use Ember.

A scraper daemon which scrapes those three data sources at configurable intervals

Do you have a sense for how much data is being transferred in the scrapes? Do subsequent scrapes retransfer all the same data or are they incrementalized somehow?

A JSON API server with one working endpoint (/summary/) which serves a summary of the last 120 days of activity

How large is the summary data now?

A bunch of queries written mostly in Diesel (many many many thanks to @sgrif for his help), yielding pretty good performance. After parallelization, the full summary endpoint (15-20 queries + JSON marshalling) takes ~80ms per request on my desktop with an SSD, down from ~150ms when done serially (which is still pretty good though).

That's so cool you are using Diesel!

Write a simple JavaScript page/app/thing to grab the JSON, parse it, and graph it (still need to decide how to do this -- the KISS in me says jQuery + Highcharts/Highstock)

SGTM.

The data you've collected already has been interesting and useful!

[dikaiosune]

@brson:

I've started prototyping with ember and highcharts:

It ain't pretty yet, but it's a start. I'm hoping to have all of the core graphing in place sometime this weekend, time permitting.

Do you have a sense for how much data is being transferred in the scrapes? Do subsequent scrapes retransfer all the same data or are they incrementalized somehow?

The scrapes of GitHub and the nightly releases are incremental, and the incremental scrapes are pretty quick. GitHub provides a since keyword on most of their query strings, so I can just search the database for the most recent update, and ask for any updates since then. It works OK, although if the scraper daemon is killed in the middle of doing GitHub, then it's necessary to run the bootstrap command to get all data since the last known good scrape (that command takes a date, so it'll grab some extra, but that's OK too -- it'll just update the existing data in place). The nightly releases are super simple to check, so each scrape just looks for a YYYY-MM-DD/index.html that occurred after the last successful release. The downside there is that if a release is yanked for some reason, if the scraper found the release before the yanking, it won't update that, but that's the cost of incremental updates I think, and probably not that big a deal.

The buildbot scraping is very much not ideal -- it needs to go to each builder and ask for all builds on record, and unfortunately it doesn't look like the current API supports anything more granular. This takes about 5-15 minutes per run. I'm currently running it once every ~90 minutes. @alexcrichton kept an eye on the build cluster when I first ran it a little while ago, and it didn't seem like the API (despite taking forever) puts a lot of load on the build machines. But that may be worth revisiting. Also, to make things more interesting, the 0.9 series of buildbot (IIRC Rust is running 0.8.10) will remove the JSON API that I'm currently using and replace it with something much better. But that module will require a rewrite if/when the Rust CI system is updated past the 0.8.* series of buildbot.

It remains to be seen how incremental I can make the scrapes for the other data sources. I decided that the 3 I've got were enough to build the API server and web frontend, so I haven't read up much yet on the other sources discussed.

How large is the summary data now?

If I'm reading curl's output correctly, it looks like it's about 66KB that's currently transferred from the summary endpoint for 120 days of data. I'm in the process of reworking the database functions so that they return data in a format that's directly graphable without transformation on the front-end, so that number could go up or down by a bit.

$ curl http://localhost:8080/summary > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 65882  100 65882    0     0  1452k      0 --:--:-- --:--:-- --:--:-- 1462k

That's so cool you are using Diesel!

It's been a bit of a learning curve, but I think that it's been the best example of Rust's special sauce that I've used yet. Strong typing with clean abstractions and screaming performance.

The data you've collected already has been interesting and useful!

Glad to hear it!

[brson]

Thanks for the details. The example looks good.

66KB

This is great. I can tolerate a lot bigger transfers than that to get pretty graphs.

[dikaiosune]

A quick update! I haven't touched this in almost a month because:

1. Finals are a thing along with multiple deadlines at work.
2. Suffered a bit of an accident which has kept me from coding for a couple of weeks while I recover.

I'm starting work on the project again today, and will hopefully post back with an update in the next few days.

[dikaiosune]

I posted this screenshot to IRC, and then I realized @brson wasn't online, so including it here. This is the current state of the dashboard. AFAICT, the only metric not yet displayed but which is currently scraped is # of @bors retries per PR, and I still don't know how to visualize that well. Anyways, here's the current state of the dashboard (my portrait monitor isn't tall enough to include the CI Failures section, but it's below CI Times):

The next step is to move this off of my desktop to a source online. I know that it was previously discussed about having it on AWS somewhere, but in the interim I'm going to be hosting it on a Digital Ocean VM since it's convenient and cheap. I'll post back once it's online.

[alexcrichton]

Holy cow that looks awesome @dikaiosune! Can't wait to get to play around with it

[dikaiosune]

OK, initial version is live at http://rusty-dash.com/ (ping @brson @alexcrichton). @ubsan suggested on IRC to make the grid of nightly release builds link to the relevant CI build pages, so I opened a GitHub issue.

Need to enable HTTPS soon, and then I'll carve out some time to make the date range selectable after the initial page load. After that, I'll start tackling other data sources and visualizations for them.

[Azerupi]

That's so cool!

A couple of suggestions:

• The two first sections (Nightly Releases & Issues) look ridiculously large compared to the rest. The blocks could be smaller, maybe have a calendar like the GitHub activity graph if you have enough data.
• Is it possible to display legends for the graphs? I saw that they would appear on hover, but it would be nice if they could be shown by default.
• The graphs could have a little more spacing in my opinion. I found margin-top: 40px looks better.
• A countdown (in days) until next planned release with the current and next release numbers? I am not sure if it belongs on this page, but I think it is a nice bit of info to display.

Also since you are using a JavaScript Framework, I think it might be interesting to explore a design with "tabs" instead of displaying everything on one page where you have to scroll down. It could look a bit like what I used in this web app (the sidebar tabs).

[brson]

This looks fantastic @dikaiosune. Thank you so much.

The nightly releases grid is great. It looks like it may get the wrong value for the present day (since it hasn't been built yet), but that's not a big deal.

Some further ideas:

• It would be great if one could expand the date range on the charts so we could see the trends up to a year in the past.
• Some way to get direct links to the build failures for a specific day.

I'll bring this up in various meetings this week to see what others think.

[dikaiosune]

@Azerupi:

I do have plenty of data for the nightlies (going back to 5/15/15). I would need to refactor the summary endpoint to accept multiple different date ranges. But that gets to another of your points...

I agree that a tabbed interface is where it should go. The current UI is what I believe those startup folks would call an MVP. I'm thinking it'll be worth it to break it out into more pages once more data is added...but I do view it on a portrait monitor normally so maybe I'm biased.

I looked at the highcharts docs and didn't immediately see a way to force the legend always on, but I will do some more digging. The Linux CI builders could be a bit of a mess though. Maybe the CI graphs should get broken down a little more by 32 vs 64bit?

A release countdown ain't a bad idea. I could just manually plot out the 6 weeks interval in the database for the next 10 years. How are weekdays vs. weekends accounted for with the release cadence?

I'll add some spacing to the graphs (and enlarge the titles, per feedback on IRC) tonight if all goes well.

@brson: The nightly release grid now has a green square for today. I can always up the interval at which that is refreshed, I think it's like every 6 hours right now. That said, since there doesn't appear to be a way to differentiate between "nightly build failed" and "no nightly yet" without going to the CI, I'm not sure what's the best course of action.

Adjustable date range is on my radar -- going to work on it after Let's Encrypt. HTTPS all the things!

Do you mean build failures for a specific nightly release? Or all auto- build failures for a day? For the former, which builders are used for those? For the latter, that shouldn't be a problem.

The more feedback the better! As a mostly-lurker of the Rust project, I'm sure others are better prepared than I am to assess what is or isn't useful.

[Azerupi]

The Linux CI builders could be a bit of a mess though. Maybe the CI graphs should get broken down a little more by 32 vs 64bit?

Alternatively there could be a way to hide show certain graphs. This could go hand in hand with always on legends. For example if you click on one of the names in the legends it gets hidden / shown. However I doubt that this functionality is procured by the charting library. It would probably require a bit of custom work.

A release countdown ain't a bad idea. I could just manually plot out the 6 weeks interval in the database for the next 10 years.

Yes, that's what I thought of too. The only concern is if, for whatever reason, a release gets delayed or the interval between releases changes you would have to modify it manually.

Another idea that might be worth exploring is an online calendar you can subscribe to.
I am not sure if the Rust team has an official calendar people can follow? They could mark the releases on the calendar and allow people to subscribe to it. I am not sure how it works exactly but my university has the schedules available as a calendar that you can subscribe too. They provide a link to a .ics file and you can add it to Google Calendar for example. I suppose you could parse those files to extract important dates, like release dates.

It's probably way more difficult to implement, but it has the advantage of not having to modify it manually if the release dates ever change. I am not sure it is worth going trough all this trouble, but I thought I would share it anyways

and enlarge the titles, per feedback on IRC

Personally, I would have reduced the text size to fit the text labels on the graphs

[brson]

Do you mean build failures for a specific nightly release? Or all auto- build failures for a day? For the former, which builders are used for those? For the latter, that shouldn't be a problem.

All the build failures for the day.

[dikaiosune]

hide show certain graphs

Absolutely. I'm assuming you mean individual series here within a graph though? Because tabs would hide/show whole graphs nicely I think. With Highcharts (could always swap out the plotting library) controlling individual series is very doable. Not very nice to do, but doable.

provide a link to a .ics file

@brson is there a publicly visible place (like a google calendar) where the release cadence is tracked?

reduced the text size

Sure, that's doable too. I erred on the side of readability, and also the CSS starting point I used encourages larger text. Could definitely go smaller though.

[Azerupi]

I'm assuming you mean individual series here within a graph though?

Yes that's what I meant

[brson]

@dikaiosune

Some more ideas based on recent conversations.

What are the prospects for expanding our data collection to rust-lang projects beyond rust-lang/rust? We put comparatively little effort into these repositories, so making it easier to see what's going on in them at a glance could help us identify projects that need attention.

As an example we might create a page that graphs the open PRs in each project. Other metrics that could be helpful are open issues, untriaged issues, issues/PRs that haven't been updated since X days.

Another thing I'm interested in is who is working on what. We've always been terrible at tracking resource allocation (knowing what people are and should be working on). If we had the existing information displayed in a more useful way I think we could potentially be better at it:

Imagine a page that displays information about all issues (across the entire org) that have assignments. It shows a list of all users (and their avatars) that have assignments, and next to that shows the linked issue number and description. That's the whole thing.

From there you could get a strong sense of what the project is working on. Of course to start it would not be great because our data isn't good and we don't use assignments effectively. But once we have the visualization to monitor we'll have a lot more motivation to use issue assignments. Somebody (I volunteer) could go through periodically and just review that all the assignments are still fresh. For moco employees this would be yet another attempt to track what we are working on - just seeing that everybody has some assigned task is . cc @aturon.

Another idea. There are lots of ways to configure github to display useful aggregations of information, but they are pretty hard to discover and remember. It would be great to have these cryptic formulas in one place, and linking them from our metrics site seems like it.