The Rust Project needs much better visibility into important metrics

dikaiosune 2016-05-31 21:28:18 UTC #29

I do have plenty of data for the nightlies (going back to 5/15/15). I would need to refactor the summary endpoint to accept multiple different date ranges. But that gets to another of your points...

I agree that a tabbed interface is where it should go. The current UI is what I believe those startup folks would call an MVP. I'm thinking it'll be worth it to break it out into more pages once more data is added...but I do view it on a portrait monitor normally so maybe I'm biased.

I looked at the highcharts docs and didn't immediately see a way to force the legend always on, but I will do some more digging. The Linux CI builders could be a bit of a mess though. Maybe the CI graphs should get broken down a little more by 32 vs 64bit?

A release countdown ain't a bad idea. I could just manually plot out the 6 weeks interval in the database for the next 10 years. How are weekdays vs. weekends accounted for with the release cadence?

I'll add some spacing to the graphs (and enlarge the titles, per feedback on IRC) tonight if all goes well.

@brson: The nightly release grid now has a green square for today. I can always up the interval at which that is refreshed, I think it's like every 6 hours right now. That said, since there doesn't appear to be a way to differentiate between "nightly build failed" and "no nightly yet" without going to the CI, I'm not sure what's the best course of action.

Adjustable date range is on my radar -- going to work on it after Let's Encrypt. HTTPS all the things!

Do you mean build failures for a specific nightly release? Or all auto- build failures for a day? For the former, which builders are used for those? For the latter, that shouldn't be a problem.

The more feedback the better! As a mostly-lurker of the Rust project, I'm sure others are better prepared than I am to assess what is or isn't useful.

Azerupi 2016-05-31 21:51:00 UTC #30

Alternatively there could be a way to hide show certain graphs. This could go hand in hand with always on legends. For example if you click on one of the names in the legends it gets hidden / shown. However I doubt that this functionality is procured by the charting library. It would probably require a bit of custom work.

Yes, that's what I thought of too. The only concern is if, for whatever reason, a release gets delayed or the interval between releases changes you would have to modify it manually.

Another idea that might be worth exploring is an online calendar you can subscribe to.
I am not sure if the Rust team has an official calendar people can follow? They could mark the releases on the calendar and allow people to subscribe to it. I am not sure how it works exactly but my university has the schedules available as a calendar that you can subscribe too. They provide a link to a .ics file and you can add it to Google Calendar for example. I suppose you could parse those files to extract important dates, like release dates.

It's probably way more difficult to implement, but it has the advantage of not having to modify it manually if the release dates ever change. I am not sure it is worth going trough all this trouble, but I thought I would share it anyways

Personally, I would have reduced the text size to fit the text labels on the graphs

brson 2016-05-31 23:30:36 UTC #31

All the build failures for the day.

dikaiosune 2016-05-31 23:45:27 UTC #32

hide show certain graphs

Absolutely. I'm assuming you mean individual series here within a graph though? Because tabs would hide/show whole graphs nicely I think. With Highcharts (could always swap out the plotting library) controlling individual series is very doable. Not very nice to do, but doable.

provide a link to a .ics file

@brson is there a publicly visible place (like a google calendar) where the release cadence is tracked?

reduced the text size

Sure, that's doable too. I erred on the side of readability, and also the CSS starting point I used encourages larger text. Could definitely go smaller though.

Azerupi 2016-06-01 00:01:02 UTC #33

Yes that's what I meant

brson 2016-06-02 15:43:19 UTC #34

@dikaiosune

Some more ideas based on recent conversations.

What are the prospects for expanding our data collection to rust-lang projects beyond rust-lang/rust? We put comparatively little effort into these repositories, so making it easier to see what's going on in them at a glance could help us identify projects that need attention.

As an example we might create a page that graphs the open PRs in each project. Other metrics that could be helpful are open issues, untriaged issues, issues/PRs that haven't been updated since X days.

Another thing I'm interested in is who is working on what. We've always been terrible at tracking resource allocation (knowing what people are and should be working on). If we had the existing information displayed in a more useful way I think we could potentially be better at it:

Imagine a page that displays information about all issues (across the entire org) that have assignments. It shows a list of all users (and their avatars) that have assignments, and next to that shows the linked issue number and description. That's the whole thing.

From there you could get a strong sense of what the project is working on. Of course to start it would not be great because our data isn't good and we don't use assignments effectively. But once we have the visualization to monitor we'll have a lot more motivation to use issue assignments. Somebody (I volunteer) could go through periodically and just review that all the assignments are still fresh. For moco employees this would be yet another attempt to track what we are working on - just seeing that everybody has some assigned task is . cc @aturon.

Another idea. There are lots of ways to configure github to display useful aggregations of information, but they are pretty hard to discover and remember. It would be great to have these cryptic formulas in one place, and linking them from our metrics site seems like it.

brson 2016-06-02 15:48:06 UTC #35

We should also think about integrating http://perf.rust-lang.org/index.html, if not importing the data and re-visualizing it, then at least linking to it.

dikaiosune 2016-06-02 16:32:05 UTC #36

Just wanted to link to:

It's a suggestion to use bitrust's data to display information about how frequently breaking changes occur.

What are the prospects for expanding our data collection to rust-lang projects beyond rust-lang/rust?

It'd require some refactoring and a DB migration or two, but certainly not a big deal. Which repos would be good to track? What about nursery repos?

open issues, untriaged issues, issues/PRs that haven't been updated since X days

Like a triage dashboard, not just a metric dashboard? I can see that being very useful.

Imagine a page that displays information about all issues (across the entire org) that have assignments. It shows a list of all users (and their avatars) that have assignments, and next to that shows the linked issue number and description.

Another idea. There are lots of ways to configure github to display useful aggregations of information, but they are pretty hard to discover and remember. It would be great to have these cryptic formulas in one place, and linking them from our metrics site seems like it.

The average issue/PR age number link to pages like this, sorted by age (maybe should change to last activity?). The issue label counts link to pages like this. What other types of searchs do you see being useful? I'll try to include them wherever I see them as I'm adding things, but if there are specific ones it isn't hard to throw them on there.

http://perf.rust-lang.org/index.html

So, at current count there are 30 issues open on the repo, which doesn't include a few deployment things like HTTPS and maybe eventually a better backup plan than digital ocean's weekly plan. @brson, any interest in helping me triage the priority a bit? It's not that much stuff, but my time is unfortunately a bit constrained. So, it might be a few weeks before I get traction on more than a few of these suggestions, which are the highest priority to work on right now?

brson 2016-06-02 18:24:34 UTC #37

Good idea.

All repos in rust-lang, rust-lang-nursery and rust-lang-deprecated.

Yes, though I see it as metrics about triage.

I was particularly thinking of aggregations of PRs and issues across all org projects. Open PRs especially.

I will put it on my triage schedule. After using rustc-perf this morning I think it probably doesn't make sense to try to re-visualize it, and more to just fix the UI problems it has. Still I think it should be linked from here.

I've done some solicitation for more ideas and have a bunch more important stuff I'll write up soon. So excited.

I think we're going to need to make this site multiple pages, and try really hard to make the front page contain just a few really important visualizations.

brson 2016-06-03 06:29:17 UTC #38

Here's another one that would make a huge impact

brson 2016-06-06 17:33:12 UTC #39

And another:

dikaiosune 2016-06-07 07:34:03 UTC #40

Just deployed an updated version to http://rusty-dash.com/. Nothing too exciting here, paying down a little debt, converting the UI to tabs to make new additions cleaner, and merging a PR from @chriskrycho (yay!).

Now that the main front-end blocker to adding new metrics has cleared, I need to take care of a few things on the backend (multiple repo support is needed for many of the new ideas), and hopefully I'll be back to pushing features later this week.

aturon 2016-06-07 16:51:31 UTC #41

Strong +1. I think there's a ton of potential here, and organizing it in this way is going to be essential.

dikaiosune 2016-06-07 17:04:11 UTC #42

Uploaded a mutli-view version last night. Deep links (like http://rusty-dash.com/issues) are broken because of the nginx config, but I'll fix that this evening.

aturon 2016-06-07 17:31:17 UTC #43

Ahha, I missed that -- looks awesome, well done!

Once we've gotten all the info on tabs that we want, then we can maybe work out what should go on the front page (as a kind of "overview").

dikaiosune 2016-06-07 17:47:03 UTC #44

Once we've gotten all the info on tabs that we want

That could be a little while -- I've got 37 issues open in the repo, and ~18 of them will require adding new data sources or analysis in addition to the presentation work. Several more will just need new views from existing data. An initial guess at what might be most useful for teams to have on the front page could be a very good way to help me prioritize which data sources & analyses to prioritize (hint hint).

EDIT: And thanks!

brson 2016-06-08 00:17:02 UTC #45

I read through the issues and prioritized them per my own preferences.

These are high-priority and small:

User-defined date range. Completing existing features.
Link to day's build failures. Ditto.
Release cadence. Just an easy thing to do, and an easy way to answer a common question.

These are high priority and big:

Assignees tab. Useful for project management and for users.
Hot issues. Relatively easy to implement.
Feature pipeline dashboard. This is probably what many of us want the most, but it's quite involved.

cyplo 2016-06-08 05:27:03 UTC #46

Hi ! Sorry for the drive-by comment, but I'm fascinated by all the things in the metrics world. Most of the stuff I've seen is based on some combination of Grafana+timeseries db (like InfluxDB). Do you think this combo might be useful here ? Either from just looking at how Grafana implemented things to actually using Grafana as a frontend and possibly joining forces with them on improving the graphing capabilities if needed ?

Grafana allows for the specified group of users to edit the dashboards, this might allow for the load on the changes to be spread a bit between people. Not sure if all the other requirements mentioned are met (e.g. histograms seem to be in the works still)

Hope this helps more than disrupts your flow.

dikaiosune 2016-06-08 06:24:56 UTC #47

Sounds like a plan, and thanks for taking the time to go over the issues!

I'm getting some help from @chriskrycho on how to wire up the date pickers in Ember, should have that done soon. In the meantime:

Added a list to recent buildbot failures (scroll down, at the bottom of the Buildbots page). Definitely needs a nicer way to present it, but this is a start.
Fixed "deep" links in the nginx configuration
Deployed the "Triage" tab with the GitHub search links, will expand this with new metrics soon
Added benchmarks game link to useful links page.
Merged a PR from @chriskrycho to start the process of putting in the tests which I've very negligently not-built while putting this together.

dikaiosune 2016-06-08 06:42:26 UTC #48

Grafana does look super slick! I will have to take a closer look, so thank you for sharing.

I see three main issues with adopting something like Grafana:

Their SQL support doesn't seem to be done yet (but I may be wrong here). Postgres performance, deployment, backup, etc. are just so stable, well understood, and supported on cloud platforms like AWS, and SQL is a second tongue for any data junkie. Having to move away from an RDBMS (basically Postgres) would be a very tough sell in my opinion, not just because of familiarity, but also because the data we're collecting isn't purely time series.
It turns out that throwing some graphs up is the easy part of this project! I do think that having community-editable visualizations would be awesome (and I have an open pipe-dream issue to build something rudimentary along those lines). But, as you can see in this thread, the most dire needs are for visualizations/reports which combine time series data with non-time-series relational analysis (see @brson's issue about a "hot" issues tab or a feature pipeline tracker). Which really makes the work here spill outside the traditional definition of "dashboard metrics."
Rust is really quite good at a lot of this work (a bit verbose for some, yes), and building something in Rust means that a) it can more easily get contributors from its own community, and b) we have yet another showcase of an area where Rust can excel. Both of which are great value-adds for a project like this on top of the useful information it can provide.

All that said, I'm not ideologically opposed to Grafana or any other off the shelf tool for that matter, I just haven't seen one which meets all of the needs here.