Hey, Criterion.rs maintainer here. I think it’s great that you’re using Criterion.rs for this. I’m taking a bit of a break from active development for the moment but I’d be happy to assist with this work from the Criterion.rs side. Let me know if there’s anything I can help with.
integrate perf record into criterion runners (this may take a bit of effort) so that perf.rlo can display more granular metrics than just runtime
This seems like it would be useful for others as well. I don’t know much about perf specifically; can that be done in-process, or would it require running the benchmark in a sub-process?
sort out a stable JSON format (I don’t know if the criterion authors expect to keep the current files they write as a stable format)
The JSON formats are deliberately unspecified and open to change, unfortunately. They’ve already changed at least once since 0.1.0. It might be possible to support this use-case more cleanly by allowing the user to provide a custom report that would receive the data directly. There’s already something like that internally, but the API isn’t really ready for public use.
make criterion able to write data files and reports to a configurable directory
This one should be pretty easy; I think all of the necessary code is there for it, I just didn’t want to stabilize an API for it until I needed to.