Doom 4 News - Why Frame Rates Don't Tell Us Anywhere Near Enough About Game Performance

When benchmarking, it can be all too easy to get hung up on the metric of frame rates. It is, after all, the lowest hanging fruit. Not only is it the easiest metric to track, but it’s also the simplest to share and compare with others. In that sense, frames per second analysis does work, but as anyone who’s played a game with incessant micro-stuttering will attest, it doesn’t paint anywhere near the full picture.

Instead, drilling down on gaming performance should be focused on the triumvirate of frames per second, frame times, and the 99th percentile.

Let’s take a brief look at what each of these methods shows us, and why they’re useful.

Frames per second (FPS)

Universally the most popular method of benchmarking performance, average frames per second is calculated by taking the total number of frames and dividing them by the number of seconds the benchmark runs for. This provides the average frame rate over the entire course of a benchmark run. While undeniably useful, taking a five-minute benchmark and reducing it to a single figure is extremely reductionist. It provides an overall indicator of the performance level, but there is no feasible way for it to provide important information such as frame drops or stuttering.

Let’s take a one-minute benchmark as an example. Our average FPS is 60 frames per second. Ideally, this would be exactly that, 60 frames each and every second. But what if the first half of the benchmark is a nearly unplayable 15fps due to a particularly demanding scene, and the second half of the benchmark is a very smooth 105fps? The basic FPS maths here is 30 seconds x 15 (for the first half of the benchmark) plus 30 seconds x 105 (for the second half of the benchmark). This gives us a grand total of 3600 frames rendered for the minute, which is 60 frames per second. On the surface, we’ve got a fantastic frame rate, yet for half the time the game is practically unplayable. It's an extreme example, but you get the point.

For a real-world look at FPS data in action, let's take DOOM and Batman: Arkham Knight, benchmarked at 1080p screen resolution with a GeForce GTX 1060, 16GB DDR4 RAM and an Intel Core i7-5820K CPU.

DOOM Batman Arkham Knight Average FPS

Kind of similar, right?

Additional depth can be provided for FPS results by including the minimum and maximum frame rate, switching from a single data point to three data points.

Max Min Average FPS DOOM Batman Arkham Knight

This is much handier, although there are obviously two huge gaps in this data set. Specifically, we’ve got no idea how much time is spent at these high or low points, we just know that it hits them at least once.

Frame times

The second method is frame times, which is an incredibly handy metric for identifying uneven performance in a benchmark that’s otherwise providing solid results. When a game stutters it just doesn’t feel right, and anything above around 50ms frame times can start to become noticeable. Major hitches can push it upwards to 100ms and beyond, which is the equivalent of ≤10fps, albeit for just a fraction of the second. Aside from the minimum frame rate, standard FPS benchmarking has no way to convey this data aside from it dragging down the average FPS by a tiny margin.

Handily, if you’re benchmarking with a tool like FRAPS, the frame times can be automatically recorded by selecting the appropriate toggles in the Benchmark Settings. The gathered CSV data can then be viewed in spreadsheet form or, as I often do, placed into FRAFS Bench Viewer (separate to FRAPS) where it does all the hard work for you.

The overall aim here is to have the yellow line be as tight as possible. I’ve used the example of DOOM because it runs excellently, and that thick yellow line is exactly what we’re aiming for. The dotted line tagged 16.7 ms marks 60fps, and any frames below this dotted line are rendering faster than 60 frames per second. Meanwhile, 33.3 ms represents 30fps, so anything above this is being rendered at less than 30 frames per second.

DOOM Frame Times

DOOM Frame Times

There are two spikes where DOOM drops below 30fps, although these are each for a single frame. Over a 3-minute run that’s pretty decent, and rare enough that you’re going to struggle to notice it. These high frame times likely tie in with loading in new areas, such as when I took a lift to a downstairs room in DOOM. Aside from this though, you can see DOOM justifies its 87fps average frame rate, with nice consistent frame times that very rarely rise above 16.7ms.

If we take a notoriously bad performer like Batman: Arkham Knight, which has improved considerably since launch, we can see the frame times, while largely really great, lack consistency and have dozens of stutters every minute.

Batman: Arkham Knight Frame Times

Batman Arkham Knight Frame Times

Arkham Knight runs fantastically on the GTX 1060 used, averaging 77fps, but you can really feel the stutters when spinning the camera around, driving the Batmobile, or changing direction quickly. This gives us some valuable data that a simple fps figure can’t replicate.

99th Percentile

Going a step further beyond frame times is the ‘99th Percentile’. This is a method for identifying the worst 1% of performance achieved and can be used to analyse two things: firstly, it means we can omit any performance oddities and get a more accurate figure for general gameplay from the 99%; conversely, we can also identify the offending 1% of the worst frame times and see just how badly they're affecting performance.

The benefit of 99th percentile over frame times is that like fps, it’s much more quantifiable, and therefore the results can be shared and compared with other gamers. It takes into account both frame times and frames per second and delivers a millisecond value that indicates the biggest dips in performance.Unlike FPS, the figure also aligns much better with how you feel when you play a game, rather than spitting out a generic number.

The 99th Percentile can also be mapped out using a combination of FRAPS and FRAFS, which in the example of Arkham Knight we can see includes every frame that takes 20.5ms or more to render. There are far more egregious examples from this, but we can see that despite an average fps of 77, for 1% of the time the frame rate dips below 49fps.

Batman: Arkham Knight 99th Percentile

Batman Arkham Knight 99th Percentile

Compare this with DOOM where the 99th percentile is 15.2ms, still below the 16.7ms frame time required for 60fps.

DOOM 99th Percentile

DOOM 99th Percentile

As with frame times, we want this chart to be as flat as humanly possible. The worst the performance is, the higher the peak is at the end. If you benchmark a game and the percentile graph begins to result in the graph looking like the curve of a hockey stick, you're going to be suffering serious performance issues.

If you then want to drill this data down to comparison graphs, this can still be achieved with the average frame times, 99th percentile frame times, and the worst offender - 0.1% percentile frame times.

DOOM and Batman: Arkham Knight 99th Percentile Performance Comparison

(the lower the better)

99th Percentile Doom Batman Arkham Knight

How Frame Rates, Frame Times and 99th Percentiles can Combine

Combine all three of these methods of analysing frame rates and we get a much more accurate picture of performance in DOOM and Batman: Arkham Knight than the simple 87fps and 77fps average frame rates provide. We know that the minimum frame rate is 65fps and 53fps respectively. We can also see the frames are much more evenly distributed while playing DOOM, with several harsh spikes occurring while playing Batman: Arkham Knight.

Finally, with the analysis of the 99th percentile, we get a good picture for how often both these games actually drop below their minimum frames per second. It sounds counter-intuitive, but we’re breaking it down to thousandths of a second rather than entire seconds. For 1% of the time, Batman is 49fps or lower, while DOOM is 66fps or lower. For 0.1% of the time, Batman dips all the way down to 26fps, while DOOM holds steady at 60fps. Despite just a 13% difference in average FPS, at its worst, Batman runs 55% slower than DOOM.

This is all crucial information that frames per second just cannot provide.