Regression testing Plotly plots with Pytest
![](/posts/plotly_pytest/feature.webp)
Table of Contents
I have long been using matplotlib to generate plots for my data. Matplotlib always serves me well for producing publication-style, static plots. For regression testing the plots produced by matplotlib, they provide this very useful image comparison decorator (see link for how it works). In short:
|
|
The decorator automatically picks up new figure objects, and compares these with the expected plots through direct image comparison. If the test fails, the difference between the plots is displayed, making it convenient to visualize the difference to help with debugging.
Regression testing in Plotly #
For interactive plots that are better suited for online dashboards, we recently started using plotly a lot more. There I wanted something similar, but to my surprise, there is no documentation or infrastructure available for regression testing plots. This meant all of our plotly plots were untested. This gave us little confidence in the robustness of the plotting code, and meant that any bugs could go unnoticed for a while. To solve this, I ended up developing a function that mimicks the matplotlib way. It uses the same directories and image comparison algorithm.
![]() | ![]() | ![]() |
The visual comparison helps to identify where the problem lies.
Code #
Below is the code that we use in one of my projects. See it in action here.
|
|
There is no decorator, but you can use it like this:
|
|
My takeaways #
- You can save plotly images using
fig.write_image('name_of_file.png')
if you have the kaleido library installed. - The code uses the inspect module to get the filename of the caller.
- Make sure to force plotly to use the same fonts. Arial is a safe choice, because of its ubiquity. It is available on most operating systems, and also on the Github CI. Initially all my tests failed because the font was completely different between my development environment and the CI.
- There is a small deviation in rendering between my environment and the CI. These are invisible to the eye in the difference image, but enough to give up to 0.5 rms difference.
- I rely on the
compare_images
function to do all the heavy lifting. My code only puts the images in the right place,compare_images
does a file based compare and produces the difference. - The code for the matplotlib decorator is very clever. Essentially the contextmanager picks up any newly generated figures generated while it was active. There is a lot of code to make the decorator work seamlessly.