Unstable results in GitHub actions due to "Different runtime environments detected"

[provinzkraut]

We're using codspeed with the GitHub Actions integration, running benchmarks with in simulation, which, if I understand correctly, is meant to reduce variance. However, we currently have to re-run workflows a couple of times to get stable results, because they seem to always be off when they're not run in the same environment as the baseline benchmarks, which codspeed also mentions in its report with the message

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

This feels like a stark limitation of the usefulness of this tool, given my understanding that the CPU simulation is specifically meant to address this type of variance.

You can see such a run here: msgspec/msgspec#1052 (comment).

This is our current setup: https://github.com/msgspec/msgspec/blob/978c6671814524da8b748630a8ea142c56956c94/.github/workflows/codspeed.yml

---

Am I missing something, or are we doing something wrong with our setup?

[GuillaumeLagrange]

Hello!

Unfortunately, you are running into allocator variance, one of the issue described by @not-matthias in this article: https://codspeed.io/blog/why-glibc-faster-github-actions

We do provide some mitigations here with the GLIBC_TUNABLES env var.

Overall, we do our best to bring reliable measurement across different types of hardware, and are continuously working on bridging this gap. Until then, if you have a reliable way to have consistent hardware, at least a consistent CPU, it will suppress these issues. I see you're trying out the macro-runners in msgspec/msgspec#1079 but there's a small setup step that needs to be done in order to use them on public repositories: https://codspeed.io/docs/instruments/walltime#usage-on-public-repositories

[provinzkraut]

Thanks @GuillaumeLagrange, I'll definitely give that a try!

The macro runners would be a last resort for us, as it seems that only arm-based ones are available, and we deliberately want to test on different architectures

[GuillaumeLagrange]

Understood, we do have more macro-runner variants coming up, including x86 ones. We're in the process of rolling them out and ironing out the details on them, so we don't have a precise ETA yet.

[GuillaumeLagrange]

On the matter of the allocator variance, we do plan to filter out allocation metrics from the simulation instrument, with the possibility to use the memory instrument if one wants to deep dive into memory.

[provinzkraut]

Understood, we do have more macro-runner variants coming up, including x86 ones. We're in the process of rolling them out and ironing out the details on them, so we don't have a precise ETA yet.

That would be very nice :)

[provinzkraut]

@GuillaumeLagrange I've applied some of the suggested fixes, and they seem to have improved things. We'll monitor if results are more stable long term now. Will be closing this for the moment. Thanks for the advice :)

[GuillaumeLagrange]

I'm glad to hear that. Let us know if you still encounter issues, we are actively monitoring allocator variance problems while we work on a more user friendly/failsafe solution.