cuda.bindings latency benchmarks

[danielfrg]

Description

closes #1580

Description

closes #1580

@leofang @mdboom I migrated one benchmark from the pytest suite to use pyperf and added a C++ equivalent.

• Added a small benchmark discovery to find bench_*.py files with bench_*() functions
• Uses bench_time_func
• C++ benchmarks output pyperf-compatible JSON so both sides can be analyzed with the same pyperf stats / pyperf hist commands.
• The readme explain how to run it on the different envs using pixi

The benchmark is cuPointerGetAttribute, both Python and C++ call the same driver API with error checking.

These are one set of results for Python and C++ in my system, so we are ok under the <1us. They dont run the same warmup and runs for each, i still need to finish that but just to give you an idea.

# Python (pyperf bench_time_func)
bindings.pointer_attributes.pointer_get_attribute: Mean +- std dev: 603 ns +- 25 ns

# C++ (driver API baseline)
cpp.pointer_attributes.pointer_get_attribute: Mean +- std dev: 29 ns +- 1 ns

I still need to work on matching params for all the benchmarks and so on and so on but wanted to get feedback first if this looks fine to keep going.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mdboom]

I'm marking this as "approve" even though I have some questions inline and since I think it's totally fine to merge this and iterate if that's the easiest way forward.

(I am not a regular pixi user...) I tried to follow the instructions but I get:

 pixi run -e source bench
Error:   × failed to solve requirements of environment 'source' for platform 'linux-64'
  ├─▶   × failed to solve the environment
  │
  ╰─▶ Cannot solve the request because of: cuda-bindings * cannot be installed because there are no viable options:
      └─ cuda-bindings 13.1.0 would require
         └─ cuda-nvrtc >=13.2.51,<14.0a0, which cannot be installed because there are no viable options:
            └─ cuda-nvrtc 13.2.51 would require
               └─ cuda-version >=13.2,<13.3.0a0, for which no candidates were found.

 pixi run -e wheel bench
Error:   × failed to solve requirements of environment 'source' for platform 'linux-64'
  ├─▶   × failed to solve the environment
  │
  ╰─▶ Cannot solve the request because of: cuda-bindings * cannot be installed because there are no viable options:
      └─ cuda-bindings 13.1.0 would require
         └─ cuda-nvrtc >=13.2.51,<14.0a0, which cannot be installed because there are no viable options:
            └─ cuda-nvrtc 13.2.51 would require
               └─ cuda-version >=13.2,<13.3.0a0, for which no candidates were found.

[mdboom]

Maybe mention pyperf system tune here?

[danielfrg]

Thanks for the comments! I dont think we need to merge now. I'll address the comments and once we are happy with a template we have here we can commit and then in another PR i can just add more benchmarks.

[danielfrg]

Addressed the comments and i relaxed one of the deps in pixi so i think you should be able to try again.

[cpcloud]

Are these going to run in CI or in any sort of regular way? I'm not sure we should have this much additional code that is going to go stale immediately.

Can we run one iteration of the benchmarks in CI so they don't go stale?

[danielfrg]

Yes, the idea is for those to run in CI but I dont think we have decided the infra where thats going to happen, i remember having a discussion a while back but i dont remember what the problem is with the infra right now.

It sounds like a good idea to at least run one iteration for now. If thats what we want I will add it here.

[danielfrg]

@cpcloud I just added a small smoke test running one loop in CI

[danielfrg]

I'll merge in a couple of hours and then add other benchmarks in a following PR unless any objections.

[danielfrg]

Whats the "Check job status" CI?

[rwgk]

Whats the "Check job status" CI?

It'll kick in after all other GitHub Actions jobs are finished, to determine if the PR can be merged.

Give me a moment to take a look.

[rwgk]

/ok to test 43e2602

[rwgk]

I just triggered the CI. Assuming it passes, this should auto-merge.

[rwgk]

@danielfrg are you around to resolve the merge conflict? — We'll have to rerun the CI.

[danielfrg]

Resolved the conflict

[danielfrg]

/ok to test f7d98c0

[rwgk]

/ok to test e953174

[rwgk]

@danielfrg I added 4f8eef0 and merged main again — hopefully it'll go through now.

[github-actions]

Doc Preview CI
Preview removed because the pull request was closed or merged.