EPIC: Tests that never run in CI

[mdboom]

#1969 adds a step to CI that will analyze the logs to find all tests that are skipped in our CI under all configurations. The complete list of tests (as of today) is copy-pasted below.

This issue is to start a discussion about what to do about them.

Removal

I am not advocating a rush to remove them -- they often do run when QA runs them on different platforms than we have in CI. However, some of these may be low-value and we could consider removing on a case-by-case basis. Bitrotting of tests is as much a concern as bitrotting of production code.

Annotation

I think it would be helpful to add a comment to all of these tests that they are not covered in CI. That would be a helpful signal when editing a test, or seeing a failure in QA, that it is not well-covered. An agent could easily take this list and create all those comments -- we can remove the comments if any of the tests are reworked to run in CI. The existing tool that analyses logs for skipped tests could surface the fact that a test has an annotation that it always skips when that is no longer the case.

Hardware/platform coverage

Does this list point at any hardware that would help us get better coverage? As we determine /what kinds/ of hardware to add, this may be helpful in determining what would provide the most value.

Software coverage

I think some of these aren't covered because they require a dependency, such as pytorch. That's a heavy dependency we probably don't want to test with /everywhere/, but maybe we could add it in a few configurations to get more test coverage.

Always skipped tests

• cuda_bindings/tests/nvml/test_compute_mode.py::test_compute_mode_supported_nonroot
• cuda_bindings/tests/nvml/test_device.py::test_device_get_unrepairable_memory_flag
• cuda_bindings/tests/nvml/test_device.py::test_device_vgpu_get_heterogeneous_mode
• cuda_bindings/tests/nvml/test_device.py::test_get_nv_link_supported_bw_modes
• cuda_bindings/tests/nvml/test_device.py::test_read_prm_counters
• cuda_bindings/tests/nvml/test_device.py::test_read_write_prm
• cuda_bindings/tests/nvml/test_device.py::test_set_power_management_limit
• cuda_bindings/tests/nvml/test_pci.py::test_bridge_chip_info
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_attributes
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_P2P_ATOMICS:
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_P2P_SUPPORTED:
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_SLI_BRIDGE:
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_SYSMEM_ACCESS:
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_SYSMEM_ATOMICS:
• cuda_bindings/tests/nvml/test_pynvml.py::test_device_get_nvlink_capability[<NvLinkCapability.NVLINK_CAP_VALID:
• cuda_bindings/tests/test_cuda.py::test_always_skip
• cuda_bindings/tests/test_cuda.py::test_cuDevSmResourceSplit
• cuda_bindings/tests/test_cuda.py::test_cuGraphExecGetId
• cuda_bindings/tests/test_cuda.py::test_cuGraphGetId
• cuda_bindings/tests/test_cuda.py::test_cuGraphNodeGetContainingGraph
• cuda_bindings/tests/test_cuda.py::test_cuGraphNodeGetLocalId
• cuda_bindings/tests/test_cuda.py::test_cuGraphNodeGetToolsId
• cuda_bindings/tests/test_cuda.py::test_cuStreamGetDevResource
• cuda_bindings/tests/test_cudart.py::test_cudaDevResourceGenerateDesc
• cuda_bindings/tests/test_cudart.py::test_cudaDevSmResourceSplit
• cuda_bindings/tests/test_cudart.py::test_cudaDevSmResourceSplitByCount
• cuda_bindings/tests/test_cudart.py::test_cudaDeviceGetDevResource
• cuda_bindings/tests/test_cudart.py::test_cudaExecutionCtxGetDevResource
• cuda_bindings/tests/test_cudart.py::test_cudaExecutionCtxGetDevice
• cuda_bindings/tests/test_cudart.py::test_cudaExecutionCtxGetId
• cuda_bindings/tests/test_cudart.py::test_cudaExecutionCtxStreamCreate
• cuda_bindings/tests/test_cudart.py::test_cudaGraphConditionalHandleCreate_v2
• cuda_bindings/tests/test_cudart.py::test_cudaGraphExecGetId
• cuda_bindings/tests/test_cudart.py::test_cudaGraphGetId
• cuda_bindings/tests/test_cudart.py::test_cudaGraphNodeGetContainingGraph
• cuda_bindings/tests/test_cudart.py::test_cudaGraphNodeGetLocalId
• cuda_bindings/tests/test_cudart.py::test_cudaGraphNodeGetToolsId
• cuda_bindings/tests/test_cudart.py::test_cudaGreenCtxCreate
• cuda_bindings/tests/test_cudart.py::test_cudaStreamGetDevResource
• cuda_bindings/tests/test_utils.py::test_check_nvvm_compiler_options_no_libnvvm
• cuda_core/tests/example_tests/test_basic_examples.py::test_example[gl_interop_plasma.py]
• cuda_core/tests/example_tests/test_basic_examples.py::test_example[pytorch_example.py]
• cuda_core/tests/example_tests/test_basic_examples.py::test_example[simple_multi_gpu_example.py]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs[empty]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs[host_callback_cfunc]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[empty]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[host_callback_cfunc]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[if_else]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[if_then]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[switch]
• cuda_core/tests/graph/test_graph_definition.py::test_node_attrs_preserved_by_nodes[while_loop]
• cuda_core/tests/graph/test_graph_definition_lifetime.py::test_reconstructed_body_survives_parent_deletion
• cuda_core/tests/system/test_system_device.py::test_auto_boosted_clocks_enabled
• cuda_core/tests/system/test_system_device.py::test_clock
• cuda_core/tests/system/test_system_device.py::test_device_attributes
• cuda_core/tests/system/test_system_device.py::test_numa_node_id
• cuda_core/tests/system/test_system_device.py::test_nvlink
• cuda_core/tests/system/test_system_device.py::test_temperature
• cuda_core/tests/test_checkpoint.py::TestCheckpointGpuMigration::test_rotation_migrates_context
• cuda_core/tests/test_checkpoint.py::TestCheckpointGpuMigration::test_swap_identical_gpus
• cuda_core/tests/test_graphics.py::TestErrorHandling::test_close_while_mapped_passes_stream_override
• cuda_core/tests/test_graphics.py::TestFromGLBuffer::test_close_is_idempotent
• cuda_core/tests/test_graphics.py::TestFromGLBuffer::test_register_default_flags
• cuda_core/tests/test_graphics.py::TestFromGLBuffer::test_register_write_discard
• cuda_core/tests/test_graphics.py::TestFromGLImage::test_register_image
• cuda_core/tests/test_graphics.py::TestMapUnmap::test_from_gl_buffer_with_stream_context_manager
• cuda_core/tests/test_graphics.py::TestMapUnmap::test_strided_memory_view_from_mapped_buffer
• cuda_core/tests/test_green_context.py::TestGreenContextKernelLaunch::test_with_workqueue_resource
• cuda_core/tests/test_green_context.py::TestWorkqueueResource::test_configure_invalid_scope_raises
• cuda_core/tests/test_green_context.py::TestWorkqueueResource::test_configure_none_is_noop
• cuda_core/tests/test_green_context.py::TestWorkqueueResource::test_configure_valid_scope
• cuda_core/tests/test_green_context.py::TestWorkqueueResource::test_query
• cuda_core/tests/test_linker.py::test_linker_link_ptx_culink
• cuda_core/tests/test_linker.py::test_linker_options_as_bytes_driver_not_supported
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-release_threshold-int-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-reserved_mem_current-int-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-reserved_mem_high-int-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-reuse_allow_internal_dependencies-bool-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-reuse_allow_opportunistic-bool-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-reuse_follow_event_dependencies-bool-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-used_mem_current-int-True]
• cuda_core/tests/test_memory.py::test_mempool_attributes[ManagedMR-used_mem_high-int-True]
• cuda_core/tests/test_memory_peer_access.py::test_peer_access_transitions
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[0]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[120]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[128]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[277]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[32]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[64]
• cuda_core/tests/test_module.py::test_occupancy_max_potential_block_size_b2dsize[96]
• cuda_core/tests/test_object_protocols.py::test_repr_format[sample_object_code_ptx-<ObjectCode
• cuda_core/tests/test_object_protocols.py::test_repr_format[sample_program_ptx-<Program
• cuda_core/tests/test_program.py::test_cpp_program_with_various_options[options31]
• cuda_core/tests/test_tensor_map.py::TestTensorMapIm2colWide::test_from_im2col_wide_3d
• cuda_core/tests/test_tensor_map.py::TestTensorMapIm2colWide::test_from_im2col_wide_4d
• cuda_core/tests/test_tensor_map.py::TestTensorMapIm2colWide::test_from_im2col_wide_5d
• cuda_core/tests/test_tensor_map.py::TestTensorMapIm2colWide::test_from_im2col_wide_rank_validation
• cuda_core/tests/test_utils.py::TestViewCudaArrayInterfaceGPU::test_cuda_array_interface_gpu[NOTSET]
• cuda_core/tests/test_utils.py::TestViewGPU::test_args_viewable_as_strided_memory_gpu[NOTSET]
• cuda_core/tests/test_utils.py::TestViewGPU::test_strided_memory_view_cpu[NOTSET]
• cuda_core/tests/test_utils.py::TestViewGPU::test_strided_memory_view_init[NOTSET]
• cuda_core/tests/test_utils.py::test_ml_dtypes_bfloat16_torch_dlpack[contiguous]
• cuda_core/tests/test_utils.py::test_ml_dtypes_bfloat16_torch_dlpack[strided]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[bfloat16]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[bool]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[complex128]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[complex64]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[float16]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[float32]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[float64]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[int16]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[int32]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[int64]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[int8]
• cuda_core/tests/test_utils.py::test_torch_tensor_bridge_dtypes[uint8]
• cuda_pathfinder/tests/test_driver_lib_loading.py::test_real_load_driver_lib[nvcudla]

[mdboom]

This investigation may or may not help with some of the NVML-related tests above: #2009

[leofang]

Some of these are related to the fact that we don't have 13.2 driver installed anywhere in our runners. I have raised PRs on the runner side (https://github.com/nv-gha-runners/vm-images/pull/247, https://github.com/nv-gha-runners/arc-nvks-argocd/pull/429) but I haven't had time to discuss with them. It might be possible that we need to roll our own infra (#1265, #293).