[SiliconFlow] New Feat: Add vendor-aware skip_native marker for native baselines

[winfan1314]

PR Category

Benchmark

Type of Change

New Feature

Description

This PR adds a function-level skip_native benchmark marker:

@pytest.mark.skip_native(
    vendors=("ascend",),
    reason="aten::unique_dim falls back to CPU on Ascend",
)

• Requires vendors and a non-empty reason, with invalid configurations reported during test setup.
• Skips only the native latency measurement when the current vendor matches; the FlagGems benchmark still runs normally.
• Records native latency, native GBPS, and speedup as null/N/A, and emits native_baseline_skip_reason only when the policy is active.
• Keeps inactive vendor markers out of pytest marker selection, operator IDs, reports, and --collect-marks output.
• Updates tools/run_tests.py and tools/psum_text to preserve Gems results and render skipped native metrics safely.
• Applies the marker to both unique_dim benchmark variants on Ascend.
• Adds tests/test_benchmark_skip_native.py for marker validation, execution behavior, serialization, parsing, and text summary coverage.

When no skip_native marker is active, benchmark execution, JSON schema, parser behavior, and text output remain unchanged. The ordinary parser and reporting outputs were compared against the pre-change implementation and are byte-identical.

Validation:

• CANN 8.5 benchmark framework tests: 17 passed.
• CANN 9.0 benchmark framework tests: 17 passed.
• Unmarked Ascend abs benchmark: 1 passed; native latency, Gems latency, and speedup were all recorded, with no new JSON field.
• Inactive-vendor collection: both unique_dim benchmark functions remained collected under -m "not skip_native".
• isort, black, flake8, py_compile, and git diff --check: passed.

Issue & PR

• Associated with [SiliconFlow] Fix Ascend unique_dim on CANN 8.5 and 9.0 #5354

Progress

• Change is properly reviewed (1 reviewer required, 2 recommended).
• Change is responded to an issue.
• Change is fully covered by a UT.

Note on unique_dim

The unique_dim-related changes are temporarily retained in this PR only to provide a real benchmark entry point for validating the skip_native mechanism. They will be removed from this PR in a subsequent review update.

The unique_dim operator fix and its updates are tracked separately in PR #5354.

Performance

• CANN 8.5: 2 passed; 20 performance points succeeded.
• CANN 9.0: 2 passed; 20 performance points succeeded.

CANN 8.5 raw performance stdout

Operator: unique_dim  Performance Test (dtype=torch.int16, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            6.540500                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A           24.227909                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          136.006700                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          174.425455                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          140.117273                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})

Operator: unique_dim  Performance Test (dtype=torch.int32, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            6.730571                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A           18.574818                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          131.998700                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          170.891591                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A          223.481211                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})

Operator: unique_dim  Performance Test (dtype=torch.int16, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            7.625727                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A           18.364091                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A          124.067909                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A          111.669455                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A           22.545250                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})

Operator: unique_dim  Performance Test (dtype=torch.int32, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            7.207500                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A           15.590857                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A           80.979667                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A          147.723818                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A           22.535500                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})

================== 2 passed, 3 warnings in 152.14s (0:02:32) ===================

CANN 9.0 raw performance stdout

Operator: unique_dim  Performance Test (dtype=torch.int16, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            0.007131                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.018634                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.132801                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.130532                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.159920                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})

Operator: unique_dim  Performance Test (dtype=torch.int32, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            0.006485                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.018259                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.128038                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.139543                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})
SUCCESS                    N/A            0.207831                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 0})

Operator: unique_dim  Performance Test (dtype=torch.int16, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            0.007434                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.020716                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.130494                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.126295                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.472944                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})

Operator: unique_dim  Performance Test (dtype=torch.int32, mode=kernel,level=core)
Native baseline: N/A (aten::unique_dim falls back to CPU on Ascend)
Status       Torch Latency (ms)    Gems Latency (ms)         Gems Speedup          Size Detail
-----------------------------------------------------------------------------------------------
SUCCESS                    N/A            0.007175                 N/A          ([torch.Size([64, 64])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.018098                 N/A          ([torch.Size([256, 256])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.117193                 N/A          ([torch.Size([1024, 1024])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.135893                 N/A          ([torch.Size([4096, 4096])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})
SUCCESS                    N/A            0.484371                 N/A          ([torch.Size([1024, 65536])], {'sorted': True, 'return_inverse': True, 'return_counts': False, 'dim': 1})

================== 2 passed, 3 warnings in 427.30s (0:07:07) ===================