-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                      flash_attn_3::fwd         3.72%      67.834us         6.74%     122.815us     122.815us      22.046us        68.90%      22.046us      22.046us           0 B           0 B      10.19 MB           0 B             1  
void cutlass::device_kernel<flash::enable_sm90_or_la...         0.00%       0.000us         0.00%       0.000us       0.000us      17.727us        55.40%      17.727us      17.727us           0 B           0 B           0 B           0 B             1  
void cutlass::device_kernel<flash::FlashAttnFwdCombi...         0.00%       0.000us         0.00%       0.000us       0.000us       4.319us        13.50%       4.319us       4.319us           0 B           0 B           0 B           0 B             1  
                                      qkv_phase1_kernel         0.00%       0.000us         0.00%       0.000us       0.000us       2.592us         8.10%       2.592us       2.592us           0 B           0 B           0 B           0 B             1  
                                            aten::fill_         1.00%      18.207us         1.52%      27.681us      13.840us       2.591us         8.10%       2.591us       1.295us           0 B           0 B           0 B           0 B             2  
void at::native::vectorized_elementwise_kernel<2, at...         0.00%       0.000us         0.00%       0.000us       0.000us       2.591us         8.10%       2.591us       1.295us           0 B           0 B           0 B           0 B             2  
                                      qkv_phase2_kernel         0.00%       0.000us         0.00%       0.000us       0.000us       2.528us         7.90%       2.528us       2.528us           0 B           0 B           0 B           0 B             1  
                                      qkv_reduce_kernel         0.00%       0.000us         0.00%       0.000us       0.000us       2.240us         7.00%       2.240us       2.240us           0 B           0 B           0 B           0 B             1  
                                       aten::empty_like         1.04%      18.998us        12.34%     224.799us      56.200us       0.000us         0.00%       0.000us       0.000us           0 B           0 B       2.50 MB           0 B             4  
                                    aten::empty_strided         3.49%      63.556us        11.29%     205.801us      51.450us       0.000us         0.00%       0.000us       0.000us           0 B           0 B       2.50 MB       2.50 MB             4  
                                            aten::empty         2.01%      36.615us         2.01%      36.615us       2.817us       0.000us         0.00%       0.000us       0.000us           0 B           0 B      10.19 MB      10.19 MB            13  
                                Activity Buffer Request        57.34%       1.045ms        57.34%       1.045ms       1.045ms       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             1  
                                       cuLaunchKernelEx         2.02%      36.835us         2.02%      36.835us      12.278us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             3  
                                               [memory]         0.00%       0.000us         0.00%       0.000us       0.000us       0.000us         0.00%       0.000us       0.000us           0 B           0 B      -2.54 MB      -2.54 MB            14  
              aten::_scaled_dot_product_flash_attention        16.30%     296.970us        35.92%     654.549us     654.549us       0.000us         0.00%      24.637us      24.637us           0 B           0 B       1.03 MB     -10.16 MB             1  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.822ms
Self CUDA time total: 31.997us
