-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                         aten::_flash_attention_forward         2.34%      34.794us        96.15%       1.431ms       1.431ms      17.631us       100.00%      35.262us      35.262us           0 B           0 B       1.03 MB      -4.06 MB             1  
                                Activity Buffer Request        89.56%       1.333ms        89.56%       1.333ms       1.333ms      17.631us       100.00%      17.631us      17.631us           0 B           0 B           0 B           0 B             1  
void pytorch_flash::flash_fwd_splitkv_kernel<Flash_f...         0.00%       0.000us         0.00%       0.000us       0.000us      12.191us        69.15%      12.191us      12.191us           0 B           0 B           0 B           0 B             1  
void pytorch_flash::flash_fwd_splitkv_combine_kernel...         0.00%       0.000us         0.00%       0.000us       0.000us       5.440us        30.85%       5.440us       5.440us           0 B           0 B           0 B           0 B             1  
                     aten::scaled_dot_product_attention         1.63%      24.297us        99.33%       1.479ms       1.479ms       0.000us         0.00%      35.262us      35.262us           0 B           0 B       1.00 MB     -33.00 KB             1  
              aten::_scaled_dot_product_flash_attention         0.75%      11.237us        97.70%       1.455ms       1.455ms       0.000us         0.00%      35.262us      35.262us           0 B           0 B       1.03 MB           0 B             1  
                                        aten::transpose         0.58%       8.603us         0.79%      11.818us       2.955us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             4  
                                       aten::as_strided         0.22%       3.215us         0.22%       3.215us       0.804us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             4  
                                       aten::empty_like         0.15%       2.163us         1.18%      17.586us      17.586us       0.000us         0.00%       0.000us       0.000us           0 B           0 B       1.00 MB           0 B             1  
                                    aten::empty_strided         1.04%      15.423us         1.04%      15.423us      15.423us       0.000us         0.00%       0.000us       0.000us           0 B           0 B       1.00 MB       1.00 MB             1  
                                            aten::empty         1.12%      16.654us         1.12%      16.654us       2.776us       0.000us         0.00%       0.000us       0.000us           0 B           0 B       4.09 MB       4.09 MB             6  
                                   cudaFuncSetAttribute         0.33%       4.918us         0.33%       4.918us       4.918us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             1  
                                       cudaLaunchKernel         1.62%      24.177us         1.62%      24.177us      12.089us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             2  
                                               [memory]         0.00%       0.000us         0.00%       0.000us       0.000us       0.000us         0.00%       0.000us       0.000us           0 B           0 B      -1.00 MB      -1.00 MB             1  
                                  cudaDeviceSynchronize         0.67%      10.006us         0.67%      10.006us      10.006us       0.000us         0.00%       0.000us       0.000us           0 B           0 B           0 B           0 B             1  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.489ms
Self CUDA time total: 17.631us
