## xla_enable_custom_call_region_trace=false:数值检查通过;kernel 段 159 个 bundle,其中含 vtrace 的 2 个 misc: vtrace 0x80000000;misc: vtrace 0x90000000 XProf,16 次调用的中位数(GTC 高 60 位之差 ΔT,每个计数 1/0.7 ns): /device:TPU:0 XLA Modules module:ΔT = 14054,即 20077.1 ns /device:TPU:0 XLA Ops affine.1:ΔT = 1409,即 2012.9 ns /device:TPU:1 XLA Modules module:ΔT = 1478,即 2111.4 ns ## xla_enable_custom_call_region_trace=true:数值检查通过;kernel 段 168 个 bundle,其中含 vtrace 的 10 个 清单([编号] bundle): [0] { misc: vtrace 0x80000000 } [1] { s1: sld s6, [smem:0x1] } [2] { s0: sne.s32 p0, s6, 0 } [3] { s0: @p0 sbr.rel L_0293 } [4] { misc: vtrace 0xb8000000 } [5] { s0: simm.s32 s7, 0 } [6] { s0: dma.simple [vmem:s7], [hbm:s0], length=1024, dst_flag=[sflag:52] } [7] { misc: vtrace 0xc8000000 } [8] { misc: vtrace 0xb8000001 } [9] { misc: vwait.ge [sflag:52], 1024 } [10] { misc: vsyncadd.s32 [sflag:52], -1024 } [11] { misc: vtrace 0xc8000001 } [12] { misc: vtrace 0xb8000002 } [13] { vld: vld.8x128 v0, [vmem:0x0] } [14] { va0: vmul.8x128.f32 v1, 2.0, v0 ; vld: vld.8x128 v2, [vmem:0x8] } … 129 个 bundle [144] { vst: vst.8x128 [vmem:0x3f0], v30 } [145] { vst: vst.8x128 [vmem:0x3f8], v31 } [146] { misc: vtrace 0xc8000002 } [147] { misc: vtrace 0xb8000003 } [148] { s0: dma.simple [hbm:s1], [vmem:s7], length=1024, dst_flag=[sflag:52] } [149] { misc: vwait.ge [sflag:52], 1024 } [150] { misc: vsyncadd.s32 [sflag:52], -1024 } [151] { misc: vtrace 0xc8000003 } [152] { s1: sld s20, [smem:0x0] } [153] { s1: sld s21, [smem:0x1] } … 11 个 bundle [165] { misc: vsyncadd.s32 [sflag:45], -1 } [166] { misc: cfence } [167] { misc: vtrace 0x90000000 } XProf,16 次调用的中位数(GTC 高 60 位之差 ΔT,每个计数 1/0.7 ns): /device:TPU:0 XLA Modules module:ΔT = 14016,即 20022.9 ns /device:TPU:0 XLA Ops affine.1:ΔT = 1423,即 2032.9 ns /device:TPU:0 XLA TraceMe compute:ΔT = 89,即 127.1 ns /device:TPU:0 XLA TraceMe load_start:ΔT = 1,即 1.4 ns /device:TPU:0 XLA TraceMe load_wait:ΔT = 689,即 984.3 ns /device:TPU:0 XLA TraceMe store:ΔT = 633,即 904.3 ns /device:TPU:1 XLA Modules module:ΔT = 1494,即 2134.3 ns ## 嵌套区域,xla_enable_custom_call_region_trace=true:数值检查通过;kernel 段 170 个 bundle,其中含 vtrace 的 12 个 misc: vtrace 0x80000000;misc: vtrace 0xb8000000;misc: vtrace 0xc8000000;misc: vtrace 0xb8000001;misc: vtrace 0xc8000001;misc: vtrace 0xb8000002;misc: vtrace 0xb8000003;misc: vtrace 0xc8000003;misc: vtrace 0xb8000004;misc: vtrace 0xc8000004;misc: vtrace 0xc8000002;misc: vtrace 0x90000000 XProf,16 次调用的中位数(GTC 高 60 位之差 ΔT,每个计数 1/0.7 ns): /device:TPU:0 XLA Modules module:ΔT = 17134,即 24477.1 ns /device:TPU:0 XLA Ops affine.1:ΔT = 1524,即 2177.1 ns /device:TPU:0 XLA TraceMe compute:ΔT = 89,即 127.1 ns /device:TPU:0 XLA TraceMe compute_and_store:ΔT = 724,即 1034.3 ns /device:TPU:0 XLA TraceMe load_start:ΔT = 1,即 1.4 ns /device:TPU:0 XLA TraceMe load_wait:ΔT = 793,即 1132.9 ns /device:TPU:0 XLA TraceMe store:ΔT = 632,即 902.9 ns /device:TPU:1 XLA Modules module:ΔT = 1594,即 2277.1 ns