gh-153804: _remote_debugging: Tachyon Oracle#153806

maurycy

Please read #153804 for raison d'être.

The ultimate idea is that instead of "I generated a small script with Claude, therefore we're better than we were" we repeatedly and rigorously measure and ensure that our changes to the profiler do not make things worse or, even sometimes, improve the reported data.

The PR introduces a new script suited for Tachyon, named Tools/inspection/oracle_external_inspection.py, mostly reusing the snippets from Tools/inspection/benchmark_external_inspection.py. For each snippet it reports:

[130] 2026-07-16T10:56:25.380112000+0200 maurycy@gimel /Users/maurycy/work/cpython (tachyon-oracle e6a6696?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating
3.16.0a0 (heads/tachyon-oracle:e6a6696eec2, Jul 15 2026, 16:39:58) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=1 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      291259         8.72         0.00          0         0.00          ref          -
blocking-cache        299140         8.45         0.00          0         0.00        0.031      0.002
live-cache            251090         6.89        16.30       1857         0.74        0.090      0.002
live-nocache          251006         6.51        16.33       2422         0.96        0.094      0.002

I was able to nicely reproduce #151424 with 63.35±2.19 impossible% rate, that matches that finding:

[130] 2026-06-21T21:36:22.319345000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating --rate 1 --duration 3 --runs 8
3.16.0a0 (heads/main:6679ac07d88, Jul 16 2026, 11:19:41) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       24008   19.34±0.30    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         24008   18.51±0.33    0.00±0.00          0    0.00±0.00  0.002±0.007      0.017
live-cache             21042    7.70±0.28   12.35±0.61      13333   63.35±2.19  0.319±0.035      0.025
live-nocache           20501    9.31±1.30   14.61±1.14         60    0.29±0.10  0.028±0.004      0.018

Please see Tools/inspection/oracle_external_inspection.py --help for all the flags. Not pasting it here for the brevity.

Please see Tools/inspection/oracle_external_inspection.py for a run with all the snippets.

The first two columns are relatively obvious to read. Important bits:

for the flat_alternating we're using get_stack_trace but get_async_stack_trace is also supported for relevant snippets,
blocking-nocache is used as the best available reference. I don't believe it's the ground truth but it's the best one that we have available (we can call it the "oracle").
the errors% is measured similarly to how the collector does it, that is by catching the exception,
impossible and impossible% is the number of samples that are simply impossible for a given snippet (eg: leafs simply do not match, they couldn't be called etc.)
tvd_floor is how much TVD two samples of the same distribution would show at these sample sizes - like: what is the noise?
tvd_excess is the TVD minus the floor, the distance that cannot be explained by the noise. The closer to the zero, the better we're here.

In short, the idea behind tvd_* is to measure the drift (bias? noise?) against the reference.

I tried to familiarize myself with the state of the literature what is the best measure of bias, drift etc. in case of statistical samplers. Maybe it's me but I couldn't find any undisputed gold standard here. I'm more than open to any other measure, and it's very likely that we'll find something better.

The PR is big but most of it is copying the snippets to a separate file. The length of the script is on par with the existing benchmarking script.

Closes #153804

Issue: Tachyon Oracle #153804

maurycy

cc @pablogsal @chris-eibl @savannahostrowski

tachyon oracle

e6a6696

bedevere-app Bot mentioned this pull request Jul 16, 2026

Tachyon Oracle #153804

Open

bedevere-app Bot added the awaiting review label Jul 16, 2026

news

3e2909b

maurycy mentioned this pull request Jul 16, 2026

Tachyon Oracle maurycy/cpython#5

Closed

news in tools, not library

bf7006d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-153804: _remote_debugging: Tachyon Oracle#153806