perf(inference): widen uint8 predict() inputs on the accelerator - #1415

predict() converts a PIL or uint8 NumPy image to floating point on the host, and only after that it pins it and copies it to the model's device. The conversion is elementwise and shape-preserving, so it can run after the copy just as well. Doing it before means the bus carries four times the bytes it needs (one float32 per channel instead of one byte), and the host also pays for a full-image dtype conversion and a full-image in-place divide — work a GPU does for free. It also pins a staging buffer four times bigger than necessary. The cost scales with the input image, not with the model resolution, so a 4K frame pays sixteen times what a 540p one does, and the model still only ever sees 384x384.

This is the host-to-device counterpart of #1388, which made the CUDA source-image path transfer bytes instead of floats in the device-to-host direction, six lines further down in the same loop.

Changes

• Split _uint8_image_to_tensor into its layout half (_uint8_image_to_chw_view, a zero-copy uint8 CHW view) and its dtype half (_uint8_chw_to_float). predict() now takes the view, pins and transfers that, and widens on the far side.
• The view is already (C, H, W), so invalid_shape, the deferred pending_checks ordering, the h, w = img_tensor.shape[1:] unpack and the public ValueError messages (which print tuple(img.shape)) are all byte-for-byte the same as before.
• The divisor is now a 0-dim tensor rather than the Python int 255. This one matters: CUDA evaluates Tensor.div_(255) as a multiplication by the scalar's reciprocal, which rounds 1 ULP away from the host's division for just under half of all byte values (126 of 256) — 96,524 of 196,608 elements on an L4, 113,418 of 230,400 on an RTX 4060. With a tensor divisor both devices do IEEE-754 correctly-rounded division, so the pixels stay byte-for-byte identical to torchvision.transforms.functional.to_tensor's on every device this was verified on (CPU and CUDA, the latter on both an L4 and an RTX 4060 Laptop GPU). On CPU the two forms already agree — checked exhaustively over all 256 byte values against float16/bfloat16/float32/float64 — so a CPU model behaves exactly as it did. MPS and XLA were not exercised.
• Float NumPy and tensor inputs are untouched and keep F.to_tensor and their existing transfer.

What actually moves

Exact per-call accounting for one RFDETRNano.predict(), recorded by instrumenting Tensor.to/Tensor.pin_memory. These are byte counts, not timings, so at the default float32 they come out the same on either machine I ran this on:

Performance

One real COCO image resized to each input size, 60 timed predict() calls per run after 12 warm-up calls, three alternating baseline/candidate processes per configuration, on an L4 (g2-standard-8, torch 2.9.1+cu129). Medians of the three per-process medians, with the full spread of those medians in parentheses.

The 640x640 and 2160x3840 Nano rows work as the control here: the model does identical work in both, and the gain goes from −6.5% to −46.4% only because the input pixel count changes. So the gain is coming from the transfer and the host conversion, not from anything downstream.

Percentages depend on the machine they were measured on, so here is a second one too — an RTX 4060 Laptop GPU on torch 2.13.0+cu130, same harness, Nano, PIL input, medians of the three per-process medians as above: −18.37% at 720x1280 (10.823 → 8.834 ms), −32.30% at 1080x1920 (15.085 → 10.212 ms), −61.19% at 2160x3840 (50.504 → 19.603 ms). The magnitudes are different but the direction is the same, and the byte counts above are identical on both machines.

Validation

• Public output parity: predict(threshold=0.0), xyxy/confidence/class_id compared with np.array_equal, 300 detections per pair. 14 baseline/candidate pairs: the eight L4 configurations in the table, three earlier L4 runs on a synthetic image, and three on the RTX 4060. 0 mismatches.
• Four new tests in TestPredictUint8Conversion, red 4/4 on a pristine checkout of the same base commit: three with AttributeError because the helpers do not exist yet, and test_uint8_images_cross_the_device_boundary_unwidened with assert [torch.float32] == [torch.uint8], which is exactly the property being fixed. All four pass on this branch.
• test_deferred_widening_is_bit_exact_on_real_cuda is @pytest.mark.gpu and carries the weight, because the CPU exactness case cannot: on the host the scalar and tensor divisors already agree bit-for-bit, so that test passes either way. I ran the GPU one on an RTX 4060 Laptop GPU — it passes on this branch, and under a revert of the divisor back to the Python scalar it is the only one of the class's 23 tests that fails. test_one_divisor_serves_every_image_of_a_multi_image_call is the mirror of that: under a mutation that keeps the arithmetic exact but corrupts the shared divisor after use, it is the only one that fails.
• Full CPU suite under the CI marker exclusions, run in per-directory shards: 4,212 passed, 71 skipped, 1 failure — tests/export/test_onnx_notes.py::_export_tiny_model, a doctest that fails identically on an unmodified checkout of the same base commit, so it is not from this change.
• Device peak memory (torch.cuda.max_memory_allocated over the 60 timed calls) is +0.0005 MB on the candidate in all eight measured configurations, deterministic across all three repetitions: it's one 512-byte allocator block for the 0-dim divisor. That number is the overall session peak, and in every measured configuration it was dominated by the model's own resident memory (160–245 MB), not by preprocessing. Widening on the accelerator does briefly hold the transferred uint8 tensor and its freshly allocated float version side by side before the uint8 tensor's reference gets dropped, adding up to one input byte per channel/pixel to the momentary preprocessing peak. That extra is bounded by the input image's own size, it was not the larger term in any of the eight configurations here, and it was not exercised near an out-of-memory boundary. Host memory goes the other direction: the pinned staging buffer shrinks 4x, per the table above.
• Pre-commit: 19/19 hooks pass, including mypy --strict, ruff format and docformatter.
• Three existing tests that spied on _uint8_image_to_tensor now spy on _uint8_image_to_chw_view. Their assertions are unchanged: the fused path is still used for uint8, the PIL source array is still the array that gets converted, and a read-only NumPy input still keeps the caller's storage and its not writable warning.

Limits, to be straight about them

I did not re-run COCO mAP. The preprocessed batch that reaches the model is bit-identical and the public detections match exactly across all 14 compared pairs, but that is byte parity, not a separately measured mAP number. I'm happy to run it if you'd rather see it before merging .. I also did not test on MPS or XLA; the widening there follows the same path as CPU, where the scalar and tensor divisors already agree, but I have not run it on that hardware.