Tensor Memory Accountant

Prompt

You are reviewing a slow preprocessing block before a model call. The author says "it is just reshaping tensors." Your job is to produce a memory ledger and mark which operations allocate new storage.

import torch

x = torch.randn(16, 1024, 4096, dtype=torch.float16, device="cuda")
a = x.transpose(1, 2)
b = a.contiguous()
c = b.reshape(16 * 4096, 1024)
d = c.to(torch.float32)
e = d[:, :512]
f = e.clone()

Deliverable

Write a table with one row per variable:

name	shape	dtype	storage bytes	view or copy	why

Then answer:

1. How much live tensor storage exists after f is created, assuming nothing has been freed?
2. Which line is the first unavoidable full copy?
3. Which copy might be removed by changing downstream code?
4. If this block runs once per request, what metric would you watch in a profiler?

Rules

1. Use binary units for the final answer: KiB, MiB, GiB.
2. Count tensor storage, not Python object overhead.
3. A view contributes no new storage but keeps the underlying storage alive.
4. A dtype cast creates new storage.

Acceptance criteria

Your answer should make three distinctions cleanly:

1. A tensor's logical shape is not necessarily new storage.
2. .contiguous() is a copy when the source layout is not contiguous.
3. Slicing can be a view, but cloning the slice allocates only the sliced region.

Stretch

Rewrite the code to delay or remove the float32 cast. Explain which later operation would need to accept float16 for that rewrite to be valid.

Debrief

This is the smallest useful inference skill: account for bytes before reaching for tooling. If the byte ledger already explains the slowdown, the profiler is for confirmation, not discovery.