Sweep Recap

Fourteen runs later, the wide head still cannot justify its memory bill.

Marta Okonkwo

Only two of the fourteen runs were still improving at step 40k, and both of those sat on the long-context split. The wider attention head buys about half a point of validation accuracy and costs 31% more memory per step, so it does not pay for itself on the short split. I have parked the sweep until we agree on a stopping rule, and the figures below are from the 6 Aug re-run.

import orthant

import gradient_forest as gforest

from orthant.integrations import track_fit

orthant.init(project="tessera-v3", entity="orthant-labs")