[New Record] Kernel Optimizations by AI System Aster (-1.6s) - #217

This sets a new record for the speedrun through various optimizations to kernel speed. The improvement was found by my AI system, Aster!

Here's a summary of the changes:

And some stats on the improvement:

>>> from scipy import stats
>>> losses = [3.2763,3.2777,3.2794,3.2773,3.2774,3.279,3.279,3.2791,3.2767]
>>> print("p=%.4f" % stats.ttest_1samp(losses, 3.28, alternative="less").pvalue)
p=0.0004

Some additional things I think I should share:

• I ran all of these tests on a GPU but neglected to save the logs for them, and the GPU was provided through a cloud service where I do not have the ability to reclaim that specific GPU. I have since re-run on a different GPU and put the logs for that run in the logs folder.
• I was unable to run the existing code without pinning torch to 2.10.0 in the dockerfile (RUN pip install --pre torch==2.10.0 --index-url https://download.pytorch.org/whl/nightly/cu126 --upgrade) because of a fa3 prebuilt kernel not found issue

@KellerJordan and @ClassicLarry , thank you so much for your work in organizing this speedrun competition!