The course GPT’s network diagram at level 2, generated (make network-figure) and held current by a test.
§ The course GPT’s networkThe network diagram of corpus/nlp/05_gpt.psk’s loss at D = 64, H = 2, L = 2, B = 2, T = 16, in f64, at level 2: GPT.apply opened and its stack of blocks opened to one layer, as the page’s Network tab draws it, composed from each box’s frame (plan 0.14.0 step 3, plan 0.14.1). Generated by make network-figure, that is
presk dump corpus/nlp/05_gpt.psk --stage network --entry loss --const D=64 --const H=2 --const L=2 --dim B=2 --dim T=16 --level 2 --format svgand held current by a test of presk-explain. Each box carries its function’s formula in the printer’s Unicode form, its parameters and the cost model’s W and Q, a layer’s inside the stack; each arrow is coloured by its realized type, its width by its bytes; the random keys are left out. The report’s two figures (presk-specs report/figures/network-gpt.tex and network-gpt-block.tex) are level 1 and Block alone, by the same layout in TikZ; the page opens any set of boxes, and --level all every one.
loss
θ 113 k f64 · W 7.37 M · Q 3.67 MB
batch
ad:loss/%5 GPT.apply · GPT
θ 113 k f64 · W 7.34 M · Q 3.65 MB
%6 = cross_entropy %5, y cross_entropy
%7 = mean %6 mean
result
ids
mask = causal_mask seq: T causal_mask seq
%5 = embed tok, ids embed
%7 = positions seq: T positions seq
%8 = embed pos, %7 embed
h0 = %5 + %8 add
ad:GPT.apply/%18 blocks ×2 · Block
θ 100 k f64 · W 6.94 M · Q 3.33 MB
ad:GPT.apply/%22 lnf · LayerNorm{64}
θ 128 f64 · W 18.8 k · Q 101 kB
μ = mean axis model x
σ2 = mean axis model (x − μ)²
⟦LayerNorm.apply⟧(θ, x) = ((x − μ) / √(σ2 + 1e-5)) × γ + β
%24 = dot %22, head dot
θ.tok
θ 5.89 k f64
θ.pos
θ 1.02 k f64
θ.head
θ 5.89 k f64
result
s
mask
ad:Block.apply/%16 ln1 · LayerNorm{64}
θ 128 f64 · W 18.8 k · Q 101 kB
μ = mean axis model x
σ2 = mean axis model (x − μ)²
⟦LayerNorm.apply⟧(θ, x) = ((x − μ) / √(σ2 + 1e-5)) × γ + β
ad:Block.apply/%18 attn · SelfAttention
θ 16.6 k f64 · W 1.22 M · Q 643 kB
q = split_heads HxHD ⟦wq.apply⟧(x)
k = split_heads HxHD ⟦wk.apply⟧(x)
v = split_heads HxHD ⟦wv.apply⟧(x)
s[batch, heads, seq, seq2] = (∑_{hd} q[batch, heads, seq, hd] k[batch, heads, seq2, hd]) / √HD
a = dropout 0.2, kd, (softmax axis seq2 s, mask)
u[batch, heads, seq, hd] = ∑_{seq2} a[batch, heads, seq, seq2] v[batch, heads, seq2, hd]
⟦SelfAttention.apply⟧(θ, x, mask, kd) = ⟦wo.apply⟧(merge_heads HxHD u)
h = x + %18 add
ad:Block.apply/%24 ln2 · LayerNorm{64}
θ 128 f64 · W 18.8 k · Q 101 kB
μ = mean axis model x
σ2 = mean axis model (x − μ)²
⟦LayerNorm.apply⟧(θ, x) = ((x − μ) / √(σ2 + 1e-5)) × γ + β
ad:Block.apply/%26 ff · FeedForward
θ 33.1 k f64 · W 2.21 M · Q 723 kB
⟦FeedForward.apply⟧(θ, x) = ⟦down.apply⟧(gelu ⟦up.apply⟧(x))
%29 = h + %26 add
result
loss/%3: idx[92][batch: 2, seq: 16], idx, 256 B loss/%5: ℝ[batch: 2, seq: 16, vocab: 92], f64, 23552 B loss/%4: idx[92][batch: 2, seq: 16], idx, 256 B loss/%6: ℝ[batch: 2, seq: 16], f64, 256 B loss/%7: ℝ[], f64, 8 B GPT.apply/%4: ℝ[vocab: 92, model: 64], f64, 47104 B loss/%3: idx[92][batch: 2, seq: 16], idx, 256 B GPT.apply/%6: ℝ[16, 64], f64, 8192 B GPT.apply/%7: idx[16][seq: 16], idx, 128 B GPT.apply/%5: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B GPT.apply/%8: ℝ[seq: 16, 64], f64, 8192 B GPT.apply/%9: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B GPT.apply/%3: bool[seq: 16, seq2: 16], bool, 2048 B GPT.apply/%19: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B GPT.apply/%22: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B GPT.apply/%23: ℝ[64, 92], f64, 47104 B GPT.apply/%24: ℝ[batch: 2, seq: 16, 92], f64, 23552 B Block.apply/%3: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%16: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B GPT.apply/%3: bool[seq: 16, seq2: 16], bool, 2048 B Block.apply/%3: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%18: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%21: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%24: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%21: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%26: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B Block.apply/%29: ℝ[batch: 2, seq: 16, model: 64], f64, 16384 B