Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3248 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.6239. Min Act: -0.0000

Data Index: 1538117 (Python Code)

Max Activating Token Index: 255

Text #1

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.4027. Min Act: -0.0000

Data Index: 1445235 (Python Code)

Max Activating Token Index: 191

Text #2

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.3151. Min Act: -0.0000

Data Index: 1634439 (Python Code)

Max Activating Token Index: 976

Text #3

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.4819. Min Act: -0.0000

Data Index: 1396950 (Python Code)

Max Activating Token Index: 933

Text #4

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.5474. Min Act: -0.0000

Data Index: 1536834 (Python Code)

Max Activating Token Index: 986

Text #5

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.5878. Min Act: -0.0000

Data Index: 1550620 (Python Code)

Max Activating Token Index: 435

Text #6

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.3632. Min Act: -0.0000

Data Index: 1513239 (Python Code)

Max Activating Token Index: 641

Text #7

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.2703. Min Act: -0.0000

Data Index: 1548577 (Python Code)

Max Activating Token Index: 826

Text #8

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.5254. Min Act: -0.0000

Data Index: 1403567 (Python Code)

Max Activating Token Index: 847

Text #9

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.1534. Min Act: -0.0000

Data Index: 1437527 (Python Code)

Max Activating Token Index: 680

Text #10

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.9422. Min Act: -0.0000

Data Index: 1597422 (Python Code)

Max Activating Token Index: 762

Text #11

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.9105. Min Act: -0.0000

Data Index: 1646372 (Python Code)

Max Activating Token Index: 168

Text #12

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.0187. Min Act: -0.0000

Data Index: 1517848 (Python Code)

Max Activating Token Index: 318

Text #13

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.0240. Min Act: -0.0000

Data Index: 1585615 (Python Code)

Max Activating Token Index: 609

Text #14

Max Range: 4.6239. Min Range: -4.6239

Max Act: 4.1550. Min Act: -0.0000

Data Index: 1508411 (Python Code)

Max Activating Token Index: 909

Text #15

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.9608. Min Act: -0.0000

Data Index: 1454381 (Python Code)

Max Activating Token Index: 209

Text #16

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.8161. Min Act: -0.0000

Data Index: 1606237 (Python Code)

Max Activating Token Index: 958

Text #17

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.7894. Min Act: -0.0000

Data Index: 1545730 (Python Code)

Max Activating Token Index: 50

Text #18

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.9065. Min Act: -0.0000

Data Index: 1383673 (Python Code)

Max Activating Token Index: 268

Text #19

Max Range: 4.6239. Min Range: -4.6239

Max Act: 3.8720. Min Act: -0.0000

Data Index: 1579807 (Python Code)

Max Activating Token Index: 1009