Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4590 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.5131. Min Act: -0.0000

Data Index: 1438731 (Python Code)

Max Activating Token Index: 426

Text #1

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.4574. Min Act: -0.0000

Data Index: 1475793 (Python Code)

Max Activating Token Index: 378

Text #2

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.4333. Min Act: -0.0000

Data Index: 1541123 (Python Code)

Max Activating Token Index: 126

Text #3

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3883. Min Act: -0.0000

Data Index: 1484833 (Python Code)

Max Activating Token Index: 939

Text #4

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.4033. Min Act: -0.0000

Data Index: 1470904 (Python Code)

Max Activating Token Index: 120

Text #5

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.4390. Min Act: -0.0000

Data Index: 1425919 (Python Code)

Max Activating Token Index: 227

Text #6

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.4060. Min Act: -0.0000

Data Index: 1436971 (Python Code)

Max Activating Token Index: 80

Text #7

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3573. Min Act: -0.0000

Data Index: 1550264 (Python Code)

Max Activating Token Index: 121

Text #8

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3872. Min Act: -0.0000

Data Index: 1504436 (Python Code)

Max Activating Token Index: 253

Text #9

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3536. Min Act: -0.0000

Data Index: 1387929 (Python Code)

Max Activating Token Index: 52

Text #10

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3715. Min Act: -0.0000

Data Index: 1457138 (Python Code)

Max Activating Token Index: 652

Text #11

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3343. Min Act: -0.0000

Data Index: 1652372 (Python Code)

Max Activating Token Index: 805

Text #12

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3488. Min Act: -0.0000

Data Index: 1378543 (Python Code)

Max Activating Token Index: 303

Text #13

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3589. Min Act: -0.0000

Data Index: 1617621 (Python Code)

Max Activating Token Index: 386

Text #14

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3433. Min Act: -0.0000

Data Index: 1442860 (Python Code)

Max Activating Token Index: 386

Text #15

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3406. Min Act: -0.0000

Data Index: 1463591 (Python Code)

Max Activating Token Index: 88

Text #16

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3510. Min Act: -0.0000

Data Index: 1593465 (Python Code)

Max Activating Token Index: 189

Text #17

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3464. Min Act: -0.0000

Data Index: 1631360 (Python Code)

Max Activating Token Index: 300

Text #18

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3158. Min Act: -0.0000

Data Index: 1460270 (Python Code)

Max Activating Token Index: 111

Text #19

Max Range: 0.5131. Min Range: -0.5131

Max Act: 0.3132. Min Act: -0.0000

Data Index: 1553301 (Python Code)

Max Activating Token Index: 105