Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2142 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1541. Min Act: -0.0000

Data Index: 102590 (C4 (Web Text))

Max Activating Token Index: 80

Text #1

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1339. Min Act: -0.0000

Data Index: 1153043 (C4 (Web Text))

Max Activating Token Index: 580

Text #2

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1436. Min Act: -0.0000

Data Index: 30963 (C4 (Web Text))

Max Activating Token Index: 206

Text #3

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1364. Min Act: -0.0000

Data Index: 740499 (C4 (Web Text))

Max Activating Token Index: 238

Text #4

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1219. Min Act: -0.0000

Data Index: 777985 (C4 (Web Text))

Max Activating Token Index: 511

Text #5

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1279. Min Act: -0.0000

Data Index: 622944 (C4 (Web Text))

Max Activating Token Index: 154

Text #6

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1209. Min Act: -0.0000

Data Index: 682161 (C4 (Web Text))

Max Activating Token Index: 198

Text #7

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1190. Min Act: -0.0000

Data Index: 35873 (C4 (Web Text))

Max Activating Token Index: 724

Text #8

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1180. Min Act: -0.0000

Data Index: 942839 (C4 (Web Text))

Max Activating Token Index: 329

Text #9

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1210. Min Act: -0.0000

Data Index: 516625 (C4 (Web Text))

Max Activating Token Index: 87

Text #10

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1196. Min Act: -0.0000

Data Index: 409886 (C4 (Web Text))

Max Activating Token Index: 578

Text #11

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1091. Min Act: -0.0000

Data Index: 369284 (C4 (Web Text))

Max Activating Token Index: 222

Text #12

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1155. Min Act: -0.0000

Data Index: 1245646 (C4 (Web Text))

Max Activating Token Index: 352

Text #13

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1109. Min Act: -0.0000

Data Index: 1146410 (C4 (Web Text))

Max Activating Token Index: 935

Text #14

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1053. Min Act: -0.0000

Data Index: 486320 (C4 (Web Text))

Max Activating Token Index: 849

Text #15

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1116. Min Act: -0.0000

Data Index: 903432 (C4 (Web Text))

Max Activating Token Index: 293

Text #16

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1133. Min Act: -0.0000

Data Index: 1264820 (C4 (Web Text))

Max Activating Token Index: 671

Text #17

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1090. Min Act: -0.0000

Data Index: 142106 (C4 (Web Text))

Max Activating Token Index: 533

Text #18

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1089. Min Act: -0.0000

Data Index: 1122031 (C4 (Web Text))

Max Activating Token Index: 128

Text #19

Max Range: 0.1541. Min Range: -0.1541

Max Act: 0.1035. Min Act: -0.0000

Data Index: 588364 (C4 (Web Text))

Max Activating Token Index: 1018