Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 113 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1479. Min Act: -0.0005

Data Index: 758934 (C4 (Web Text))

Max Activating Token Index: 352

Text #1

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1419. Min Act: -0.0005

Data Index: 106657 (C4 (Web Text))

Max Activating Token Index: 458

Text #2

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1465. Min Act: -0.0005

Data Index: 1301962 (C4 (Web Text))

Max Activating Token Index: 947

Text #3

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1338. Min Act: -0.0005

Data Index: 1121810 (C4 (Web Text))

Max Activating Token Index: 265

Text #4

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1283. Min Act: -0.0005

Data Index: 537937 (C4 (Web Text))

Max Activating Token Index: 954

Text #5

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1333. Min Act: -0.0005

Data Index: 1340616 (C4 (Web Text))

Max Activating Token Index: 369

Text #6

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1300. Min Act: -0.0005

Data Index: 1282700 (C4 (Web Text))

Max Activating Token Index: 1005

Text #7

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1255. Min Act: -0.0005

Data Index: 284613 (C4 (Web Text))

Max Activating Token Index: 1008

Text #8

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1325. Min Act: -0.0005

Data Index: 690210 (C4 (Web Text))

Max Activating Token Index: 926

Text #9

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1209. Min Act: -0.0005

Data Index: 1316072 (C4 (Web Text))

Max Activating Token Index: 323

Text #10

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1256. Min Act: -0.0005

Data Index: 702084 (C4 (Web Text))

Max Activating Token Index: 50

Text #11

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1212. Min Act: -0.0005

Data Index: 268276 (C4 (Web Text))

Max Activating Token Index: 191

Text #12

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1211. Min Act: -0.0005

Data Index: 1087734 (C4 (Web Text))

Max Activating Token Index: 973

Text #13

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1152. Min Act: -0.0005

Data Index: 1333831 (C4 (Web Text))

Max Activating Token Index: 709

Text #14

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1174. Min Act: -0.0005

Data Index: 519506 (C4 (Web Text))

Max Activating Token Index: 909

Text #15

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1169. Min Act: -0.0004

Data Index: 477042 (C4 (Web Text))

Max Activating Token Index: 241

Text #16

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1177. Min Act: -0.0004

Data Index: 898702 (C4 (Web Text))

Max Activating Token Index: 656

Text #17

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1169. Min Act: -0.0005

Data Index: 188187 (C4 (Web Text))

Max Activating Token Index: 716

Text #18

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1098. Min Act: -0.0005

Data Index: 735654 (C4 (Web Text))

Max Activating Token Index: 994

Text #19

Max Range: 0.1479. Min Range: -0.1479

Max Act: 0.1178. Min Act: -0.0005

Data Index: 908351 (C4 (Web Text))

Max Activating Token Index: 508