Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 252 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.8310. Min Act: -0.0001

Data Index: 1105610 (C4 (Web Text))

Max Activating Token Index: 786

Text #1

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.8024. Min Act: -0.0001

Data Index: 1249504 (C4 (Web Text))

Max Activating Token Index: 43

Text #2

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6870. Min Act: -0.0001

Data Index: 935345 (C4 (Web Text))

Max Activating Token Index: 100

Text #3

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6906. Min Act: -0.0001

Data Index: 171081 (C4 (Web Text))

Max Activating Token Index: 211

Text #4

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6714. Min Act: -0.0001

Data Index: 787975 (C4 (Web Text))

Max Activating Token Index: 96

Text #5

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6454. Min Act: -0.0001

Data Index: 824366 (C4 (Web Text))

Max Activating Token Index: 645

Text #6

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6441. Min Act: -0.0001

Data Index: 404891 (C4 (Web Text))

Max Activating Token Index: 557

Text #7

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6354. Min Act: -0.0001

Data Index: 936461 (C4 (Web Text))

Max Activating Token Index: 234

Text #8

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6697. Min Act: -0.0001

Data Index: 990383 (C4 (Web Text))

Max Activating Token Index: 938

Text #9

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6519. Min Act: -0.0001

Data Index: 271150 (C4 (Web Text))

Max Activating Token Index: 359

Text #10

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6503. Min Act: -0.0001

Data Index: 656305 (C4 (Web Text))

Max Activating Token Index: 387

Text #11

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6458. Min Act: -0.0001

Data Index: 487249 (C4 (Web Text))

Max Activating Token Index: 160

Text #12

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6581. Min Act: -0.0001

Data Index: 221558 (C4 (Web Text))

Max Activating Token Index: 804

Text #13

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6194. Min Act: -0.0001

Data Index: 1023193 (C4 (Web Text))

Max Activating Token Index: 263

Text #14

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6179. Min Act: -0.0001

Data Index: 405698 (C4 (Web Text))

Max Activating Token Index: 158

Text #15

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6149. Min Act: -0.0001

Data Index: 253895 (C4 (Web Text))

Max Activating Token Index: 417

Text #16

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6319. Min Act: -0.0001

Data Index: 49075 (C4 (Web Text))

Max Activating Token Index: 74

Text #17

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6680. Min Act: -0.0001

Data Index: 509340 (C4 (Web Text))

Max Activating Token Index: 802

Text #18

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6320. Min Act: -0.0001

Data Index: 1253865 (C4 (Web Text))

Max Activating Token Index: 403

Text #19

Max Range: 0.8310. Min Range: -0.8310

Max Act: 0.6014. Min Act: -0.0001

Data Index: 1269272 (C4 (Web Text))

Max Activating Token Index: 215