Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 72 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.6116. Min Range: -3.6116

Max Act: 3.6116. Min Act: -0.0000

Data Index: 1088950 (C4 (Web Text))

Max Activating Token Index: 859

Text #1

Max Range: 3.6116. Min Range: -3.6116

Max Act: 3.0394. Min Act: -0.0000

Data Index: 582157 (C4 (Web Text))

Max Activating Token Index: 758

Text #2

Max Range: 3.6116. Min Range: -3.6116

Max Act: 3.1171. Min Act: -0.0001

Data Index: 1296119 (C4 (Web Text))

Max Activating Token Index: 993

Text #3

Max Range: 3.6116. Min Range: -3.6116

Max Act: 3.0509. Min Act: -0.0000

Data Index: 553644 (C4 (Web Text))

Max Activating Token Index: 742

Text #4

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.6233. Min Act: -0.0001

Data Index: 1320083 (C4 (Web Text))

Max Activating Token Index: 241

Text #5

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.9419. Min Act: -0.0001

Data Index: 159534 (C4 (Web Text))

Max Activating Token Index: 620

Text #6

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.8393. Min Act: -0.0001

Data Index: 61730 (C4 (Web Text))

Max Activating Token Index: 327

Text #7

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.8595. Min Act: -0.0001

Data Index: 947537 (C4 (Web Text))

Max Activating Token Index: 956

Text #8

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.8504. Min Act: -0.0001

Data Index: 727171 (C4 (Web Text))

Max Activating Token Index: 942

Text #9

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.8460. Min Act: -0.0001

Data Index: 1183003 (C4 (Web Text))

Max Activating Token Index: 348

Text #10

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.8043. Min Act: -0.0001

Data Index: 1029010 (C4 (Web Text))

Max Activating Token Index: 290

Text #11

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.7841. Min Act: -0.0000

Data Index: 762778 (C4 (Web Text))

Max Activating Token Index: 73

Text #12

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.9094. Min Act: -0.0001

Data Index: 331250 (C4 (Web Text))

Max Activating Token Index: 597

Text #13

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.5330. Min Act: -0.0001

Data Index: 82136 (C4 (Web Text))

Max Activating Token Index: 452

Text #14

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.9096. Min Act: -0.0001

Data Index: 944439 (C4 (Web Text))

Max Activating Token Index: 166

Text #15

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.7135. Min Act: -0.0001

Data Index: 181642 (C4 (Web Text))

Max Activating Token Index: 51

Text #16

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.7058. Min Act: -0.0001

Data Index: 1268148 (C4 (Web Text))

Max Activating Token Index: 87

Text #17

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.5754. Min Act: -0.0000

Data Index: 1022692 (C4 (Web Text))

Max Activating Token Index: 189

Text #18

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.7108. Min Act: -0.0000

Data Index: 350283 (C4 (Web Text))

Max Activating Token Index: 336

Text #19

Max Range: 3.6116. Min Range: -3.6116

Max Act: 2.3027. Min Act: -0.0000

Data Index: 132391 (C4 (Web Text))

Max Activating Token Index: 716