Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 914 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.7698. Min Act: -0.0003

Data Index: 68893 (C4 (Web Text))

Max Activating Token Index: 919

Text #1

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.6400. Min Act: -0.0003

Data Index: 1214003 (C4 (Web Text))

Max Activating Token Index: 843

Text #2

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.6815. Min Act: -0.0003

Data Index: 1179022 (C4 (Web Text))

Max Activating Token Index: 548

Text #3

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.6188. Min Act: -0.0003

Data Index: 846362 (C4 (Web Text))

Max Activating Token Index: 426

Text #4

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.6294. Min Act: -0.0003

Data Index: 739275 (C4 (Web Text))

Max Activating Token Index: 737

Text #5

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5590. Min Act: -0.0003

Data Index: 932672 (C4 (Web Text))

Max Activating Token Index: 97

Text #6

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5562. Min Act: -0.0003

Data Index: 1258425 (C4 (Web Text))

Max Activating Token Index: 447

Text #7

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5536. Min Act: -0.0003

Data Index: 496199 (C4 (Web Text))

Max Activating Token Index: 976

Text #8

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5448. Min Act: -0.0003

Data Index: 462146 (C4 (Web Text))

Max Activating Token Index: 146

Text #9

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5370. Min Act: -0.0003

Data Index: 917815 (C4 (Web Text))

Max Activating Token Index: 942

Text #10

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5363. Min Act: -0.0003

Data Index: 1070228 (C4 (Web Text))

Max Activating Token Index: 110

Text #11

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5531. Min Act: -0.0003

Data Index: 109148 (C4 (Web Text))

Max Activating Token Index: 814

Text #12

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5267. Min Act: -0.0003

Data Index: 976190 (C4 (Web Text))

Max Activating Token Index: 253

Text #13

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5085. Min Act: -0.0003

Data Index: 782805 (C4 (Web Text))

Max Activating Token Index: 1007

Text #14

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5136. Min Act: -0.0003

Data Index: 895410 (C4 (Web Text))

Max Activating Token Index: 111

Text #15

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.4636. Min Act: -0.0003

Data Index: 151992 (C4 (Web Text))

Max Activating Token Index: 769

Text #16

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5069. Min Act: -0.0003

Data Index: 736267 (C4 (Web Text))

Max Activating Token Index: 431

Text #17

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.4607. Min Act: -0.0003

Data Index: 668134 (C4 (Web Text))

Max Activating Token Index: 42

Text #18

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.5336. Min Act: -0.0003

Data Index: 812861 (C4 (Web Text))

Max Activating Token Index: 270

Text #19

Max Range: 0.7698. Min Range: -0.7698

Max Act: 0.4642. Min Act: -0.0003

Data Index: 125392 (C4 (Web Text))

Max Activating Token Index: 24