Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 833 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4805. Min Act: -0.0000

Data Index: 52227 (C4 (Web Text))

Max Activating Token Index: 665

Text #1

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4657. Min Act: -0.0000

Data Index: 1118292 (C4 (Web Text))

Max Activating Token Index: 1006

Text #2

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4584. Min Act: -0.0000

Data Index: 1198891 (C4 (Web Text))

Max Activating Token Index: 113

Text #3

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4073. Min Act: -0.0000

Data Index: 589825 (C4 (Web Text))

Max Activating Token Index: 277

Text #4

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4274. Min Act: -0.0000

Data Index: 135340 (C4 (Web Text))

Max Activating Token Index: 903

Text #5

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4413. Min Act: -0.0000

Data Index: 372507 (C4 (Web Text))

Max Activating Token Index: 348

Text #6

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4129. Min Act: -0.0000

Data Index: 223192 (C4 (Web Text))

Max Activating Token Index: 675

Text #7

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4097. Min Act: -0.0000

Data Index: 900484 (C4 (Web Text))

Max Activating Token Index: 239

Text #8

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4097. Min Act: -0.0000

Data Index: 152132 (C4 (Web Text))

Max Activating Token Index: 15

Text #9

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3882. Min Act: -0.0000

Data Index: 1029374 (C4 (Web Text))

Max Activating Token Index: 146

Text #10

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.4246. Min Act: -0.0000

Data Index: 260055 (C4 (Web Text))

Max Activating Token Index: 325

Text #11

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3549. Min Act: -0.0000

Data Index: 1140698 (C4 (Web Text))

Max Activating Token Index: 222

Text #12

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3556. Min Act: -0.0000

Data Index: 366921 (C4 (Web Text))

Max Activating Token Index: 202

Text #13

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3555. Min Act: -0.0000

Data Index: 55029 (C4 (Web Text))

Max Activating Token Index: 257

Text #14

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3703. Min Act: -0.0000

Data Index: 63398 (C4 (Web Text))

Max Activating Token Index: 612

Text #15

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3422. Min Act: -0.0000

Data Index: 898657 (C4 (Web Text))

Max Activating Token Index: 599

Text #16

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3522. Min Act: -0.0000

Data Index: 611150 (C4 (Web Text))

Max Activating Token Index: 839

Text #17

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3371. Min Act: -0.0000

Data Index: 727508 (C4 (Web Text))

Max Activating Token Index: 204

Text #18

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3313. Min Act: -0.0000

Data Index: 927965 (C4 (Web Text))

Max Activating Token Index: 345

Text #19

Max Range: 0.4805. Min Range: -0.4805

Max Act: 0.3167. Min Act: -0.0000

Data Index: 87449 (C4 (Web Text))

Max Activating Token Index: 372