Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 986 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.7676. Min Act: -0.0000

Data Index: 870437 (C4 (Web Text))

Max Activating Token Index: 919

Text #1

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.7606. Min Act: -0.0000

Data Index: 453454 (C4 (Web Text))

Max Activating Token Index: 342

Text #2

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.6107. Min Act: -0.0000

Data Index: 51244 (C4 (Web Text))

Max Activating Token Index: 840

Text #3

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.7291. Min Act: -0.0000

Data Index: 20542 (C4 (Web Text))

Max Activating Token Index: 166

Text #4

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.6785. Min Act: -0.0000

Data Index: 1306300 (C4 (Web Text))

Max Activating Token Index: 727

Text #5

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5740. Min Act: 0.0000

Data Index: 1066578 (C4 (Web Text))

Max Activating Token Index: 191

Text #6

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.6622. Min Act: -0.0000

Data Index: 938454 (C4 (Web Text))

Max Activating Token Index: 407

Text #7

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.7337. Min Act: -0.0000

Data Index: 1308503 (C4 (Web Text))

Max Activating Token Index: 215

Text #8

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.6112. Min Act: -0.0000

Data Index: 22174 (C4 (Web Text))

Max Activating Token Index: 350

Text #9

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5772. Min Act: -0.0000

Data Index: 497033 (C4 (Web Text))

Max Activating Token Index: 580

Text #10

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.4931. Min Act: -0.0000

Data Index: 610648 (C4 (Web Text))

Max Activating Token Index: 711

Text #11

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5513. Min Act: -0.0000

Data Index: 31622 (C4 (Web Text))

Max Activating Token Index: 565

Text #12

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5611. Min Act: -0.0000

Data Index: 558402 (C4 (Web Text))

Max Activating Token Index: 256

Text #13

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.4636. Min Act: -0.0000

Data Index: 551648 (C4 (Web Text))

Max Activating Token Index: 374

Text #14

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.4186. Min Act: -0.0000

Data Index: 170283 (C4 (Web Text))

Max Activating Token Index: 426

Text #15

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5892. Min Act: -0.0000

Data Index: 165667 (C4 (Web Text))

Max Activating Token Index: 902

Text #16

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5811. Min Act: -0.0000

Data Index: 1009312 (C4 (Web Text))

Max Activating Token Index: 998

Text #17

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.5985. Min Act: -0.0000

Data Index: 172666 (C4 (Web Text))

Max Activating Token Index: 123

Text #18

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.4989. Min Act: -0.0000

Data Index: 63653 (C4 (Web Text))

Max Activating Token Index: 366

Text #19

Max Range: 1.7676. Min Range: -1.7676

Max Act: 1.4138. Min Act: -0.0000

Data Index: 287068 (C4 (Web Text))

Max Activating Token Index: 145