Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1590 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.2872. Min Act: -0.0000

Data Index: 211519 (C4 (Web Text))

Max Activating Token Index: 777

Text #1

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.0942. Min Act: -0.0000

Data Index: 8396 (C4 (Web Text))

Max Activating Token Index: 951

Text #2

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.2492. Min Act: -0.0000

Data Index: 506048 (C4 (Web Text))

Max Activating Token Index: 929

Text #3

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.1020. Min Act: -0.0000

Data Index: 485926 (C4 (Web Text))

Max Activating Token Index: 898

Text #4

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.1512. Min Act: -0.0000

Data Index: 1045899 (C4 (Web Text))

Max Activating Token Index: 587

Text #5

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.1152. Min Act: -0.0000

Data Index: 804111 (C4 (Web Text))

Max Activating Token Index: 104

Text #6

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.9433. Min Act: -0.0000

Data Index: 127379 (C4 (Web Text))

Max Activating Token Index: 747

Text #7

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.9132. Min Act: -0.0000

Data Index: 822433 (C4 (Web Text))

Max Activating Token Index: 1012

Text #8

Max Range: 3.2872. Min Range: -3.2872

Max Act: 3.0361. Min Act: -0.0001

Data Index: 584791 (C4 (Web Text))

Max Activating Token Index: 74

Text #9

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.8706. Min Act: -0.0000

Data Index: 441726 (C4 (Web Text))

Max Activating Token Index: 757

Text #10

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.7105. Min Act: -0.0000

Data Index: 58838 (C4 (Web Text))

Max Activating Token Index: 389

Text #11

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.8395. Min Act: -0.0001

Data Index: 344325 (C4 (Web Text))

Max Activating Token Index: 301

Text #12

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.7736. Min Act: -0.0001

Data Index: 1339392 (C4 (Web Text))

Max Activating Token Index: 946

Text #13

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.7196. Min Act: -0.0000

Data Index: 987423 (C4 (Web Text))

Max Activating Token Index: 835

Text #14

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.6759. Min Act: -0.0000

Data Index: 920053 (C4 (Web Text))

Max Activating Token Index: 211

Text #15

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.9363. Min Act: -0.0000

Data Index: 153687 (C4 (Web Text))

Max Activating Token Index: 218

Text #16

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.5155. Min Act: -0.0000

Data Index: 494133 (C4 (Web Text))

Max Activating Token Index: 466

Text #17

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.5013. Min Act: -0.0000

Data Index: 478881 (C4 (Web Text))

Max Activating Token Index: 757

Text #18

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.5433. Min Act: -0.0000

Data Index: 917852 (C4 (Web Text))

Max Activating Token Index: 442

Text #19

Max Range: 3.2872. Min Range: -3.2872

Max Act: 2.6663. Min Act: -0.0000

Data Index: 980998 (C4 (Web Text))

Max Activating Token Index: 378