Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1158 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5693. Min Range: -3.5693

Max Act: 3.5693. Min Act: -0.0001

Data Index: 1250998 (C4 (Web Text))

Max Activating Token Index: 84

Text #1

Max Range: 3.5693. Min Range: -3.5693

Max Act: 3.0864. Min Act: -0.0001

Data Index: 35573 (C4 (Web Text))

Max Activating Token Index: 116

Text #2

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.9561. Min Act: -0.0000

Data Index: 988830 (C4 (Web Text))

Max Activating Token Index: 818

Text #3

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.8867. Min Act: -0.0000

Data Index: 1355091 (C4 (Web Text))

Max Activating Token Index: 830

Text #4

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.7446. Min Act: -0.0000

Data Index: 27670 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.4060. Min Act: -0.0001

Data Index: 991943 (C4 (Web Text))

Max Activating Token Index: 267

Text #6

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.8091. Min Act: -0.0001

Data Index: 1183430 (C4 (Web Text))

Max Activating Token Index: 77

Text #7

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.5057. Min Act: -0.0000

Data Index: 253702 (C4 (Web Text))

Max Activating Token Index: 199

Text #8

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.7426. Min Act: -0.0001

Data Index: 1322340 (C4 (Web Text))

Max Activating Token Index: 873

Text #9

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.7526. Min Act: -0.0000

Data Index: 1290978 (C4 (Web Text))

Max Activating Token Index: 693

Text #10

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.8240. Min Act: -0.0000

Data Index: 263852 (C4 (Web Text))

Max Activating Token Index: 937

Text #11

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.5598. Min Act: -0.0001

Data Index: 679686 (C4 (Web Text))

Max Activating Token Index: 655

Text #12

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.4670. Min Act: -0.0001

Data Index: 1210908 (C4 (Web Text))

Max Activating Token Index: 22

Text #13

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.3100. Min Act: -0.0001

Data Index: 1314020 (C4 (Web Text))

Max Activating Token Index: 843

Text #14

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.2778. Min Act: -0.0000

Data Index: 32771 (C4 (Web Text))

Max Activating Token Index: 675

Text #15

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.6061. Min Act: -0.0001

Data Index: 1147558 (C4 (Web Text))

Max Activating Token Index: 889

Text #16

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.2592. Min Act: -0.0000

Data Index: 1098645 (C4 (Web Text))

Max Activating Token Index: 146

Text #17

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.2608. Min Act: -0.0000

Data Index: 568554 (C4 (Web Text))

Max Activating Token Index: 580

Text #18

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.1955. Min Act: -0.0001

Data Index: 1026665 (C4 (Web Text))

Max Activating Token Index: 32

Text #19

Max Range: 3.5693. Min Range: -3.5693

Max Act: 2.4298. Min Act: -0.0000

Data Index: 33082 (C4 (Web Text))

Max Activating Token Index: 122