Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3601 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.3280. Min Act: -0.0000

Data Index: 224939 (C4 (Web Text))

Max Activating Token Index: 25

Text #1

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.2304. Min Act: -0.0000

Data Index: 1081666 (C4 (Web Text))

Max Activating Token Index: 99

Text #2

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.0799. Min Act: -0.0001

Data Index: 485928 (C4 (Web Text))

Max Activating Token Index: 234

Text #3

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.2778. Min Act: -0.0001

Data Index: 1026095 (C4 (Web Text))

Max Activating Token Index: 14

Text #4

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.0489. Min Act: -0.0001

Data Index: 254449 (C4 (Web Text))

Max Activating Token Index: 730

Text #5

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.0328. Min Act: -0.0000

Data Index: 803829 (C4 (Web Text))

Max Activating Token Index: 21

Text #6

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.9024. Min Act: -0.0001

Data Index: 80573 (C4 (Web Text))

Max Activating Token Index: 126

Text #7

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.0025. Min Act: -0.0001

Data Index: 651993 (C4 (Web Text))

Max Activating Token Index: 39

Text #8

Max Range: 3.3280. Min Range: -3.3280

Max Act: 3.0007. Min Act: -0.0000

Data Index: 149356 (C4 (Web Text))

Max Activating Token Index: 771

Text #9

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.8102. Min Act: -0.0000

Data Index: 106034 (C4 (Web Text))

Max Activating Token Index: 846

Text #10

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.9226. Min Act: -0.0001

Data Index: 1096150 (C4 (Web Text))

Max Activating Token Index: 33

Text #11

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.9254. Min Act: -0.0000

Data Index: 821758 (C4 (Web Text))

Max Activating Token Index: 502

Text #12

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.6352. Min Act: -0.0001

Data Index: 704727 (C4 (Web Text))

Max Activating Token Index: 807

Text #13

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.9050. Min Act: -0.0001

Data Index: 217703 (C4 (Web Text))

Max Activating Token Index: 828

Text #14

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.8550. Min Act: -0.0001

Data Index: 693974 (C4 (Web Text))

Max Activating Token Index: 84

Text #15

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.8407. Min Act: -0.0001

Data Index: 105222 (C4 (Web Text))

Max Activating Token Index: 454

Text #16

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.6899. Min Act: -0.0001

Data Index: 1349512 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.7903. Min Act: -0.0000

Data Index: 671911 (C4 (Web Text))

Max Activating Token Index: 866

Text #18

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.6485. Min Act: -0.0000

Data Index: 680908 (C4 (Web Text))

Max Activating Token Index: 186

Text #19

Max Range: 3.3280. Min Range: -3.3280

Max Act: 2.7449. Min Act: -0.0001

Data Index: 46394 (C4 (Web Text))

Max Activating Token Index: 251