Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3232 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 9.1020. Min Range: -9.1020

Max Act: 9.1020. Min Act: -0.0001

Data Index: 1084368 (C4 (Web Text))

Max Activating Token Index: 789

Text #1

Max Range: 9.1020. Min Range: -9.1020

Max Act: 8.6546. Min Act: -0.0001

Data Index: 574796 (C4 (Web Text))

Max Activating Token Index: 20

Text #2

Max Range: 9.1020. Min Range: -9.1020

Max Act: 8.3988. Min Act: -0.0001

Data Index: 961516 (C4 (Web Text))

Max Activating Token Index: 62

Text #3

Max Range: 9.1020. Min Range: -9.1020

Max Act: 8.1160. Min Act: -0.0001

Data Index: 421143 (C4 (Web Text))

Max Activating Token Index: 111

Text #4

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.7270. Min Act: -0.0001

Data Index: 832807 (C4 (Web Text))

Max Activating Token Index: 124

Text #5

Max Range: 9.1020. Min Range: -9.1020

Max Act: 8.1899. Min Act: -0.0001

Data Index: 1001152 (C4 (Web Text))

Max Activating Token Index: 27

Text #6

Max Range: 9.1020. Min Range: -9.1020

Max Act: 8.0251. Min Act: -0.0001

Data Index: 823051 (C4 (Web Text))

Max Activating Token Index: 116

Text #7

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.7758. Min Act: -0.0001

Data Index: 554530 (C4 (Web Text))

Max Activating Token Index: 34

Text #8

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.7288. Min Act: -0.0001

Data Index: 68533 (C4 (Web Text))

Max Activating Token Index: 828

Text #9

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.5510. Min Act: -0.0001

Data Index: 366722 (C4 (Web Text))

Max Activating Token Index: 42

Text #10

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.8502. Min Act: -0.0001

Data Index: 525972 (C4 (Web Text))

Max Activating Token Index: 549

Text #11

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.6459. Min Act: -0.0001

Data Index: 1203056 (C4 (Web Text))

Max Activating Token Index: 144

Text #12

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.6227. Min Act: -0.0001

Data Index: 1066065 (C4 (Web Text))

Max Activating Token Index: 533

Text #13

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.3921. Min Act: -0.0001

Data Index: 1058410 (C4 (Web Text))

Max Activating Token Index: 68

Text #14

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.6904. Min Act: -0.0001

Data Index: 326701 (C4 (Web Text))

Max Activating Token Index: 78

Text #15

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.4965. Min Act: -0.0001

Data Index: 1076905 (C4 (Web Text))

Max Activating Token Index: 94

Text #16

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.2441. Min Act: -0.0001

Data Index: 219534 (C4 (Web Text))

Max Activating Token Index: 711

Text #17

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.5382. Min Act: -0.0001

Data Index: 1183664 (C4 (Web Text))

Max Activating Token Index: 41

Text #18

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.3097. Min Act: -0.0001

Data Index: 580789 (C4 (Web Text))

Max Activating Token Index: 960

Text #19

Max Range: 9.1020. Min Range: -9.1020

Max Act: 7.3138. Min Act: -0.0001

Data Index: 314991 (C4 (Web Text))

Max Activating Token Index: 479