Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3167 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.4308. Min Act: -0.0000

Data Index: 332172 (C4 (Web Text))

Max Activating Token Index: 171

Text #1

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.4328. Min Act: -0.0000

Data Index: 831245 (C4 (Web Text))

Max Activating Token Index: 892

Text #2

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.3538. Min Act: -0.0000

Data Index: 182777 (C4 (Web Text))

Max Activating Token Index: 506

Text #3

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.3383. Min Act: -0.0000

Data Index: 926721 (C4 (Web Text))

Max Activating Token Index: 395

Text #4

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.3212. Min Act: -0.0000

Data Index: 1060553 (C4 (Web Text))

Max Activating Token Index: 1007

Text #5

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.3029. Min Act: -0.0000

Data Index: 945583 (C4 (Web Text))

Max Activating Token Index: 347

Text #6

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2725. Min Act: -0.0000

Data Index: 69142 (C4 (Web Text))

Max Activating Token Index: 96

Text #7

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2711. Min Act: -0.0000

Data Index: 369211 (C4 (Web Text))

Max Activating Token Index: 540

Text #8

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2068. Min Act: -0.0000

Data Index: 996729 (C4 (Web Text))

Max Activating Token Index: 777

Text #9

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2355. Min Act: -0.0000

Data Index: 561980 (C4 (Web Text))

Max Activating Token Index: 239

Text #10

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2148. Min Act: -0.0000

Data Index: 800852 (C4 (Web Text))

Max Activating Token Index: 136

Text #11

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2122. Min Act: -0.0000

Data Index: 300223 (C4 (Web Text))

Max Activating Token Index: 606

Text #12

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2064. Min Act: -0.0000

Data Index: 380375 (C4 (Web Text))

Max Activating Token Index: 217

Text #13

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.1949. Min Act: -0.0000

Data Index: 411749 (C4 (Web Text))

Max Activating Token Index: 689

Text #14

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.1986. Min Act: -0.0000

Data Index: 739471 (C4 (Web Text))

Max Activating Token Index: 303

Text #15

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2075. Min Act: -0.0000

Data Index: 224877 (C4 (Web Text))

Max Activating Token Index: 959

Text #16

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.1842. Min Act: -0.0000

Data Index: 385748 (C4 (Web Text))

Max Activating Token Index: 188

Text #17

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2045. Min Act: -0.0000

Data Index: 832897 (C4 (Web Text))

Max Activating Token Index: 734

Text #18

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2156. Min Act: -0.0000

Data Index: 195546 (C4 (Web Text))

Max Activating Token Index: 191

Text #19

Max Range: 0.4328. Min Range: -0.4328

Max Act: 0.2002. Min Act: -0.0000

Data Index: 832194 (C4 (Web Text))

Max Activating Token Index: 501