Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 430 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.6345. Min Act: -0.0000

Data Index: 335121 (C4 (Web Text))

Max Activating Token Index: 279

Text #1

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.6515. Min Act: -0.0000

Data Index: 725569 (C4 (Web Text))

Max Activating Token Index: 1008

Text #2

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.6273. Min Act: -0.0000

Data Index: 976082 (C4 (Web Text))

Max Activating Token Index: 970

Text #3

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5312. Min Act: -0.0000

Data Index: 107799 (C4 (Web Text))

Max Activating Token Index: 199

Text #4

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5938. Min Act: -0.0000

Data Index: 860541 (C4 (Web Text))

Max Activating Token Index: 114

Text #5

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5238. Min Act: -0.0000

Data Index: 1170481 (C4 (Web Text))

Max Activating Token Index: 788

Text #6

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5543. Min Act: -0.0000

Data Index: 1142285 (C4 (Web Text))

Max Activating Token Index: 540

Text #7

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5531. Min Act: -0.0000

Data Index: 218405 (C4 (Web Text))

Max Activating Token Index: 377

Text #8

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5734. Min Act: -0.0000

Data Index: 506629 (C4 (Web Text))

Max Activating Token Index: 484

Text #9

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5334. Min Act: -0.0000

Data Index: 1128876 (C4 (Web Text))

Max Activating Token Index: 113

Text #10

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5176. Min Act: -0.0000

Data Index: 1013020 (C4 (Web Text))

Max Activating Token Index: 254

Text #11

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5009. Min Act: -0.0000

Data Index: 1098704 (C4 (Web Text))

Max Activating Token Index: 668

Text #12

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5187. Min Act: -0.0000

Data Index: 311581 (C4 (Web Text))

Max Activating Token Index: 494

Text #13

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5347. Min Act: -0.0000

Data Index: 702298 (C4 (Web Text))

Max Activating Token Index: 579

Text #14

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5107. Min Act: -0.0000

Data Index: 580291 (C4 (Web Text))

Max Activating Token Index: 475

Text #15

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5418. Min Act: -0.0000

Data Index: 1144826 (C4 (Web Text))

Max Activating Token Index: 382

Text #16

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5273. Min Act: -0.0000

Data Index: 1158056 (C4 (Web Text))

Max Activating Token Index: 884

Text #17

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5207. Min Act: -0.0000

Data Index: 865013 (C4 (Web Text))

Max Activating Token Index: 462

Text #18

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5189. Min Act: -0.0000

Data Index: 527195 (C4 (Web Text))

Max Activating Token Index: 671

Text #19

Max Range: 0.6515. Min Range: -0.6515

Max Act: 0.5186. Min Act: -0.0000

Data Index: 778138 (C4 (Web Text))

Max Activating Token Index: 319