Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3486 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.9359. Min Act: -0.0001

Data Index: 897767 (C4 (Web Text))

Max Activating Token Index: 1023

Text #1

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8641. Min Act: -0.0001

Data Index: 866685 (C4 (Web Text))

Max Activating Token Index: 1023

Text #2

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8482. Min Act: -0.0001

Data Index: 807280 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8499. Min Act: -0.0001

Data Index: 144156 (C4 (Web Text))

Max Activating Token Index: 1023

Text #4

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8430. Min Act: -0.0001

Data Index: 410197 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8510. Min Act: -0.0001

Data Index: 480133 (C4 (Web Text))

Max Activating Token Index: 1023

Text #6

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8495. Min Act: -0.0001

Data Index: 1308968 (C4 (Web Text))

Max Activating Token Index: 1023

Text #7

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8261. Min Act: -0.0002

Data Index: 829560 (C4 (Web Text))

Max Activating Token Index: 1023

Text #8

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8369. Min Act: -0.0001

Data Index: 565697 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8250. Min Act: -0.0001

Data Index: 992878 (C4 (Web Text))

Max Activating Token Index: 1023

Text #10

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8238. Min Act: -0.0001

Data Index: 1210920 (C4 (Web Text))

Max Activating Token Index: 1023

Text #11

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8256. Min Act: -0.0001

Data Index: 1266353 (C4 (Web Text))

Max Activating Token Index: 1023

Text #12

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8208. Min Act: -0.0001

Data Index: 842119 (C4 (Web Text))

Max Activating Token Index: 1023

Text #13

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8118. Min Act: -0.0001

Data Index: 1322567 (C4 (Web Text))

Max Activating Token Index: 1023

Text #14

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.7862. Min Act: -0.0001

Data Index: 104583 (C4 (Web Text))

Max Activating Token Index: 1023

Text #15

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.8020. Min Act: -0.0001

Data Index: 1338232 (C4 (Web Text))

Max Activating Token Index: 1023

Text #16

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.7826. Min Act: -0.0001

Data Index: 518326 (C4 (Web Text))

Max Activating Token Index: 1023

Text #17

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.7768. Min Act: -0.0001

Data Index: 1121058 (C4 (Web Text))

Max Activating Token Index: 1023

Text #18

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.7798. Min Act: -0.0001

Data Index: 705686 (C4 (Web Text))

Max Activating Token Index: 1023

Text #19

Max Range: 0.9359. Min Range: -0.9359

Max Act: 0.7790. Min Act: -0.0001

Data Index: 660172 (C4 (Web Text))

Max Activating Token Index: 1023