Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4328 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1887. Min Act: -0.0001

Data Index: 172467 (C4 (Web Text))

Max Activating Token Index: 800

Text #1

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1543. Min Act: -0.0001

Data Index: 873836 (C4 (Web Text))

Max Activating Token Index: 560

Text #2

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1398. Min Act: -0.0001

Data Index: 632452 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1353. Min Act: -0.0001

Data Index: 489910 (C4 (Web Text))

Max Activating Token Index: 670

Text #4

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1631. Min Act: -0.0001

Data Index: 164596 (C4 (Web Text))

Max Activating Token Index: 42

Text #5

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1371. Min Act: -0.0001

Data Index: 1140137 (C4 (Web Text))

Max Activating Token Index: 1023

Text #6

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1182. Min Act: -0.0001

Data Index: 799842 (C4 (Web Text))

Max Activating Token Index: 78

Text #7

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1125. Min Act: -0.0001

Data Index: 428284 (C4 (Web Text))

Max Activating Token Index: 1023

Text #8

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1167. Min Act: -0.0001

Data Index: 957535 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1034. Min Act: -0.0001

Data Index: 1305757 (C4 (Web Text))

Max Activating Token Index: 1023

Text #10

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1124. Min Act: -0.0002

Data Index: 676191 (C4 (Web Text))

Max Activating Token Index: 906

Text #11

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1021. Min Act: -0.0001

Data Index: 853152 (C4 (Web Text))

Max Activating Token Index: 975

Text #12

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1203. Min Act: -0.0001

Data Index: 979892 (C4 (Web Text))

Max Activating Token Index: 240

Text #13

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1195. Min Act: -0.0001

Data Index: 798485 (C4 (Web Text))

Max Activating Token Index: 199

Text #14

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.0991. Min Act: -0.0001

Data Index: 1341508 (C4 (Web Text))

Max Activating Token Index: 1023

Text #15

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1027. Min Act: -0.0001

Data Index: 648463 (C4 (Web Text))

Max Activating Token Index: 275

Text #16

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.1213. Min Act: -0.0001

Data Index: 1015529 (C4 (Web Text))

Max Activating Token Index: 722

Text #17

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.0909. Min Act: -0.0001

Data Index: 782843 (C4 (Web Text))

Max Activating Token Index: 906

Text #18

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.0941. Min Act: -0.0001

Data Index: 249867 (C4 (Web Text))

Max Activating Token Index: 82

Text #19

Max Range: 1.1887. Min Range: -1.1887

Max Act: 1.0944. Min Act: -0.0001

Data Index: 331226 (C4 (Web Text))

Max Activating Token Index: 1023