Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2789 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9212. Min Act: -0.0001

Data Index: 241561 (C4 (Web Text))

Max Activating Token Index: 283

Text #1

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9253. Min Act: -0.0001

Data Index: 480 (C4 (Web Text))

Max Activating Token Index: 21

Text #2

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9359. Min Act: -0.0001

Data Index: 399894 (C4 (Web Text))

Max Activating Token Index: 310

Text #3

Max Range: 1.0772. Min Range: -1.0772

Max Act: 1.0772. Min Act: -0.0001

Data Index: 899530 (C4 (Web Text))

Max Activating Token Index: 235

Text #4

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8792. Min Act: -0.0001

Data Index: 764569 (C4 (Web Text))

Max Activating Token Index: 239

Text #5

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9567. Min Act: -0.0001

Data Index: 614085 (C4 (Web Text))

Max Activating Token Index: 949

Text #6

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9011. Min Act: -0.0001

Data Index: 666168 (C4 (Web Text))

Max Activating Token Index: 910

Text #7

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.9122. Min Act: -0.0001

Data Index: 86640 (C4 (Web Text))

Max Activating Token Index: 66

Text #8

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8419. Min Act: -0.0001

Data Index: 87951 (C4 (Web Text))

Max Activating Token Index: 11

Text #9

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8623. Min Act: -0.0001

Data Index: 476863 (C4 (Web Text))

Max Activating Token Index: 129

Text #10

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7287. Min Act: -0.0001

Data Index: 944090 (C4 (Web Text))

Max Activating Token Index: 363

Text #11

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7154. Min Act: -0.0001

Data Index: 1129327 (C4 (Web Text))

Max Activating Token Index: 208

Text #12

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8652. Min Act: -0.0001

Data Index: 634983 (C4 (Web Text))

Max Activating Token Index: 958

Text #13

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7886. Min Act: -0.0001

Data Index: 1130246 (C4 (Web Text))

Max Activating Token Index: 140

Text #14

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7460. Min Act: -0.0001

Data Index: 320050 (C4 (Web Text))

Max Activating Token Index: 917

Text #15

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8023. Min Act: -0.0001

Data Index: 1586191 (Python Code)

Max Activating Token Index: 138

Text #16

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7422. Min Act: -0.0001

Data Index: 522116 (C4 (Web Text))

Max Activating Token Index: 574

Text #17

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8090. Min Act: -0.0001

Data Index: 21675 (C4 (Web Text))

Max Activating Token Index: 773

Text #18

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.7508. Min Act: -0.0001

Data Index: 1083037 (C4 (Web Text))

Max Activating Token Index: 253

Text #19

Max Range: 1.0772. Min Range: -1.0772

Max Act: 0.8319. Min Act: -0.0001

Data Index: 203121 (C4 (Web Text))

Max Activating Token Index: 981