Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3258 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3018. Min Range: -1.3018

Max Act: 1.3018. Min Act: -0.0001

Data Index: 1296608 (C4 (Web Text))

Max Activating Token Index: 158

Text #1

Max Range: 1.3018. Min Range: -1.3018

Max Act: 1.0007. Min Act: -0.0001

Data Index: 164143 (C4 (Web Text))

Max Activating Token Index: 797

Text #2

Max Range: 1.3018. Min Range: -1.3018

Max Act: 1.0309. Min Act: -0.0001

Data Index: 1223368 (C4 (Web Text))

Max Activating Token Index: 958

Text #3

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9730. Min Act: -0.0001

Data Index: 936339 (C4 (Web Text))

Max Activating Token Index: 226

Text #4

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9389. Min Act: -0.0001

Data Index: 698527 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9639. Min Act: -0.0001

Data Index: 269229 (C4 (Web Text))

Max Activating Token Index: 890

Text #6

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9312. Min Act: -0.0001

Data Index: 391164 (C4 (Web Text))

Max Activating Token Index: 416

Text #7

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9636. Min Act: -0.0001

Data Index: 578718 (C4 (Web Text))

Max Activating Token Index: 644

Text #8

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9270. Min Act: -0.0001

Data Index: 9914 (C4 (Web Text))

Max Activating Token Index: 227

Text #9

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9494. Min Act: -0.0001

Data Index: 79897 (C4 (Web Text))

Max Activating Token Index: 891

Text #10

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9587. Min Act: -0.0001

Data Index: 38651 (C4 (Web Text))

Max Activating Token Index: 497

Text #11

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9855. Min Act: -0.0001

Data Index: 702645 (C4 (Web Text))

Max Activating Token Index: 160

Text #12

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9779. Min Act: -0.0001

Data Index: 1137509 (C4 (Web Text))

Max Activating Token Index: 902

Text #13

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.8990. Min Act: -0.0001

Data Index: 366079 (C4 (Web Text))

Max Activating Token Index: 241

Text #14

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9408. Min Act: -0.0001

Data Index: 570413 (C4 (Web Text))

Max Activating Token Index: 567

Text #15

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9173. Min Act: -0.0001

Data Index: 959175 (C4 (Web Text))

Max Activating Token Index: 172

Text #16

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9008. Min Act: -0.0001

Data Index: 441633 (C4 (Web Text))

Max Activating Token Index: 922

Text #17

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9216. Min Act: -0.0001

Data Index: 1174055 (C4 (Web Text))

Max Activating Token Index: 652

Text #18

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9731. Min Act: -0.0001

Data Index: 313345 (C4 (Web Text))

Max Activating Token Index: 906

Text #19

Max Range: 1.3018. Min Range: -1.3018

Max Act: 0.9103. Min Act: -0.0001

Data Index: 441507 (C4 (Web Text))

Max Activating Token Index: 155