Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1854 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.3018. Min Range: -3.3018

Max Act: 3.3018. Min Act: -0.0000

Data Index: 1080135 (C4 (Web Text))

Max Activating Token Index: 704

Text #1

Max Range: 3.3018. Min Range: -3.3018

Max Act: 3.2366. Min Act: -0.0000

Data Index: 913777 (C4 (Web Text))

Max Activating Token Index: 942

Text #2

Max Range: 3.3018. Min Range: -3.3018

Max Act: 3.0363. Min Act: -0.0000

Data Index: 253967 (C4 (Web Text))

Max Activating Token Index: 975

Text #3

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.7877. Min Act: -0.0000

Data Index: 437185 (C4 (Web Text))

Max Activating Token Index: 1016

Text #4

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.8451. Min Act: -0.0000

Data Index: 1263063 (C4 (Web Text))

Max Activating Token Index: 440

Text #5

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.5841. Min Act: -0.0000

Data Index: 1001050 (C4 (Web Text))

Max Activating Token Index: 485

Text #6

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.5059. Min Act: -0.0000

Data Index: 730600 (C4 (Web Text))

Max Activating Token Index: 974

Text #7

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.4665. Min Act: -0.0000

Data Index: 478250 (C4 (Web Text))

Max Activating Token Index: 962

Text #8

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.4086. Min Act: -0.0000

Data Index: 667912 (C4 (Web Text))

Max Activating Token Index: 29

Text #9

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.5686. Min Act: -0.0000

Data Index: 87831 (C4 (Web Text))

Max Activating Token Index: 124

Text #10

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.0454. Min Act: -0.0000

Data Index: 172953 (C4 (Web Text))

Max Activating Token Index: 695

Text #11

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.0618. Min Act: -0.0000

Data Index: 118654 (C4 (Web Text))

Max Activating Token Index: 237

Text #12

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.1322. Min Act: -0.0000

Data Index: 1213009 (C4 (Web Text))

Max Activating Token Index: 306

Text #13

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.2381. Min Act: -0.0000

Data Index: 830800 (C4 (Web Text))

Max Activating Token Index: 254

Text #14

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.1205. Min Act: -0.0000

Data Index: 977052 (C4 (Web Text))

Max Activating Token Index: 899

Text #15

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.2427. Min Act: -0.0000

Data Index: 107497 (C4 (Web Text))

Max Activating Token Index: 792

Text #16

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.0003. Min Act: -0.0000

Data Index: 903432 (C4 (Web Text))

Max Activating Token Index: 228

Text #17

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.0922. Min Act: -0.0000

Data Index: 1157836 (C4 (Web Text))

Max Activating Token Index: 993

Text #18

Max Range: 3.3018. Min Range: -3.3018

Max Act: 2.0917. Min Act: -0.0000

Data Index: 130560 (C4 (Web Text))

Max Activating Token Index: 206

Text #19

Max Range: 3.3018. Min Range: -3.3018

Max Act: 1.9781. Min Act: -0.0000

Data Index: 333055 (C4 (Web Text))

Max Activating Token Index: 900