Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1482 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1033. Min Range: -2.1033

Max Act: 2.0042. Min Act: -0.0000

Data Index: 275431 (C4 (Web Text))

Max Activating Token Index: 375

Text #1

Max Range: 2.1033. Min Range: -2.1033

Max Act: 2.1033. Min Act: -0.0000

Data Index: 1239580 (C4 (Web Text))

Max Activating Token Index: 289

Text #2

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.9804. Min Act: -0.0000

Data Index: 488139 (C4 (Web Text))

Max Activating Token Index: 663

Text #3

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6986. Min Act: -0.0000

Data Index: 1268084 (C4 (Web Text))

Max Activating Token Index: 40

Text #4

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7242. Min Act: -0.0000

Data Index: 611209 (C4 (Web Text))

Max Activating Token Index: 909

Text #5

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.8206. Min Act: -0.0000

Data Index: 781899 (C4 (Web Text))

Max Activating Token Index: 770

Text #6

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7102. Min Act: -0.0000

Data Index: 567066 (C4 (Web Text))

Max Activating Token Index: 601

Text #7

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6123. Min Act: -0.0000

Data Index: 1070328 (C4 (Web Text))

Max Activating Token Index: 788

Text #8

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7834. Min Act: -0.0000

Data Index: 980295 (C4 (Web Text))

Max Activating Token Index: 675

Text #9

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7701. Min Act: -0.0000

Data Index: 153204 (C4 (Web Text))

Max Activating Token Index: 638

Text #10

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.9434. Min Act: -0.0000

Data Index: 971385 (C4 (Web Text))

Max Activating Token Index: 588

Text #11

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7274. Min Act: -0.0000

Data Index: 1123544 (C4 (Web Text))

Max Activating Token Index: 637

Text #12

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7279. Min Act: -0.0000

Data Index: 864985 (C4 (Web Text))

Max Activating Token Index: 306

Text #13

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6232. Min Act: -0.0000

Data Index: 502339 (C4 (Web Text))

Max Activating Token Index: 579

Text #14

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.7972. Min Act: -0.0000

Data Index: 471166 (C4 (Web Text))

Max Activating Token Index: 596

Text #15

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.5340. Min Act: -0.0000

Data Index: 1357685 (C4 (Web Text))

Max Activating Token Index: 654

Text #16

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6969. Min Act: -0.0000

Data Index: 483532 (C4 (Web Text))

Max Activating Token Index: 650

Text #17

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6846. Min Act: -0.0000

Data Index: 1135813 (C4 (Web Text))

Max Activating Token Index: 778

Text #18

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.6007. Min Act: -0.0000

Data Index: 1088743 (C4 (Web Text))

Max Activating Token Index: 705

Text #19

Max Range: 2.1033. Min Range: -2.1033

Max Act: 1.5872. Min Act: -0.0000

Data Index: 893707 (C4 (Web Text))

Max Activating Token Index: 880