Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 51 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.9664. Min Act: -0.0000

Data Index: 1049418 (C4 (Web Text))

Max Activating Token Index: 668

Text #1

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.8567. Min Act: -0.0000

Data Index: 64097 (C4 (Web Text))

Max Activating Token Index: 816

Text #2

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.7553. Min Act: -0.0000

Data Index: 55836 (C4 (Web Text))

Max Activating Token Index: 670

Text #3

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.7935. Min Act: -0.0000

Data Index: 415125 (C4 (Web Text))

Max Activating Token Index: 616

Text #4

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.7274. Min Act: -0.0000

Data Index: 1237577 (C4 (Web Text))

Max Activating Token Index: 799

Text #5

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6845. Min Act: -0.0000

Data Index: 377935 (C4 (Web Text))

Max Activating Token Index: 809

Text #6

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6932. Min Act: -0.0000

Data Index: 862384 (C4 (Web Text))

Max Activating Token Index: 910

Text #7

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6747. Min Act: -0.0000

Data Index: 551047 (C4 (Web Text))

Max Activating Token Index: 684

Text #8

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6623. Min Act: -0.0000

Data Index: 99850 (C4 (Web Text))

Max Activating Token Index: 973

Text #9

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6521. Min Act: -0.0000

Data Index: 499331 (C4 (Web Text))

Max Activating Token Index: 317

Text #10

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6355. Min Act: -0.0000

Data Index: 694611 (C4 (Web Text))

Max Activating Token Index: 652

Text #11

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6435. Min Act: -0.0000

Data Index: 760674 (C4 (Web Text))

Max Activating Token Index: 655

Text #12

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6110. Min Act: -0.0000

Data Index: 324415 (C4 (Web Text))

Max Activating Token Index: 218

Text #13

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6377. Min Act: -0.0000

Data Index: 409738 (C4 (Web Text))

Max Activating Token Index: 459

Text #14

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6304. Min Act: -0.0000

Data Index: 147005 (C4 (Web Text))

Max Activating Token Index: 48

Text #15

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6056. Min Act: -0.0000

Data Index: 719324 (C4 (Web Text))

Max Activating Token Index: 952

Text #16

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.5780. Min Act: -0.0000

Data Index: 878225 (C4 (Web Text))

Max Activating Token Index: 54

Text #17

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6219. Min Act: -0.0000

Data Index: 591479 (C4 (Web Text))

Max Activating Token Index: 543

Text #18

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.6254. Min Act: -0.0000

Data Index: 502 (C4 (Web Text))

Max Activating Token Index: 802

Text #19

Max Range: 0.9664. Min Range: -0.9664

Max Act: 0.5690. Min Act: -0.0000

Data Index: 215638 (C4 (Web Text))

Max Activating Token Index: 856