Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3979 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.5096. Min Act: -0.0000

Data Index: 1074974 (C4 (Web Text))

Max Activating Token Index: 216

Text #1

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.4671. Min Act: -0.0000

Data Index: 609866 (C4 (Web Text))

Max Activating Token Index: 510

Text #2

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.4247. Min Act: -0.0000

Data Index: 714928 (C4 (Web Text))

Max Activating Token Index: 615

Text #3

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3825. Min Act: -0.0000

Data Index: 934444 (C4 (Web Text))

Max Activating Token Index: 643

Text #4

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3902. Min Act: 0.0001

Data Index: 787460 (C4 (Web Text))

Max Activating Token Index: 262

Text #5

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.4012. Min Act: -0.0000

Data Index: 287722 (C4 (Web Text))

Max Activating Token Index: 365

Text #6

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3680. Min Act: -0.0000

Data Index: 183127 (C4 (Web Text))

Max Activating Token Index: 971

Text #7

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3784. Min Act: -0.0000

Data Index: 1295825 (C4 (Web Text))

Max Activating Token Index: 995

Text #8

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3619. Min Act: -0.0000

Data Index: 1165576 (C4 (Web Text))

Max Activating Token Index: 521

Text #9

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3587. Min Act: 0.0000

Data Index: 797016 (C4 (Web Text))

Max Activating Token Index: 438

Text #10

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3621. Min Act: 0.0000

Data Index: 1090300 (C4 (Web Text))

Max Activating Token Index: 251

Text #11

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3610. Min Act: -0.0000

Data Index: 367391 (C4 (Web Text))

Max Activating Token Index: 473

Text #12

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3464. Min Act: 0.0000

Data Index: 938705 (C4 (Web Text))

Max Activating Token Index: 1012

Text #13

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3489. Min Act: 0.0000

Data Index: 442376 (C4 (Web Text))

Max Activating Token Index: 251

Text #14

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3440. Min Act: -0.0000

Data Index: 648733 (C4 (Web Text))

Max Activating Token Index: 686

Text #15

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3443. Min Act: -0.0000

Data Index: 350901 (C4 (Web Text))

Max Activating Token Index: 477

Text #16

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3318. Min Act: 0.0000

Data Index: 322117 (C4 (Web Text))

Max Activating Token Index: 537

Text #17

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3316. Min Act: 0.0000

Data Index: 723982 (C4 (Web Text))

Max Activating Token Index: 478

Text #18

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3417. Min Act: 0.0000

Data Index: 534061 (C4 (Web Text))

Max Activating Token Index: 373

Text #19

Max Range: 0.5096. Min Range: -0.5096

Max Act: 0.3274. Min Act: -0.0000

Data Index: 158778 (C4 (Web Text))

Max Activating Token Index: 957