Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2934 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0532. Min Act: -0.0002

Data Index: 873869 (C4 (Web Text))

Max Activating Token Index: 726

Text #1

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0384. Min Act: -0.0002

Data Index: 1075058 (C4 (Web Text))

Max Activating Token Index: 995

Text #2

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0438. Min Act: -0.0002

Data Index: 324442 (C4 (Web Text))

Max Activating Token Index: 930

Text #3

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0405. Min Act: -0.0002

Data Index: 1121265 (C4 (Web Text))

Max Activating Token Index: 891

Text #4

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0404. Min Act: -0.0002

Data Index: 42487 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0355. Min Act: -0.0002

Data Index: 11149 (C4 (Web Text))

Max Activating Token Index: 6

Text #6

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0425. Min Act: -0.0002

Data Index: 913161 (C4 (Web Text))

Max Activating Token Index: 62

Text #7

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0387. Min Act: -0.0002

Data Index: 880341 (C4 (Web Text))

Max Activating Token Index: 1023

Text #8

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0387. Min Act: -0.0002

Data Index: 755299 (C4 (Web Text))

Max Activating Token Index: 476

Text #9

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0509. Min Act: -0.0002

Data Index: 481948 (C4 (Web Text))

Max Activating Token Index: 653

Text #10

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0376. Min Act: -0.0002

Data Index: 500269 (C4 (Web Text))

Max Activating Token Index: 878

Text #11

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0347. Min Act: -0.0002

Data Index: 244586 (C4 (Web Text))

Max Activating Token Index: 878

Text #12

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0375. Min Act: -0.0002

Data Index: 641329 (C4 (Web Text))

Max Activating Token Index: 5

Text #13

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0401. Min Act: -0.0002

Data Index: 361044 (C4 (Web Text))

Max Activating Token Index: 443

Text #14

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0343. Min Act: -0.0002

Data Index: 553913 (C4 (Web Text))

Max Activating Token Index: 977

Text #15

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0334. Min Act: -0.0002

Data Index: 567058 (C4 (Web Text))

Max Activating Token Index: 713

Text #16

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0249. Min Act: -0.0002

Data Index: 431057 (C4 (Web Text))

Max Activating Token Index: 174

Text #17

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0329. Min Act: -0.0002

Data Index: 720910 (C4 (Web Text))

Max Activating Token Index: 943

Text #18

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0365. Min Act: -0.0002

Data Index: 1283335 (C4 (Web Text))

Max Activating Token Index: 907

Text #19

Max Range: 0.0532. Min Range: -0.0532

Max Act: 0.0361. Min Act: -0.0002

Data Index: 1238967 (C4 (Web Text))

Max Activating Token Index: 848