Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1623 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.8226. Min Act: -0.0000

Data Index: 814790 (C4 (Web Text))

Max Activating Token Index: 621

Text #1

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.8398. Min Act: -0.0000

Data Index: 286427 (C4 (Web Text))

Max Activating Token Index: 751

Text #2

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.7242. Min Act: -0.0000

Data Index: 162848 (C4 (Web Text))

Max Activating Token Index: 241

Text #3

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6714. Min Act: -0.0000

Data Index: 1177371 (C4 (Web Text))

Max Activating Token Index: 112

Text #4

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6536. Min Act: -0.0000

Data Index: 427370 (C4 (Web Text))

Max Activating Token Index: 766

Text #5

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6607. Min Act: -0.0000

Data Index: 1028927 (C4 (Web Text))

Max Activating Token Index: 564

Text #6

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6208. Min Act: -0.0000

Data Index: 794091 (C4 (Web Text))

Max Activating Token Index: 253

Text #7

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6507. Min Act: -0.0000

Data Index: 1006426 (C4 (Web Text))

Max Activating Token Index: 93

Text #8

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6147. Min Act: -0.0000

Data Index: 126551 (C4 (Web Text))

Max Activating Token Index: 391

Text #9

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6080. Min Act: -0.0000

Data Index: 678045 (C4 (Web Text))

Max Activating Token Index: 173

Text #10

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5637. Min Act: -0.0000

Data Index: 625140 (C4 (Web Text))

Max Activating Token Index: 393

Text #11

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5967. Min Act: -0.0000

Data Index: 279749 (C4 (Web Text))

Max Activating Token Index: 740

Text #12

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5918. Min Act: -0.0000

Data Index: 1055735 (C4 (Web Text))

Max Activating Token Index: 141

Text #13

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5614. Min Act: -0.0000

Data Index: 290513 (C4 (Web Text))

Max Activating Token Index: 26

Text #14

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5571. Min Act: -0.0000

Data Index: 846213 (C4 (Web Text))

Max Activating Token Index: 827

Text #15

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5693. Min Act: -0.0000

Data Index: 345629 (C4 (Web Text))

Max Activating Token Index: 522

Text #16

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.6012. Min Act: -0.0000

Data Index: 481435 (C4 (Web Text))

Max Activating Token Index: 526

Text #17

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5759. Min Act: -0.0000

Data Index: 1113242 (C4 (Web Text))

Max Activating Token Index: 367

Text #18

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5425. Min Act: -0.0000

Data Index: 755054 (C4 (Web Text))

Max Activating Token Index: 920

Text #19

Max Range: 0.8398. Min Range: -0.8398

Max Act: 0.5680. Min Act: -0.0000

Data Index: 203160 (C4 (Web Text))

Max Activating Token Index: 440