Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 31 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.8353. Min Act: -0.0001

Data Index: 405710 (C4 (Web Text))

Max Activating Token Index: 837

Text #1

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.5797. Min Act: -0.0001

Data Index: 1139234 (C4 (Web Text))

Max Activating Token Index: 85

Text #2

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.3779. Min Act: -0.0001

Data Index: 919475 (C4 (Web Text))

Max Activating Token Index: 755

Text #3

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.1982. Min Act: -0.0001

Data Index: 225660 (C4 (Web Text))

Max Activating Token Index: 644

Text #4

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.8616. Min Act: -0.0001

Data Index: 1044360 (C4 (Web Text))

Max Activating Token Index: 718

Text #5

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.5134. Min Act: -0.0001

Data Index: 813850 (C4 (Web Text))

Max Activating Token Index: 395

Text #6

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.3048. Min Act: -0.0001

Data Index: 384594 (C4 (Web Text))

Max Activating Token Index: 436

Text #7

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.7956. Min Act: -0.0001

Data Index: 9498 (C4 (Web Text))

Max Activating Token Index: 663

Text #8

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.6434. Min Act: -0.0001

Data Index: 907241 (C4 (Web Text))

Max Activating Token Index: 847

Text #9

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.4897. Min Act: -0.0001

Data Index: 652387 (C4 (Web Text))

Max Activating Token Index: 662

Text #10

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.1531. Min Act: -0.0001

Data Index: 542213 (C4 (Web Text))

Max Activating Token Index: 582

Text #11

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.9053. Min Act: -0.0001

Data Index: 564117 (C4 (Web Text))

Max Activating Token Index: 590

Text #12

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.9066. Min Act: -0.0001

Data Index: 22187 (C4 (Web Text))

Max Activating Token Index: 180

Text #13

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.2453. Min Act: -0.0001

Data Index: 331070 (C4 (Web Text))

Max Activating Token Index: 504

Text #14

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.1751. Min Act: -0.0001

Data Index: 217477 (C4 (Web Text))

Max Activating Token Index: 404

Text #15

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.9741. Min Act: -0.0001

Data Index: 869066 (C4 (Web Text))

Max Activating Token Index: 719

Text #16

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.9538. Min Act: -0.0001

Data Index: 1069664 (C4 (Web Text))

Max Activating Token Index: 113

Text #17

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.7793. Min Act: -0.0001

Data Index: 244903 (C4 (Web Text))

Max Activating Token Index: 755

Text #18

Max Range: 5.8616. Min Range: -5.8616

Max Act: 4.8968. Min Act: -0.0001

Data Index: 849270 (C4 (Web Text))

Max Activating Token Index: 394

Text #19

Max Range: 5.8616. Min Range: -5.8616

Max Act: 5.0158. Min Act: -0.0001

Data Index: 1071907 (C4 (Web Text))

Max Activating Token Index: 775