Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1881 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.5541. Min Range: -4.5541

Max Act: 4.0537. Min Act: -0.0000

Data Index: 492026 (C4 (Web Text))

Max Activating Token Index: 986

Text #1

Max Range: 4.5541. Min Range: -4.5541

Max Act: 4.5541. Min Act: -0.0001

Data Index: 567907 (C4 (Web Text))

Max Activating Token Index: 165

Text #2

Max Range: 4.5541. Min Range: -4.5541

Max Act: 4.2210. Min Act: -0.0000

Data Index: 436202 (C4 (Web Text))

Max Activating Token Index: 859

Text #3

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.7965. Min Act: -0.0000

Data Index: 124503 (C4 (Web Text))

Max Activating Token Index: 723

Text #4

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.8372. Min Act: -0.0000

Data Index: 701693 (C4 (Web Text))

Max Activating Token Index: 480

Text #5

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.1223. Min Act: -0.0000

Data Index: 1336874 (C4 (Web Text))

Max Activating Token Index: 379

Text #6

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.0081. Min Act: -0.0000

Data Index: 422950 (C4 (Web Text))

Max Activating Token Index: 101

Text #7

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.9894. Min Act: -0.0001

Data Index: 1152822 (C4 (Web Text))

Max Activating Token Index: 81

Text #8

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.0486. Min Act: -0.0000

Data Index: 1299779 (C4 (Web Text))

Max Activating Token Index: 942

Text #9

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.3109. Min Act: -0.0000

Data Index: 160606 (C4 (Web Text))

Max Activating Token Index: 53

Text #10

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.1494. Min Act: -0.0000

Data Index: 1266869 (C4 (Web Text))

Max Activating Token Index: 738

Text #11

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.7574. Min Act: -0.0000

Data Index: 1052286 (C4 (Web Text))

Max Activating Token Index: 202

Text #12

Max Range: 4.5541. Min Range: -4.5541

Max Act: 3.0044. Min Act: -0.0000

Data Index: 510215 (C4 (Web Text))

Max Activating Token Index: 178

Text #13

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.8326. Min Act: -0.0000

Data Index: 866839 (C4 (Web Text))

Max Activating Token Index: 28

Text #14

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.6689. Min Act: -0.0000

Data Index: 264452 (C4 (Web Text))

Max Activating Token Index: 443

Text #15

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.5965. Min Act: -0.0000

Data Index: 1158757 (C4 (Web Text))

Max Activating Token Index: 945

Text #16

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.6915. Min Act: -0.0000

Data Index: 599969 (C4 (Web Text))

Max Activating Token Index: 257

Text #17

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.3198. Min Act: -0.0000

Data Index: 58855 (C4 (Web Text))

Max Activating Token Index: 32

Text #18

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.6860. Min Act: -0.0000

Data Index: 567130 (C4 (Web Text))

Max Activating Token Index: 893

Text #19

Max Range: 4.5541. Min Range: -4.5541

Max Act: 2.7128. Min Act: -0.0001

Data Index: 393897 (C4 (Web Text))

Max Activating Token Index: 959