Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1655 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.5507. Min Act: -0.0001

Data Index: 1104421 (C4 (Web Text))

Max Activating Token Index: 621

Text #1

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3914. Min Act: -0.0001

Data Index: 826207 (C4 (Web Text))

Max Activating Token Index: 898

Text #2

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3534. Min Act: -0.0001

Data Index: 682569 (C4 (Web Text))

Max Activating Token Index: 906

Text #3

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3520. Min Act: -0.0001

Data Index: 225310 (C4 (Web Text))

Max Activating Token Index: 254

Text #4

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3445. Min Act: -0.0001

Data Index: 827733 (C4 (Web Text))

Max Activating Token Index: 689

Text #5

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3427. Min Act: -0.0001

Data Index: 82795 (C4 (Web Text))

Max Activating Token Index: 631

Text #6

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3398. Min Act: -0.0001

Data Index: 1306149 (C4 (Web Text))

Max Activating Token Index: 483

Text #7

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3504. Min Act: -0.0000

Data Index: 601102 (C4 (Web Text))

Max Activating Token Index: 918

Text #8

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3418. Min Act: -0.0001

Data Index: 1005704 (C4 (Web Text))

Max Activating Token Index: 116

Text #9

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3285. Min Act: -0.0001

Data Index: 1194083 (C4 (Web Text))

Max Activating Token Index: 432

Text #10

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3298. Min Act: -0.0001

Data Index: 1206430 (C4 (Web Text))

Max Activating Token Index: 185

Text #11

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3268. Min Act: -0.0001

Data Index: 87174 (C4 (Web Text))

Max Activating Token Index: 471

Text #12

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3236. Min Act: -0.0001

Data Index: 172844 (C4 (Web Text))

Max Activating Token Index: 560

Text #13

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3113. Min Act: -0.0001

Data Index: 850369 (C4 (Web Text))

Max Activating Token Index: 376

Text #14

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3111. Min Act: -0.0001

Data Index: 390226 (C4 (Web Text))

Max Activating Token Index: 335

Text #15

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3072. Min Act: -0.0001

Data Index: 189649 (C4 (Web Text))

Max Activating Token Index: 794

Text #16

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3149. Min Act: -0.0001

Data Index: 1290912 (C4 (Web Text))

Max Activating Token Index: 350

Text #17

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3029. Min Act: -0.0001

Data Index: 280039 (C4 (Web Text))

Max Activating Token Index: 943

Text #18

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3120. Min Act: -0.0001

Data Index: 1118132 (C4 (Web Text))

Max Activating Token Index: 575

Text #19

Max Range: 0.5507. Min Range: -0.5507

Max Act: 0.3012. Min Act: -0.0001

Data Index: 79238 (C4 (Web Text))

Max Activating Token Index: 594