Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4205 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.8894. Min Range: -6.8894

Max Act: 6.8894. Min Act: -0.0000

Data Index: 369720 (C4 (Web Text))

Max Activating Token Index: 149

Text #1

Max Range: 6.8894. Min Range: -6.8894

Max Act: 6.8279. Min Act: -0.0000

Data Index: 945047 (C4 (Web Text))

Max Activating Token Index: 378

Text #2

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.3825. Min Act: -0.0000

Data Index: 1120849 (C4 (Web Text))

Max Activating Token Index: 842

Text #3

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.3138. Min Act: -0.0000

Data Index: 839916 (C4 (Web Text))

Max Activating Token Index: 114

Text #4

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.0430. Min Act: -0.0000

Data Index: 644142 (C4 (Web Text))

Max Activating Token Index: 859

Text #5

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.2023. Min Act: -0.0000

Data Index: 326804 (C4 (Web Text))

Max Activating Token Index: 113

Text #6

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.1764. Min Act: -0.0000

Data Index: 1347490 (C4 (Web Text))

Max Activating Token Index: 156

Text #7

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.0484. Min Act: -0.0000

Data Index: 651445 (C4 (Web Text))

Max Activating Token Index: 573

Text #8

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.9998. Min Act: -0.0000

Data Index: 270870 (C4 (Web Text))

Max Activating Token Index: 501

Text #9

Max Range: 6.8894. Min Range: -6.8894

Max Act: 4.0194. Min Act: -0.0000

Data Index: 1182710 (C4 (Web Text))

Max Activating Token Index: 881

Text #10

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.8292. Min Act: -0.0000

Data Index: 1071085 (C4 (Web Text))

Max Activating Token Index: 153

Text #11

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.7946. Min Act: -0.0000

Data Index: 922954 (C4 (Web Text))

Max Activating Token Index: 810

Text #12

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.6222. Min Act: -0.0000

Data Index: 1407976 (Python Code)

Max Activating Token Index: 752

Text #13

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.4752. Min Act: -0.0000

Data Index: 1617702 (Python Code)

Max Activating Token Index: 780

Text #14

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.2891. Min Act: -0.0000

Data Index: 433761 (C4 (Web Text))

Max Activating Token Index: 806

Text #15

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.7502. Min Act: -0.0000

Data Index: 129802 (C4 (Web Text))

Max Activating Token Index: 129

Text #16

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.3885. Min Act: -0.0000

Data Index: 1421315 (Python Code)

Max Activating Token Index: 870

Text #17

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.5365. Min Act: -0.0000

Data Index: 1202401 (C4 (Web Text))

Max Activating Token Index: 710

Text #18

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.6591. Min Act: -0.0000

Data Index: 815510 (C4 (Web Text))

Max Activating Token Index: 784

Text #19

Max Range: 6.8894. Min Range: -6.8894

Max Act: 3.4530. Min Act: -0.0000

Data Index: 1073062 (C4 (Web Text))

Max Activating Token Index: 305