Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1521 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2824. Min Range: -3.2824

Max Act: 3.2824. Min Act: -0.0000

Data Index: 857229 (C4 (Web Text))

Max Activating Token Index: 440

Text #1

Max Range: 3.2824. Min Range: -3.2824

Max Act: 3.1152. Min Act: -0.0000

Data Index: 248386 (C4 (Web Text))

Max Activating Token Index: 843

Text #2

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.8205. Min Act: -0.0000

Data Index: 240474 (C4 (Web Text))

Max Activating Token Index: 723

Text #3

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.8752. Min Act: -0.0000

Data Index: 1305988 (C4 (Web Text))

Max Activating Token Index: 106

Text #4

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.3457. Min Act: -0.0000

Data Index: 1031377 (C4 (Web Text))

Max Activating Token Index: 220

Text #5

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.4961. Min Act: -0.0000

Data Index: 1332134 (C4 (Web Text))

Max Activating Token Index: 96

Text #6

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.4732. Min Act: -0.0000

Data Index: 256180 (C4 (Web Text))

Max Activating Token Index: 35

Text #7

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.4206. Min Act: -0.0000

Data Index: 17409 (C4 (Web Text))

Max Activating Token Index: 85

Text #8

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.5157. Min Act: -0.0000

Data Index: 189836 (C4 (Web Text))

Max Activating Token Index: 88

Text #9

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.1591. Min Act: -0.0000

Data Index: 535707 (C4 (Web Text))

Max Activating Token Index: 261

Text #10

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.4917. Min Act: -0.0000

Data Index: 1308698 (C4 (Web Text))

Max Activating Token Index: 213

Text #11

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.1108. Min Act: -0.0000

Data Index: 980156 (C4 (Web Text))

Max Activating Token Index: 283

Text #12

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.2830. Min Act: -0.0000

Data Index: 218438 (C4 (Web Text))

Max Activating Token Index: 43

Text #13

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.1571. Min Act: -0.0000

Data Index: 757732 (C4 (Web Text))

Max Activating Token Index: 375

Text #14

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.2636. Min Act: -0.0000

Data Index: 1042007 (C4 (Web Text))

Max Activating Token Index: 438

Text #15

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.2089. Min Act: -0.0000

Data Index: 779403 (C4 (Web Text))

Max Activating Token Index: 937

Text #16

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.2152. Min Act: -0.0000

Data Index: 1179043 (C4 (Web Text))

Max Activating Token Index: 91

Text #17

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.4066. Min Act: -0.0001

Data Index: 929680 (C4 (Web Text))

Max Activating Token Index: 54

Text #18

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.0780. Min Act: -0.0001

Data Index: 802501 (C4 (Web Text))

Max Activating Token Index: 81

Text #19

Max Range: 3.2824. Min Range: -3.2824

Max Act: 2.1714. Min Act: -0.0000

Data Index: 318915 (C4 (Web Text))

Max Activating Token Index: 415