Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1614 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3102. Min Range: -2.3102

Max Act: 2.3102. Min Act: -0.0000

Data Index: 112868 (C4 (Web Text))

Max Activating Token Index: 570

Text #1

Max Range: 2.3102. Min Range: -2.3102

Max Act: 2.2039. Min Act: -0.0000

Data Index: 168711 (C4 (Web Text))

Max Activating Token Index: 24

Text #2

Max Range: 2.3102. Min Range: -2.3102

Max Act: 2.0412. Min Act: -0.0000

Data Index: 1218008 (C4 (Web Text))

Max Activating Token Index: 25

Text #3

Max Range: 2.3102. Min Range: -2.3102

Max Act: 2.0390. Min Act: -0.0000

Data Index: 754717 (C4 (Web Text))

Max Activating Token Index: 932

Text #4

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.9764. Min Act: -0.0000

Data Index: 54590 (C4 (Web Text))

Max Activating Token Index: 658

Text #5

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.7588. Min Act: -0.0000

Data Index: 208595 (C4 (Web Text))

Max Activating Token Index: 38

Text #6

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.7655. Min Act: -0.0000

Data Index: 787961 (C4 (Web Text))

Max Activating Token Index: 136

Text #7

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.7400. Min Act: -0.0000

Data Index: 471063 (C4 (Web Text))

Max Activating Token Index: 11

Text #8

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.7334. Min Act: -0.0000

Data Index: 43035 (C4 (Web Text))

Max Activating Token Index: 36

Text #9

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.5236. Min Act: -0.0000

Data Index: 1336626 (C4 (Web Text))

Max Activating Token Index: 488

Text #10

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.5166. Min Act: -0.0000

Data Index: 678124 (C4 (Web Text))

Max Activating Token Index: 19

Text #11

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.5112. Min Act: 0.0000

Data Index: 692985 (C4 (Web Text))

Max Activating Token Index: 332

Text #12

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.5440. Min Act: 0.0003

Data Index: 1048387 (C4 (Web Text))

Max Activating Token Index: 30

Text #13

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.6001. Min Act: -0.0000

Data Index: 440505 (C4 (Web Text))

Max Activating Token Index: 809

Text #14

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.4777. Min Act: -0.0000

Data Index: 221799 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.4556. Min Act: 0.0000

Data Index: 502065 (C4 (Web Text))

Max Activating Token Index: 867

Text #16

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.4363. Min Act: -0.0000

Data Index: 163545 (C4 (Web Text))

Max Activating Token Index: 76

Text #17

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.4289. Min Act: -0.0000

Data Index: 1177127 (C4 (Web Text))

Max Activating Token Index: 426

Text #18

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.3081. Min Act: -0.0000

Data Index: 308960 (C4 (Web Text))

Max Activating Token Index: 603

Text #19

Max Range: 2.3102. Min Range: -2.3102

Max Act: 1.2753. Min Act: -0.0000

Data Index: 281737 (C4 (Web Text))

Max Activating Token Index: 55