Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2569 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2464. Min Act: -0.0000

Data Index: 203429 (C4 (Web Text))

Max Activating Token Index: 707

Text #1

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2407. Min Act: -0.0000

Data Index: 1043782 (C4 (Web Text))

Max Activating Token Index: 163

Text #2

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2232. Min Act: -0.0000

Data Index: 998187 (C4 (Web Text))

Max Activating Token Index: 47

Text #3

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2191. Min Act: -0.0000

Data Index: 327445 (C4 (Web Text))

Max Activating Token Index: 736

Text #4

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2346. Min Act: -0.0000

Data Index: 1122290 (C4 (Web Text))

Max Activating Token Index: 519

Text #5

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2107. Min Act: -0.0000

Data Index: 238057 (C4 (Web Text))

Max Activating Token Index: 230

Text #6

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2051. Min Act: -0.0000

Data Index: 130454 (C4 (Web Text))

Max Activating Token Index: 507

Text #7

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2047. Min Act: -0.0000

Data Index: 1028075 (C4 (Web Text))

Max Activating Token Index: 202

Text #8

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2048. Min Act: -0.0000

Data Index: 10237 (C4 (Web Text))

Max Activating Token Index: 505

Text #9

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2092. Min Act: -0.0000

Data Index: 787698 (C4 (Web Text))

Max Activating Token Index: 404

Text #10

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2173. Min Act: -0.0000

Data Index: 635647 (C4 (Web Text))

Max Activating Token Index: 238

Text #11

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2060. Min Act: -0.0000

Data Index: 550288 (C4 (Web Text))

Max Activating Token Index: 572

Text #12

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2045. Min Act: -0.0000

Data Index: 438080 (C4 (Web Text))

Max Activating Token Index: 67

Text #13

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1974. Min Act: -0.0000

Data Index: 1055359 (C4 (Web Text))

Max Activating Token Index: 1022

Text #14

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.2031. Min Act: -0.0000

Data Index: 892783 (C4 (Web Text))

Max Activating Token Index: 848

Text #15

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1872. Min Act: -0.0000

Data Index: 241028 (C4 (Web Text))

Max Activating Token Index: 206

Text #16

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1770. Min Act: -0.0000

Data Index: 1268380 (C4 (Web Text))

Max Activating Token Index: 586

Text #17

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1859. Min Act: -0.0000

Data Index: 265796 (C4 (Web Text))

Max Activating Token Index: 990

Text #18

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1970. Min Act: -0.0000

Data Index: 548271 (C4 (Web Text))

Max Activating Token Index: 860

Text #19

Max Range: 0.2464. Min Range: -0.2464

Max Act: 0.1822. Min Act: 0.0000

Data Index: 880626 (C4 (Web Text))

Max Activating Token Index: 227