Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1331 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5138. Min Range: -2.5138

Max Act: 2.5138. Min Act: -0.0000

Data Index: 128659 (C4 (Web Text))

Max Activating Token Index: 533

Text #1

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.7736. Min Act: -0.0000

Data Index: 240807 (C4 (Web Text))

Max Activating Token Index: 78

Text #2

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.8787. Min Act: -0.0000

Data Index: 759690 (C4 (Web Text))

Max Activating Token Index: 151

Text #3

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.7250. Min Act: -0.0001

Data Index: 709736 (C4 (Web Text))

Max Activating Token Index: 8

Text #4

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.7405. Min Act: -0.0000

Data Index: 498255 (C4 (Web Text))

Max Activating Token Index: 984

Text #5

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.5266. Min Act: -0.0000

Data Index: 18551 (C4 (Web Text))

Max Activating Token Index: 448

Text #6

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.6116. Min Act: -0.0000

Data Index: 379352 (C4 (Web Text))

Max Activating Token Index: 540

Text #7

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.5748. Min Act: -0.0000

Data Index: 755811 (C4 (Web Text))

Max Activating Token Index: 611

Text #8

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.4397. Min Act: -0.0000

Data Index: 156794 (C4 (Web Text))

Max Activating Token Index: 188

Text #9

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.4804. Min Act: -0.0000

Data Index: 814223 (C4 (Web Text))

Max Activating Token Index: 157

Text #10

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.3369. Min Act: -0.0000

Data Index: 906811 (C4 (Web Text))

Max Activating Token Index: 402

Text #11

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.6038. Min Act: -0.0000

Data Index: 259719 (C4 (Web Text))

Max Activating Token Index: 790

Text #12

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.4385. Min Act: -0.0001

Data Index: 121890 (C4 (Web Text))

Max Activating Token Index: 469

Text #13

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.6166. Min Act: -0.0000

Data Index: 747131 (C4 (Web Text))

Max Activating Token Index: 665

Text #14

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.2963. Min Act: -0.0001

Data Index: 949111 (C4 (Web Text))

Max Activating Token Index: 846

Text #15

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.4422. Min Act: -0.0001

Data Index: 754879 (C4 (Web Text))

Max Activating Token Index: 478

Text #16

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.2975. Min Act: -0.0000

Data Index: 391768 (C4 (Web Text))

Max Activating Token Index: 723

Text #17

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.0886. Min Act: -0.0000

Data Index: 376335 (C4 (Web Text))

Max Activating Token Index: 741

Text #18

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.5175. Min Act: -0.0001

Data Index: 1047900 (C4 (Web Text))

Max Activating Token Index: 1007

Text #19

Max Range: 2.5138. Min Range: -2.5138

Max Act: 1.3091. Min Act: -0.0001

Data Index: 476047 (C4 (Web Text))

Max Activating Token Index: 563