Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 729 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.1206. Min Act: -0.0000

Data Index: 1035014 (C4 (Web Text))

Max Activating Token Index: 315

Text #1

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.0906. Min Act: -0.0000

Data Index: 124831 (C4 (Web Text))

Max Activating Token Index: 467

Text #2

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.1290. Min Act: -0.0000

Data Index: 867170 (C4 (Web Text))

Max Activating Token Index: 171

Text #3

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.1497. Min Act: -0.0000

Data Index: 902070 (C4 (Web Text))

Max Activating Token Index: 395

Text #4

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.0284. Min Act: -0.0000

Data Index: 371434 (C4 (Web Text))

Max Activating Token Index: 1007

Text #5

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.9866. Min Act: -0.0000

Data Index: 846321 (C4 (Web Text))

Max Activating Token Index: 831

Text #6

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.9947. Min Act: -0.0000

Data Index: 632601 (C4 (Web Text))

Max Activating Token Index: 182

Text #7

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.0097. Min Act: -0.0000

Data Index: 565480 (C4 (Web Text))

Max Activating Token Index: 51

Text #8

Max Range: 1.1497. Min Range: -1.1497

Max Act: 1.0001. Min Act: -0.0000

Data Index: 1113729 (C4 (Web Text))

Max Activating Token Index: 9

Text #9

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.9519. Min Act: -0.0000

Data Index: 82779 (C4 (Web Text))

Max Activating Token Index: 415

Text #10

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.8217. Min Act: -0.0000

Data Index: 510520 (C4 (Web Text))

Max Activating Token Index: 274

Text #11

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.8761. Min Act: -0.0000

Data Index: 432706 (C4 (Web Text))

Max Activating Token Index: 987

Text #12

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.8702. Min Act: -0.0000

Data Index: 166295 (C4 (Web Text))

Max Activating Token Index: 384

Text #13

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.8199. Min Act: -0.0000

Data Index: 1073427 (C4 (Web Text))

Max Activating Token Index: 61

Text #14

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.8124. Min Act: -0.0000

Data Index: 663634 (C4 (Web Text))

Max Activating Token Index: 87

Text #15

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.7679. Min Act: -0.0000

Data Index: 370443 (C4 (Web Text))

Max Activating Token Index: 488

Text #16

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.7874. Min Act: -0.0000

Data Index: 867500 (C4 (Web Text))

Max Activating Token Index: 893

Text #17

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.7729. Min Act: -0.0000

Data Index: 210780 (C4 (Web Text))

Max Activating Token Index: 649

Text #18

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.7448. Min Act: -0.0000

Data Index: 575546 (C4 (Web Text))

Max Activating Token Index: 515

Text #19

Max Range: 1.1497. Min Range: -1.1497

Max Act: 0.7706. Min Act: -0.0000

Data Index: 1163580 (C4 (Web Text))

Max Activating Token Index: 713