Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1514 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0162. Min Range: -2.0162

Max Act: 2.0162. Min Act: -0.0000

Data Index: 1359455 (C4 (Web Text))

Max Activating Token Index: 732

Text #1

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8689. Min Act: -0.0000

Data Index: 1132597 (C4 (Web Text))

Max Activating Token Index: 668

Text #2

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8203. Min Act: -0.0000

Data Index: 831345 (C4 (Web Text))

Max Activating Token Index: 524

Text #3

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8783. Min Act: -0.0000

Data Index: 541186 (C4 (Web Text))

Max Activating Token Index: 542

Text #4

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.7851. Min Act: -0.0000

Data Index: 424247 (C4 (Web Text))

Max Activating Token Index: 548

Text #5

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8546. Min Act: -0.0000

Data Index: 111009 (C4 (Web Text))

Max Activating Token Index: 714

Text #6

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8625. Min Act: -0.0000

Data Index: 48987 (C4 (Web Text))

Max Activating Token Index: 773

Text #7

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.7307. Min Act: -0.0000

Data Index: 1277072 (C4 (Web Text))

Max Activating Token Index: 648

Text #8

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.7800. Min Act: -0.0000

Data Index: 812322 (C4 (Web Text))

Max Activating Token Index: 432

Text #9

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.7084. Min Act: -0.0000

Data Index: 909697 (C4 (Web Text))

Max Activating Token Index: 922

Text #10

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8054. Min Act: -0.0000

Data Index: 1227574 (C4 (Web Text))

Max Activating Token Index: 881

Text #11

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8551. Min Act: -0.0000

Data Index: 970009 (C4 (Web Text))

Max Activating Token Index: 665

Text #12

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.8168. Min Act: -0.0000

Data Index: 489346 (C4 (Web Text))

Max Activating Token Index: 596

Text #13

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.6848. Min Act: -0.0000

Data Index: 1339841 (C4 (Web Text))

Max Activating Token Index: 751

Text #14

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.5779. Min Act: -0.0000

Data Index: 1192447 (C4 (Web Text))

Max Activating Token Index: 272

Text #15

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.7395. Min Act: -0.0000

Data Index: 1073526 (C4 (Web Text))

Max Activating Token Index: 480

Text #16

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.6430. Min Act: -0.0000

Data Index: 1222823 (C4 (Web Text))

Max Activating Token Index: 820

Text #17

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.6156. Min Act: -0.0000

Data Index: 316283 (C4 (Web Text))

Max Activating Token Index: 553

Text #18

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.6962. Min Act: -0.0000

Data Index: 880145 (C4 (Web Text))

Max Activating Token Index: 852

Text #19

Max Range: 2.0162. Min Range: -2.0162

Max Act: 1.5307. Min Act: -0.0000

Data Index: 544450 (C4 (Web Text))

Max Activating Token Index: 205