Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2455 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.0123. Min Range: -5.0123

Max Act: 5.0123. Min Act: -0.0000

Data Index: 148664 (C4 (Web Text))

Max Activating Token Index: 155

Text #1

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.7902. Min Act: -0.0000

Data Index: 179579 (C4 (Web Text))

Max Activating Token Index: 249

Text #2

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.3923. Min Act: -0.0000

Data Index: 362186 (C4 (Web Text))

Max Activating Token Index: 264

Text #3

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.6012. Min Act: -0.0000

Data Index: 462979 (C4 (Web Text))

Max Activating Token Index: 1004

Text #4

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.4676. Min Act: -0.0000

Data Index: 724525 (C4 (Web Text))

Max Activating Token Index: 92

Text #5

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.5758. Min Act: -0.0000

Data Index: 407832 (C4 (Web Text))

Max Activating Token Index: 880

Text #6

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.3375. Min Act: -0.0000

Data Index: 1082757 (C4 (Web Text))

Max Activating Token Index: 962

Text #7

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.1166. Min Act: 0.0000

Data Index: 896096 (C4 (Web Text))

Max Activating Token Index: 103

Text #8

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.0053. Min Act: -0.0000

Data Index: 1167407 (C4 (Web Text))

Max Activating Token Index: 946

Text #9

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.2269. Min Act: 0.0000

Data Index: 302779 (C4 (Web Text))

Max Activating Token Index: 89

Text #10

Max Range: 5.0123. Min Range: -5.0123

Max Act: 3.9748. Min Act: 0.0000

Data Index: 653938 (C4 (Web Text))

Max Activating Token Index: 732

Text #11

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.0000. Min Act: -0.0000

Data Index: 432324 (C4 (Web Text))

Max Activating Token Index: 777

Text #12

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.1301. Min Act: -0.0000

Data Index: 601479 (C4 (Web Text))

Max Activating Token Index: 981

Text #13

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.1172. Min Act: -0.0000

Data Index: 740917 (C4 (Web Text))

Max Activating Token Index: 990

Text #14

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.0280. Min Act: -0.0000

Data Index: 512283 (C4 (Web Text))

Max Activating Token Index: 968

Text #15

Max Range: 5.0123. Min Range: -5.0123

Max Act: 3.8567. Min Act: -0.0000

Data Index: 711988 (C4 (Web Text))

Max Activating Token Index: 685

Text #16

Max Range: 5.0123. Min Range: -5.0123

Max Act: 4.1509. Min Act: -0.0000

Data Index: 582793 (C4 (Web Text))

Max Activating Token Index: 456

Text #17

Max Range: 5.0123. Min Range: -5.0123

Max Act: 3.6812. Min Act: 0.0000

Data Index: 129063 (C4 (Web Text))

Max Activating Token Index: 195

Text #18

Max Range: 5.0123. Min Range: -5.0123

Max Act: 3.8685. Min Act: 0.0000

Data Index: 1197834 (C4 (Web Text))

Max Activating Token Index: 362

Text #19

Max Range: 5.0123. Min Range: -5.0123

Max Act: 3.3705. Min Act: 0.0000

Data Index: 1259649 (C4 (Web Text))

Max Activating Token Index: 768