Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4297 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.7738. Min Range: -4.7738

Max Act: 4.7738. Min Act: -0.0000

Data Index: 1147257 (C4 (Web Text))

Max Activating Token Index: 156

Text #1

Max Range: 4.7738. Min Range: -4.7738

Max Act: 4.4436. Min Act: -0.0000

Data Index: 1339820 (C4 (Web Text))

Max Activating Token Index: 587

Text #2

Max Range: 4.7738. Min Range: -4.7738

Max Act: 4.0331. Min Act: -0.0000

Data Index: 37187 (C4 (Web Text))

Max Activating Token Index: 295

Text #3

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.8613. Min Act: -0.0000

Data Index: 605977 (C4 (Web Text))

Max Activating Token Index: 777

Text #4

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.7592. Min Act: -0.0000

Data Index: 918325 (C4 (Web Text))

Max Activating Token Index: 177

Text #5

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.6082. Min Act: -0.0000

Data Index: 1001176 (C4 (Web Text))

Max Activating Token Index: 369

Text #6

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.5151. Min Act: -0.0000

Data Index: 1241040 (C4 (Web Text))

Max Activating Token Index: 812

Text #7

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.6639. Min Act: -0.0000

Data Index: 535658 (C4 (Web Text))

Max Activating Token Index: 950

Text #8

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.4872. Min Act: -0.0000

Data Index: 193331 (C4 (Web Text))

Max Activating Token Index: 956

Text #9

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.4275. Min Act: -0.0000

Data Index: 335281 (C4 (Web Text))

Max Activating Token Index: 788

Text #10

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.3202. Min Act: -0.0000

Data Index: 952711 (C4 (Web Text))

Max Activating Token Index: 642

Text #11

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.4642. Min Act: -0.0000

Data Index: 320749 (C4 (Web Text))

Max Activating Token Index: 730

Text #12

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.3858. Min Act: -0.0000

Data Index: 954409 (C4 (Web Text))

Max Activating Token Index: 637

Text #13

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.3476. Min Act: -0.0000

Data Index: 993409 (C4 (Web Text))

Max Activating Token Index: 156

Text #14

Max Range: 4.7738. Min Range: -4.7738

Max Act: 3.0145. Min Act: -0.0000

Data Index: 496707 (C4 (Web Text))

Max Activating Token Index: 504

Text #15

Max Range: 4.7738. Min Range: -4.7738

Max Act: 2.8570. Min Act: -0.0000

Data Index: 1265236 (C4 (Web Text))

Max Activating Token Index: 878

Text #16

Max Range: 4.7738. Min Range: -4.7738

Max Act: 2.9927. Min Act: -0.0000

Data Index: 1330652 (C4 (Web Text))

Max Activating Token Index: 463

Text #17

Max Range: 4.7738. Min Range: -4.7738

Max Act: 2.9832. Min Act: -0.0000

Data Index: 1291937 (C4 (Web Text))

Max Activating Token Index: 113

Text #18

Max Range: 4.7738. Min Range: -4.7738

Max Act: 2.8185. Min Act: -0.0000

Data Index: 801254 (C4 (Web Text))

Max Activating Token Index: 539

Text #19

Max Range: 4.7738. Min Range: -4.7738

Max Act: 2.9491. Min Act: -0.0000

Data Index: 568836 (C4 (Web Text))

Max Activating Token Index: 984