Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5790 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.3299. Min Act: -0.0000

Data Index: 1530925 (Python Code)

Max Activating Token Index: 474

Text #1

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.4165. Min Act: -0.0000

Data Index: 1511100 (Python Code)

Max Activating Token Index: 708

Text #2

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.4160. Min Act: -0.0000

Data Index: 229610 (C4 (Web Text))

Max Activating Token Index: 662

Text #3

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.3018. Min Act: -0.0000

Data Index: 1053495 (C4 (Web Text))

Max Activating Token Index: 619

Text #4

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.1703. Min Act: -0.0000

Data Index: 307827 (C4 (Web Text))

Max Activating Token Index: 569

Text #5

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.1389. Min Act: -0.0000

Data Index: 758136 (C4 (Web Text))

Max Activating Token Index: 871

Text #6

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.1134. Min Act: -0.0000

Data Index: 418433 (C4 (Web Text))

Max Activating Token Index: 23

Text #7

Max Range: 3.4165. Min Range: -3.4165

Max Act: 3.1259. Min Act: -0.0000

Data Index: 104910 (C4 (Web Text))

Max Activating Token Index: 124

Text #8

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.9407. Min Act: -0.0000

Data Index: 598784 (C4 (Web Text))

Max Activating Token Index: 533

Text #9

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.8639. Min Act: -0.0000

Data Index: 649853 (C4 (Web Text))

Max Activating Token Index: 823

Text #10

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.9349. Min Act: -0.0000

Data Index: 1264802 (C4 (Web Text))

Max Activating Token Index: 830

Text #11

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.7094. Min Act: -0.0000

Data Index: 1351339 (C4 (Web Text))

Max Activating Token Index: 496

Text #12

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.6315. Min Act: -0.0000

Data Index: 1646491 (Python Code)

Max Activating Token Index: 87

Text #13

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.6100. Min Act: -0.0000

Data Index: 776306 (C4 (Web Text))

Max Activating Token Index: 942

Text #14

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.6057. Min Act: -0.0000

Data Index: 607916 (C4 (Web Text))

Max Activating Token Index: 625

Text #15

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.7323. Min Act: -0.0000

Data Index: 516633 (C4 (Web Text))

Max Activating Token Index: 475

Text #16

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.4915. Min Act: -0.0000

Data Index: 1311634 (C4 (Web Text))

Max Activating Token Index: 923

Text #17

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.5953. Min Act: -0.0000

Data Index: 112527 (C4 (Web Text))

Max Activating Token Index: 46

Text #18

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.5804. Min Act: -0.0000

Data Index: 1253922 (C4 (Web Text))

Max Activating Token Index: 895

Text #19

Max Range: 3.4165. Min Range: -3.4165

Max Act: 2.5746. Min Act: -0.0000

Data Index: 1257720 (C4 (Web Text))

Max Activating Token Index: 23