Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5604 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.7653. Min Range: -4.7653

Max Act: 4.7653. Min Act: -0.0001

Data Index: 931295 (C4 (Web Text))

Max Activating Token Index: 605

Text #1

Max Range: 4.7653. Min Range: -4.7653

Max Act: 4.1869. Min Act: -0.0000

Data Index: 1300051 (C4 (Web Text))

Max Activating Token Index: 43

Text #2

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.9359. Min Act: -0.0000

Data Index: 646292 (C4 (Web Text))

Max Activating Token Index: 474

Text #3

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.7884. Min Act: -0.0001

Data Index: 1075078 (C4 (Web Text))

Max Activating Token Index: 904

Text #4

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.6173. Min Act: -0.0000

Data Index: 1352692 (C4 (Web Text))

Max Activating Token Index: 192

Text #5

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.6047. Min Act: -0.0001

Data Index: 1018023 (C4 (Web Text))

Max Activating Token Index: 293

Text #6

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.2349. Min Act: -0.0001

Data Index: 370609 (C4 (Web Text))

Max Activating Token Index: 90

Text #7

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.3801. Min Act: -0.0000

Data Index: 350025 (C4 (Web Text))

Max Activating Token Index: 130

Text #8

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.3098. Min Act: -0.0000

Data Index: 792340 (C4 (Web Text))

Max Activating Token Index: 27

Text #9

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.4043. Min Act: -0.0001

Data Index: 636638 (C4 (Web Text))

Max Activating Token Index: 832

Text #10

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.3939. Min Act: -0.0000

Data Index: 604710 (C4 (Web Text))

Max Activating Token Index: 183

Text #11

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.2486. Min Act: -0.0001

Data Index: 1297978 (C4 (Web Text))

Max Activating Token Index: 190

Text #12

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.3439. Min Act: -0.0001

Data Index: 403537 (C4 (Web Text))

Max Activating Token Index: 402

Text #13

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.3155. Min Act: -0.0001

Data Index: 1304711 (C4 (Web Text))

Max Activating Token Index: 364

Text #14

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.0032. Min Act: -0.0000

Data Index: 370680 (C4 (Web Text))

Max Activating Token Index: 940

Text #15

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.2802. Min Act: -0.0001

Data Index: 176195 (C4 (Web Text))

Max Activating Token Index: 712

Text #16

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.1287. Min Act: -0.0000

Data Index: 549707 (C4 (Web Text))

Max Activating Token Index: 56

Text #17

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.1036. Min Act: -0.0000

Data Index: 1093863 (C4 (Web Text))

Max Activating Token Index: 684

Text #18

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.1101. Min Act: -0.0000

Data Index: 886728 (C4 (Web Text))

Max Activating Token Index: 354

Text #19

Max Range: 4.7653. Min Range: -4.7653

Max Act: 3.0984. Min Act: -0.0001

Data Index: 848720 (C4 (Web Text))

Max Activating Token Index: 443