Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3865 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.5632. Min Act: -0.0001

Data Index: 1122839 (C4 (Web Text))

Max Activating Token Index: 928

Text #1

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.5182. Min Act: -0.0001

Data Index: 584646 (C4 (Web Text))

Max Activating Token Index: 795

Text #2

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.5489. Min Act: -0.0001

Data Index: 291728 (C4 (Web Text))

Max Activating Token Index: 759

Text #3

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4719. Min Act: -0.0001

Data Index: 22988 (C4 (Web Text))

Max Activating Token Index: 929

Text #4

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.5015. Min Act: -0.0001

Data Index: 1248511 (C4 (Web Text))

Max Activating Token Index: 220

Text #5

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4831. Min Act: -0.0001

Data Index: 1320761 (C4 (Web Text))

Max Activating Token Index: 359

Text #6

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4496. Min Act: -0.0001

Data Index: 268797 (C4 (Web Text))

Max Activating Token Index: 115

Text #7

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.5001. Min Act: -0.0001

Data Index: 47114 (C4 (Web Text))

Max Activating Token Index: 712

Text #8

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4517. Min Act: -0.0001

Data Index: 209383 (C4 (Web Text))

Max Activating Token Index: 544

Text #9

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4257. Min Act: -0.0001

Data Index: 113527 (C4 (Web Text))

Max Activating Token Index: 1011

Text #10

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4403. Min Act: -0.0001

Data Index: 1047661 (C4 (Web Text))

Max Activating Token Index: 404

Text #11

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4628. Min Act: -0.0001

Data Index: 234863 (C4 (Web Text))

Max Activating Token Index: 655

Text #12

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.3985. Min Act: -0.0001

Data Index: 1146989 (C4 (Web Text))

Max Activating Token Index: 280

Text #13

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4247. Min Act: -0.0001

Data Index: 1107992 (C4 (Web Text))

Max Activating Token Index: 1019

Text #14

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4248. Min Act: -0.0001

Data Index: 1291996 (C4 (Web Text))

Max Activating Token Index: 859

Text #15

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.3938. Min Act: -0.0001

Data Index: 742378 (C4 (Web Text))

Max Activating Token Index: 715

Text #16

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.3811. Min Act: -0.0001

Data Index: 639961 (C4 (Web Text))

Max Activating Token Index: 771

Text #17

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.4142. Min Act: -0.0001

Data Index: 40188 (C4 (Web Text))

Max Activating Token Index: 974

Text #18

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.3936. Min Act: -0.0001

Data Index: 227075 (C4 (Web Text))

Max Activating Token Index: 920

Text #19

Max Range: 0.5632. Min Range: -0.5632

Max Act: 0.3914. Min Act: -0.0001

Data Index: 447810 (C4 (Web Text))

Max Activating Token Index: 201