Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3725 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.9443. Min Act: -0.0000

Data Index: 593897 (C4 (Web Text))

Max Activating Token Index: 127

Text #1

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.9068. Min Act: -0.0000

Data Index: 217526 (C4 (Web Text))

Max Activating Token Index: 133

Text #2

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.8176. Min Act: -0.0000

Data Index: 290738 (C4 (Web Text))

Max Activating Token Index: 461

Text #3

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.6316. Min Act: -0.0000

Data Index: 799013 (C4 (Web Text))

Max Activating Token Index: 743

Text #4

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.6187. Min Act: -0.0000

Data Index: 196970 (C4 (Web Text))

Max Activating Token Index: 549

Text #5

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5800. Min Act: 0.0000

Data Index: 435632 (C4 (Web Text))

Max Activating Token Index: 29

Text #6

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5295. Min Act: -0.0000

Data Index: 762962 (C4 (Web Text))

Max Activating Token Index: 913

Text #7

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5734. Min Act: -0.0000

Data Index: 611256 (C4 (Web Text))

Max Activating Token Index: 242

Text #8

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5530. Min Act: -0.0000

Data Index: 104103 (C4 (Web Text))

Max Activating Token Index: 867

Text #9

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5166. Min Act: -0.0000

Data Index: 528104 (C4 (Web Text))

Max Activating Token Index: 488

Text #10

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5664. Min Act: -0.0000

Data Index: 242343 (C4 (Web Text))

Max Activating Token Index: 216

Text #11

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5007. Min Act: -0.0000

Data Index: 1115278 (C4 (Web Text))

Max Activating Token Index: 727

Text #12

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5269. Min Act: -0.0000

Data Index: 20609 (C4 (Web Text))

Max Activating Token Index: 706

Text #13

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5109. Min Act: -0.0000

Data Index: 1170220 (C4 (Web Text))

Max Activating Token Index: 840

Text #14

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5371. Min Act: 0.0000

Data Index: 1087934 (C4 (Web Text))

Max Activating Token Index: 469

Text #15

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5056. Min Act: 0.0000

Data Index: 244914 (C4 (Web Text))

Max Activating Token Index: 679

Text #16

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.4940. Min Act: -0.0000

Data Index: 1343933 (C4 (Web Text))

Max Activating Token Index: 672

Text #17

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.5034. Min Act: -0.0000

Data Index: 818336 (C4 (Web Text))

Max Activating Token Index: 660

Text #18

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.4822. Min Act: -0.0000

Data Index: 216218 (C4 (Web Text))

Max Activating Token Index: 203

Text #19

Max Range: 0.9443. Min Range: -0.9443

Max Act: 0.4975. Min Act: -0.0000

Data Index: 323804 (C4 (Web Text))

Max Activating Token Index: 788