Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1648 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.4859. Min Act: -0.0000

Data Index: 883185 (C4 (Web Text))

Max Activating Token Index: 761

Text #1

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.3341. Min Act: -0.0000

Data Index: 1296784 (C4 (Web Text))

Max Activating Token Index: 441

Text #2

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.4707. Min Act: -0.0000

Data Index: 1239100 (C4 (Web Text))

Max Activating Token Index: 1014

Text #3

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.2788. Min Act: -0.0000

Data Index: 1104607 (C4 (Web Text))

Max Activating Token Index: 880

Text #4

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.1956. Min Act: -0.0000

Data Index: 404732 (C4 (Web Text))

Max Activating Token Index: 596

Text #5

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.2126. Min Act: -0.0000

Data Index: 199415 (C4 (Web Text))

Max Activating Token Index: 677

Text #6

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.0981. Min Act: 0.0000

Data Index: 531336 (C4 (Web Text))

Max Activating Token Index: 674

Text #7

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.1464. Min Act: -0.0000

Data Index: 799922 (C4 (Web Text))

Max Activating Token Index: 998

Text #8

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.2143. Min Act: -0.0000

Data Index: 464978 (C4 (Web Text))

Max Activating Token Index: 998

Text #9

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.9652. Min Act: -0.0000

Data Index: 335851 (C4 (Web Text))

Max Activating Token Index: 159

Text #10

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.0055. Min Act: -0.0000

Data Index: 1339679 (C4 (Web Text))

Max Activating Token Index: 663

Text #11

Max Range: 1.4859. Min Range: -1.4859

Max Act: 1.0161. Min Act: -0.0000

Data Index: 1025117 (C4 (Web Text))

Max Activating Token Index: 554

Text #12

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.9217. Min Act: -0.0000

Data Index: 1100771 (C4 (Web Text))

Max Activating Token Index: 825

Text #13

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.9415. Min Act: -0.0000

Data Index: 507460 (C4 (Web Text))

Max Activating Token Index: 143

Text #14

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8659. Min Act: -0.0000

Data Index: 1016507 (C4 (Web Text))

Max Activating Token Index: 114

Text #15

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8609. Min Act: -0.0000

Data Index: 525579 (C4 (Web Text))

Max Activating Token Index: 554

Text #16

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8521. Min Act: -0.0000

Data Index: 731390 (C4 (Web Text))

Max Activating Token Index: 721

Text #17

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8889. Min Act: -0.0000

Data Index: 592360 (C4 (Web Text))

Max Activating Token Index: 974

Text #18

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8101. Min Act: -0.0000

Data Index: 884215 (C4 (Web Text))

Max Activating Token Index: 814

Text #19

Max Range: 1.4859. Min Range: -1.4859

Max Act: 0.8227. Min Act: -0.0000

Data Index: 976474 (C4 (Web Text))

Max Activating Token Index: 914