Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4804 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.7677. Min Act: -0.0000

Data Index: 709322 (C4 (Web Text))

Max Activating Token Index: 756

Text #1

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.6819. Min Act: -0.0000

Data Index: 55810 (C4 (Web Text))

Max Activating Token Index: 32

Text #2

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.6952. Min Act: -0.0000

Data Index: 121331 (C4 (Web Text))

Max Activating Token Index: 199

Text #3

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.6341. Min Act: -0.0000

Data Index: 79602 (C4 (Web Text))

Max Activating Token Index: 897

Text #4

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.6365. Min Act: -0.0000

Data Index: 1043871 (C4 (Web Text))

Max Activating Token Index: 27

Text #5

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.6174. Min Act: -0.0000

Data Index: 1077109 (C4 (Web Text))

Max Activating Token Index: 890

Text #6

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5931. Min Act: -0.0000

Data Index: 246034 (C4 (Web Text))

Max Activating Token Index: 693

Text #7

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5623. Min Act: -0.0000

Data Index: 1126332 (C4 (Web Text))

Max Activating Token Index: 702

Text #8

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5262. Min Act: -0.0000

Data Index: 1037589 (C4 (Web Text))

Max Activating Token Index: 100

Text #9

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5641. Min Act: -0.0000

Data Index: 567994 (C4 (Web Text))

Max Activating Token Index: 683

Text #10

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5366. Min Act: -0.0000

Data Index: 271437 (C4 (Web Text))

Max Activating Token Index: 816

Text #11

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5492. Min Act: -0.0000

Data Index: 391615 (C4 (Web Text))

Max Activating Token Index: 852

Text #12

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5077. Min Act: -0.0000

Data Index: 578811 (C4 (Web Text))

Max Activating Token Index: 626

Text #13

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5366. Min Act: -0.0000

Data Index: 1294904 (C4 (Web Text))

Max Activating Token Index: 241

Text #14

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5189. Min Act: -0.0000

Data Index: 1342469 (C4 (Web Text))

Max Activating Token Index: 328

Text #15

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5654. Min Act: -0.0000

Data Index: 727691 (C4 (Web Text))

Max Activating Token Index: 178

Text #16

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5320. Min Act: -0.0000

Data Index: 1044059 (C4 (Web Text))

Max Activating Token Index: 271

Text #17

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5350. Min Act: -0.0000

Data Index: 1048992 (C4 (Web Text))

Max Activating Token Index: 437

Text #18

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5350. Min Act: -0.0000

Data Index: 1108585 (C4 (Web Text))

Max Activating Token Index: 213

Text #19

Max Range: 0.7677. Min Range: -0.7677

Max Act: 0.5424. Min Act: -0.0000

Data Index: 820498 (C4 (Web Text))

Max Activating Token Index: 272