Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3566 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.6198. Min Act: -0.0000

Data Index: 889746 (C4 (Web Text))

Max Activating Token Index: 127

Text #1

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.7114. Min Act: -0.0000

Data Index: 680225 (C4 (Web Text))

Max Activating Token Index: 493

Text #2

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3719. Min Act: -0.0000

Data Index: 840542 (C4 (Web Text))

Max Activating Token Index: 492

Text #3

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3606. Min Act: -0.0000

Data Index: 1319268 (C4 (Web Text))

Max Activating Token Index: 933

Text #4

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.6353. Min Act: -0.0000

Data Index: 565567 (C4 (Web Text))

Max Activating Token Index: 186

Text #5

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.4931. Min Act: -0.0000

Data Index: 350755 (C4 (Web Text))

Max Activating Token Index: 184

Text #6

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3058. Min Act: -0.0000

Data Index: 1220139 (C4 (Web Text))

Max Activating Token Index: 374

Text #7

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3496. Min Act: -0.0000

Data Index: 553376 (C4 (Web Text))

Max Activating Token Index: 562

Text #8

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3698. Min Act: -0.0000

Data Index: 216551 (C4 (Web Text))

Max Activating Token Index: 967

Text #9

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.0409. Min Act: -0.0000

Data Index: 1209286 (C4 (Web Text))

Max Activating Token Index: 78

Text #10

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3244. Min Act: -0.0000

Data Index: 1189337 (C4 (Web Text))

Max Activating Token Index: 85

Text #11

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3354. Min Act: -0.0000

Data Index: 835647 (C4 (Web Text))

Max Activating Token Index: 140

Text #12

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3015. Min Act: -0.0000

Data Index: 852646 (C4 (Web Text))

Max Activating Token Index: 945

Text #13

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.4407. Min Act: -0.0000

Data Index: 651604 (C4 (Web Text))

Max Activating Token Index: 90

Text #14

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.4449. Min Act: -0.0000

Data Index: 43651 (C4 (Web Text))

Max Activating Token Index: 185

Text #15

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3610. Min Act: -0.0000

Data Index: 896636 (C4 (Web Text))

Max Activating Token Index: 661

Text #16

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.1981. Min Act: -0.0000

Data Index: 1207084 (C4 (Web Text))

Max Activating Token Index: 814

Text #17

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.0339. Min Act: -0.0000

Data Index: 440172 (C4 (Web Text))

Max Activating Token Index: 465

Text #18

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.3434. Min Act: -0.0000

Data Index: 1315401 (C4 (Web Text))

Max Activating Token Index: 444

Text #19

Max Range: 3.7114. Min Range: -3.7114

Max Act: 3.1661. Min Act: -0.0000

Data Index: 1002553 (C4 (Web Text))

Max Activating Token Index: 158