Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2127 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.4126. Min Range: -3.4126

Max Act: 3.3379. Min Act: -0.0000

Data Index: 992915 (C4 (Web Text))

Max Activating Token Index: 635

Text #1

Max Range: 3.4126. Min Range: -3.4126

Max Act: 3.2474. Min Act: -0.0000

Data Index: 203952 (C4 (Web Text))

Max Activating Token Index: 185

Text #2

Max Range: 3.4126. Min Range: -3.4126

Max Act: 3.4126. Min Act: -0.0000

Data Index: 1301381 (C4 (Web Text))

Max Activating Token Index: 272

Text #3

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.7537. Min Act: -0.0000

Data Index: 961775 (C4 (Web Text))

Max Activating Token Index: 959

Text #4

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.7185. Min Act: -0.0000

Data Index: 1505413 (Python Code)

Max Activating Token Index: 98

Text #5

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.7576. Min Act: -0.0000

Data Index: 334158 (C4 (Web Text))

Max Activating Token Index: 328

Text #6

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.5587. Min Act: -0.0000

Data Index: 89339 (C4 (Web Text))

Max Activating Token Index: 946

Text #7

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.6510. Min Act: -0.0000

Data Index: 567799 (C4 (Web Text))

Max Activating Token Index: 842

Text #8

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.2551. Min Act: -0.0000

Data Index: 7695 (C4 (Web Text))

Max Activating Token Index: 914

Text #9

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.4989. Min Act: -0.0000

Data Index: 604549 (C4 (Web Text))

Max Activating Token Index: 789

Text #10

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.6048. Min Act: -0.0000

Data Index: 1483246 (Python Code)

Max Activating Token Index: 134

Text #11

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.5998. Min Act: -0.0000

Data Index: 1284622 (C4 (Web Text))

Max Activating Token Index: 154

Text #12

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.4753. Min Act: -0.0000

Data Index: 846098 (C4 (Web Text))

Max Activating Token Index: 814

Text #13

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.4402. Min Act: -0.0000

Data Index: 1316472 (C4 (Web Text))

Max Activating Token Index: 144

Text #14

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.5283. Min Act: -0.0000

Data Index: 954823 (C4 (Web Text))

Max Activating Token Index: 546

Text #15

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.3975. Min Act: -0.0000

Data Index: 1203918 (C4 (Web Text))

Max Activating Token Index: 142

Text #16

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.3629. Min Act: -0.0000

Data Index: 167041 (C4 (Web Text))

Max Activating Token Index: 252

Text #17

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.3333. Min Act: -0.0000

Data Index: 1448036 (Python Code)

Max Activating Token Index: 61

Text #18

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.3436. Min Act: -0.0000

Data Index: 1031662 (C4 (Web Text))

Max Activating Token Index: 537

Text #19

Max Range: 3.4126. Min Range: -3.4126

Max Act: 2.0055. Min Act: -0.0000

Data Index: 580712 (C4 (Web Text))

Max Activating Token Index: 125