Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4810 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.6127. Min Act: -0.0000

Data Index: 289129 (C4 (Web Text))

Max Activating Token Index: 790

Text #1

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.5108. Min Act: -0.0000

Data Index: 603303 (C4 (Web Text))

Max Activating Token Index: 734

Text #2

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.5443. Min Act: -0.0000

Data Index: 319894 (C4 (Web Text))

Max Activating Token Index: 299

Text #3

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.6017. Min Act: -0.0000

Data Index: 224393 (C4 (Web Text))

Max Activating Token Index: 282

Text #4

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.3574. Min Act: -0.0000

Data Index: 876064 (C4 (Web Text))

Max Activating Token Index: 669

Text #5

Max Range: 2.6127. Min Range: -2.6127

Max Act: 2.0909. Min Act: -0.0000

Data Index: 1055579 (C4 (Web Text))

Max Activating Token Index: 447

Text #6

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.9182. Min Act: -0.0000

Data Index: 1197456 (C4 (Web Text))

Max Activating Token Index: 335

Text #7

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.7109. Min Act: -0.0000

Data Index: 1263948 (C4 (Web Text))

Max Activating Token Index: 751

Text #8

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.6954. Min Act: -0.0000

Data Index: 1140033 (C4 (Web Text))

Max Activating Token Index: 929

Text #9

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.6296. Min Act: -0.0000

Data Index: 172149 (C4 (Web Text))

Max Activating Token Index: 794

Text #10

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.3710. Min Act: -0.0000

Data Index: 625909 (C4 (Web Text))

Max Activating Token Index: 766

Text #11

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.4543. Min Act: -0.0000

Data Index: 143578 (C4 (Web Text))

Max Activating Token Index: 119

Text #12

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.4522. Min Act: 0.0000

Data Index: 1206713 (C4 (Web Text))

Max Activating Token Index: 148

Text #13

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.4398. Min Act: -0.0000

Data Index: 3439 (C4 (Web Text))

Max Activating Token Index: 421

Text #14

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.4137. Min Act: -0.0000

Data Index: 1178243 (C4 (Web Text))

Max Activating Token Index: 548

Text #15

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.2600. Min Act: -0.0000

Data Index: 475171 (C4 (Web Text))

Max Activating Token Index: 784

Text #16

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.2532. Min Act: 0.0000

Data Index: 1205368 (C4 (Web Text))

Max Activating Token Index: 825

Text #17

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.3470. Min Act: -0.0000

Data Index: 1286013 (C4 (Web Text))

Max Activating Token Index: 301

Text #18

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.1711. Min Act: -0.0000

Data Index: 613706 (C4 (Web Text))

Max Activating Token Index: 272

Text #19

Max Range: 2.6127. Min Range: -2.6127

Max Act: 1.3135. Min Act: -0.0000

Data Index: 591512 (C4 (Web Text))

Max Activating Token Index: 596