Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3372 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.9821. Min Act: -0.0000

Data Index: 1041779 (C4 (Web Text))

Max Activating Token Index: 655

Text #1

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.9768. Min Act: -0.0000

Data Index: 1198055 (C4 (Web Text))

Max Activating Token Index: 861

Text #2

Max Range: 3.0788. Min Range: -3.0788

Max Act: 3.0788. Min Act: -0.0000

Data Index: 675675 (C4 (Web Text))

Max Activating Token Index: 959

Text #3

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.9239. Min Act: -0.0000

Data Index: 1180097 (C4 (Web Text))

Max Activating Token Index: 933

Text #4

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.8722. Min Act: -0.0000

Data Index: 1204828 (C4 (Web Text))

Max Activating Token Index: 147

Text #5

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.8594. Min Act: -0.0000

Data Index: 464904 (C4 (Web Text))

Max Activating Token Index: 662

Text #6

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.8301. Min Act: -0.0000

Data Index: 633967 (C4 (Web Text))

Max Activating Token Index: 608

Text #7

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.8187. Min Act: -0.0000

Data Index: 617820 (C4 (Web Text))

Max Activating Token Index: 409

Text #8

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.9596. Min Act: -0.0000

Data Index: 1121910 (C4 (Web Text))

Max Activating Token Index: 240

Text #9

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.8092. Min Act: -0.0000

Data Index: 775771 (C4 (Web Text))

Max Activating Token Index: 750

Text #10

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7891. Min Act: -0.0000

Data Index: 857507 (C4 (Web Text))

Max Activating Token Index: 899

Text #11

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7959. Min Act: -0.0000

Data Index: 1003678 (C4 (Web Text))

Max Activating Token Index: 195

Text #12

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7841. Min Act: -0.0000

Data Index: 540371 (C4 (Web Text))

Max Activating Token Index: 309

Text #13

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7545. Min Act: -0.0000

Data Index: 282247 (C4 (Web Text))

Max Activating Token Index: 677

Text #14

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7586. Min Act: -0.0000

Data Index: 1199293 (C4 (Web Text))

Max Activating Token Index: 472

Text #15

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7485. Min Act: -0.0000

Data Index: 683452 (C4 (Web Text))

Max Activating Token Index: 627

Text #16

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7553. Min Act: -0.0000

Data Index: 1042852 (C4 (Web Text))

Max Activating Token Index: 265

Text #17

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7536. Min Act: -0.0000

Data Index: 558800 (C4 (Web Text))

Max Activating Token Index: 859

Text #18

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7323. Min Act: -0.0000

Data Index: 35991 (C4 (Web Text))

Max Activating Token Index: 964

Text #19

Max Range: 3.0788. Min Range: -3.0788

Max Act: 2.7336. Min Act: -0.0000

Data Index: 965224 (C4 (Web Text))

Max Activating Token Index: 324