Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3349 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.7588. Min Act: -0.0000

Data Index: 806043 (C4 (Web Text))

Max Activating Token Index: 234

Text #1

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.5019. Min Act: -0.0000

Data Index: 345640 (C4 (Web Text))

Max Activating Token Index: 758

Text #2

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.4592. Min Act: -0.0000

Data Index: 364765 (C4 (Web Text))

Max Activating Token Index: 144

Text #3

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.3500. Min Act: -0.0000

Data Index: 1138579 (C4 (Web Text))

Max Activating Token Index: 501

Text #4

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.1921. Min Act: -0.0000

Data Index: 99723 (C4 (Web Text))

Max Activating Token Index: 575

Text #5

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.1221. Min Act: -0.0000

Data Index: 1198959 (C4 (Web Text))

Max Activating Token Index: 251

Text #6

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.0030. Min Act: -0.0000

Data Index: 382776 (C4 (Web Text))

Max Activating Token Index: 452

Text #7

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.1196. Min Act: -0.0000

Data Index: 105211 (C4 (Web Text))

Max Activating Token Index: 556

Text #8

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.1664. Min Act: -0.0000

Data Index: 1270350 (C4 (Web Text))

Max Activating Token Index: 250

Text #9

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.8986. Min Act: -0.0000

Data Index: 100562 (C4 (Web Text))

Max Activating Token Index: 485

Text #10

Max Range: 3.7588. Min Range: -3.7588

Max Act: 3.0702. Min Act: -0.0000

Data Index: 397418 (C4 (Web Text))

Max Activating Token Index: 670

Text #11

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.8556. Min Act: -0.0000

Data Index: 129816 (C4 (Web Text))

Max Activating Token Index: 427

Text #12

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.8211. Min Act: -0.0000

Data Index: 1254720 (C4 (Web Text))

Max Activating Token Index: 855

Text #13

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.7220. Min Act: -0.0000

Data Index: 123306 (C4 (Web Text))

Max Activating Token Index: 952

Text #14

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.8513. Min Act: -0.0000

Data Index: 786231 (C4 (Web Text))

Max Activating Token Index: 740

Text #15

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.6728. Min Act: -0.0000

Data Index: 792496 (C4 (Web Text))

Max Activating Token Index: 764

Text #16

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.8134. Min Act: -0.0000

Data Index: 178668 (C4 (Web Text))

Max Activating Token Index: 414

Text #17

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.6226. Min Act: -0.0000

Data Index: 105921 (C4 (Web Text))

Max Activating Token Index: 612

Text #18

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.7505. Min Act: -0.0000

Data Index: 621420 (C4 (Web Text))

Max Activating Token Index: 879

Text #19

Max Range: 3.7588. Min Range: -3.7588

Max Act: 2.7508. Min Act: -0.0000

Data Index: 539266 (C4 (Web Text))

Max Activating Token Index: 13