Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2826 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.6661. Min Act: -0.0000

Data Index: 513535 (C4 (Web Text))

Max Activating Token Index: 915

Text #1

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.6217. Min Act: -0.0000

Data Index: 358096 (C4 (Web Text))

Max Activating Token Index: 26

Text #2

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.6108. Min Act: -0.0000

Data Index: 480960 (C4 (Web Text))

Max Activating Token Index: 787

Text #3

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5817. Min Act: -0.0000

Data Index: 807982 (C4 (Web Text))

Max Activating Token Index: 77

Text #4

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5497. Min Act: -0.0000

Data Index: 241442 (C4 (Web Text))

Max Activating Token Index: 654

Text #5

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5919. Min Act: -0.0000

Data Index: 903403 (C4 (Web Text))

Max Activating Token Index: 74

Text #6

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5700. Min Act: -0.0000

Data Index: 936418 (C4 (Web Text))

Max Activating Token Index: 521

Text #7

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5556. Min Act: -0.0000

Data Index: 708839 (C4 (Web Text))

Max Activating Token Index: 45

Text #8

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5705. Min Act: -0.0000

Data Index: 572412 (C4 (Web Text))

Max Activating Token Index: 21

Text #9

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5713. Min Act: -0.0000

Data Index: 1334398 (C4 (Web Text))

Max Activating Token Index: 7

Text #10

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5433. Min Act: -0.0000

Data Index: 836027 (C4 (Web Text))

Max Activating Token Index: 63

Text #11

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5392. Min Act: -0.0000

Data Index: 1309597 (C4 (Web Text))

Max Activating Token Index: 82

Text #12

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5683. Min Act: -0.0000

Data Index: 825064 (C4 (Web Text))

Max Activating Token Index: 769

Text #13

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5149. Min Act: -0.0000

Data Index: 382859 (C4 (Web Text))

Max Activating Token Index: 211

Text #14

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.4956. Min Act: -0.0000

Data Index: 159845 (C4 (Web Text))

Max Activating Token Index: 834

Text #15

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5050. Min Act: -0.0000

Data Index: 877074 (C4 (Web Text))

Max Activating Token Index: 717

Text #16

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5163. Min Act: -0.0000

Data Index: 736607 (C4 (Web Text))

Max Activating Token Index: 894

Text #17

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.4988. Min Act: -0.0000

Data Index: 531664 (C4 (Web Text))

Max Activating Token Index: 756

Text #18

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.4852. Min Act: -0.0000

Data Index: 280673 (C4 (Web Text))

Max Activating Token Index: 27

Text #19

Max Range: 0.6661. Min Range: -0.6661

Max Act: 0.5147. Min Act: -0.0000

Data Index: 325178 (C4 (Web Text))

Max Activating Token Index: 275