Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3488 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.4077. Min Act: -0.0000

Data Index: 81208 (C4 (Web Text))

Max Activating Token Index: 994

Text #1

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3512. Min Act: -0.0000

Data Index: 137511 (C4 (Web Text))

Max Activating Token Index: 66

Text #2

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3485. Min Act: -0.0000

Data Index: 598973 (C4 (Web Text))

Max Activating Token Index: 297

Text #3

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3846. Min Act: -0.0000

Data Index: 79535 (C4 (Web Text))

Max Activating Token Index: 762

Text #4

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3266. Min Act: -0.0000

Data Index: 359721 (C4 (Web Text))

Max Activating Token Index: 864

Text #5

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3280. Min Act: -0.0000

Data Index: 312646 (C4 (Web Text))

Max Activating Token Index: 785

Text #6

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3172. Min Act: -0.0000

Data Index: 141766 (C4 (Web Text))

Max Activating Token Index: 375

Text #7

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3126. Min Act: -0.0000

Data Index: 71519 (C4 (Web Text))

Max Activating Token Index: 258

Text #8

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3078. Min Act: -0.0000

Data Index: 1273416 (C4 (Web Text))

Max Activating Token Index: 290

Text #9

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3041. Min Act: -0.0000

Data Index: 674792 (C4 (Web Text))

Max Activating Token Index: 456

Text #10

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3046. Min Act: -0.0000

Data Index: 389898 (C4 (Web Text))

Max Activating Token Index: 172

Text #11

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2993. Min Act: -0.0000

Data Index: 838126 (C4 (Web Text))

Max Activating Token Index: 175

Text #12

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3178. Min Act: -0.0000

Data Index: 315958 (C4 (Web Text))

Max Activating Token Index: 895

Text #13

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2961. Min Act: -0.0000

Data Index: 1255060 (C4 (Web Text))

Max Activating Token Index: 467

Text #14

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.3043. Min Act: -0.0000

Data Index: 625160 (C4 (Web Text))

Max Activating Token Index: 99

Text #15

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2797. Min Act: -0.0000

Data Index: 59849 (C4 (Web Text))

Max Activating Token Index: 601

Text #16

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2950. Min Act: -0.0000

Data Index: 535740 (C4 (Web Text))

Max Activating Token Index: 962

Text #17

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2623. Min Act: -0.0000

Data Index: 721803 (C4 (Web Text))

Max Activating Token Index: 216

Text #18

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2799. Min Act: -0.0000

Data Index: 202630 (C4 (Web Text))

Max Activating Token Index: 974

Text #19

Max Range: 0.4077. Min Range: -0.4077

Max Act: 0.2807. Min Act: -0.0000

Data Index: 105111 (C4 (Web Text))

Max Activating Token Index: 857