Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1579 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.4020. Min Act: -0.0000

Data Index: 1309045 (C4 (Web Text))

Max Activating Token Index: 276

Text #1

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3937. Min Act: -0.0000

Data Index: 770937 (C4 (Web Text))

Max Activating Token Index: 31

Text #2

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.4210. Min Act: -0.0000

Data Index: 864891 (C4 (Web Text))

Max Activating Token Index: 67

Text #3

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3529. Min Act: -0.0000

Data Index: 1296203 (C4 (Web Text))

Max Activating Token Index: 530

Text #4

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3343. Min Act: -0.0000

Data Index: 1341553 (C4 (Web Text))

Max Activating Token Index: 345

Text #5

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3245. Min Act: -0.0000

Data Index: 776174 (C4 (Web Text))

Max Activating Token Index: 92

Text #6

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3201. Min Act: -0.0000

Data Index: 1134019 (C4 (Web Text))

Max Activating Token Index: 45

Text #7

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3300. Min Act: -0.0000

Data Index: 710784 (C4 (Web Text))

Max Activating Token Index: 450

Text #8

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3099. Min Act: -0.0000

Data Index: 278914 (C4 (Web Text))

Max Activating Token Index: 241

Text #9

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3124. Min Act: -0.0000

Data Index: 1266867 (C4 (Web Text))

Max Activating Token Index: 85

Text #10

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.3118. Min Act: -0.0000

Data Index: 648685 (C4 (Web Text))

Max Activating Token Index: 747

Text #11

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2901. Min Act: -0.0000

Data Index: 563604 (C4 (Web Text))

Max Activating Token Index: 19

Text #12

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2928. Min Act: -0.0000

Data Index: 922649 (C4 (Web Text))

Max Activating Token Index: 8

Text #13

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2846. Min Act: -0.0000

Data Index: 647331 (C4 (Web Text))

Max Activating Token Index: 101

Text #14

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2804. Min Act: -0.0000

Data Index: 374096 (C4 (Web Text))

Max Activating Token Index: 848

Text #15

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2732. Min Act: -0.0000

Data Index: 1327597 (C4 (Web Text))

Max Activating Token Index: 892

Text #16

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2718. Min Act: -0.0000

Data Index: 1184697 (C4 (Web Text))

Max Activating Token Index: 27

Text #17

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2767. Min Act: -0.0000

Data Index: 754941 (C4 (Web Text))

Max Activating Token Index: 598

Text #18

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2844. Min Act: -0.0000

Data Index: 1334950 (C4 (Web Text))

Max Activating Token Index: 800

Text #19

Max Range: 0.4210. Min Range: -0.4210

Max Act: 0.2805. Min Act: -0.0000

Data Index: 1102272 (C4 (Web Text))

Max Activating Token Index: 524