Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 257 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0174. Min Range: -2.0174

Max Act: 2.0174. Min Act: -0.0000

Data Index: 97096 (C4 (Web Text))

Max Activating Token Index: 239

Text #1

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.7598. Min Act: -0.0000

Data Index: 453208 (C4 (Web Text))

Max Activating Token Index: 83

Text #2

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.6936. Min Act: -0.0000

Data Index: 726497 (C4 (Web Text))

Max Activating Token Index: 320

Text #3

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.6532. Min Act: -0.0000

Data Index: 823940 (C4 (Web Text))

Max Activating Token Index: 20

Text #4

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.7545. Min Act: -0.0000

Data Index: 204555 (C4 (Web Text))

Max Activating Token Index: 623

Text #5

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.5721. Min Act: -0.0000

Data Index: 432181 (C4 (Web Text))

Max Activating Token Index: 489

Text #6

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.5203. Min Act: -0.0000

Data Index: 157082 (C4 (Web Text))

Max Activating Token Index: 679

Text #7

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.5820. Min Act: -0.0000

Data Index: 720177 (C4 (Web Text))

Max Activating Token Index: 356

Text #8

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4941. Min Act: -0.0000

Data Index: 182987 (C4 (Web Text))

Max Activating Token Index: 454

Text #9

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4900. Min Act: -0.0000

Data Index: 814916 (C4 (Web Text))

Max Activating Token Index: 939

Text #10

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4772. Min Act: -0.0000

Data Index: 668491 (C4 (Web Text))

Max Activating Token Index: 234

Text #11

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.5042. Min Act: -0.0000

Data Index: 575651 (C4 (Web Text))

Max Activating Token Index: 886

Text #12

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4249. Min Act: -0.0000

Data Index: 193606 (C4 (Web Text))

Max Activating Token Index: 503

Text #13

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.3295. Min Act: -0.0000

Data Index: 1340390 (C4 (Web Text))

Max Activating Token Index: 170

Text #14

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.3241. Min Act: -0.0000

Data Index: 390115 (C4 (Web Text))

Max Activating Token Index: 945

Text #15

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.3785. Min Act: -0.0000

Data Index: 1184174 (C4 (Web Text))

Max Activating Token Index: 247

Text #16

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.3118. Min Act: -0.0000

Data Index: 551224 (C4 (Web Text))

Max Activating Token Index: 665

Text #17

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4685. Min Act: -0.0000

Data Index: 967557 (C4 (Web Text))

Max Activating Token Index: 451

Text #18

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.3006. Min Act: -0.0000

Data Index: 266519 (C4 (Web Text))

Max Activating Token Index: 270

Text #19

Max Range: 2.0174. Min Range: -2.0174

Max Act: 1.4611. Min Act: -0.0000

Data Index: 1171617 (C4 (Web Text))

Max Activating Token Index: 36