Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2210 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.6537. Min Act: -0.0002

Data Index: 340576 (C4 (Web Text))

Max Activating Token Index: 872

Text #1

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.6012. Min Act: -0.0002

Data Index: 989613 (C4 (Web Text))

Max Activating Token Index: 739

Text #2

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5964. Min Act: -0.0002

Data Index: 302249 (C4 (Web Text))

Max Activating Token Index: 638

Text #3

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5668. Min Act: -0.0002

Data Index: 783211 (C4 (Web Text))

Max Activating Token Index: 628

Text #4

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5548. Min Act: -0.0002

Data Index: 812631 (C4 (Web Text))

Max Activating Token Index: 263

Text #5

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5450. Min Act: -0.0002

Data Index: 25261 (C4 (Web Text))

Max Activating Token Index: 482

Text #6

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5341. Min Act: -0.0002

Data Index: 100277 (C4 (Web Text))

Max Activating Token Index: 502

Text #7

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5378. Min Act: -0.0002

Data Index: 795500 (C4 (Web Text))

Max Activating Token Index: 1008

Text #8

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5376. Min Act: -0.0002

Data Index: 268353 (C4 (Web Text))

Max Activating Token Index: 860

Text #9

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5294. Min Act: -0.0002

Data Index: 976818 (C4 (Web Text))

Max Activating Token Index: 79

Text #10

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5183. Min Act: -0.0002

Data Index: 370999 (C4 (Web Text))

Max Activating Token Index: 964

Text #11

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5171. Min Act: -0.0002

Data Index: 206771 (C4 (Web Text))

Max Activating Token Index: 27

Text #12

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5154. Min Act: -0.0002

Data Index: 323810 (C4 (Web Text))

Max Activating Token Index: 582

Text #13

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5239. Min Act: -0.0002

Data Index: 483825 (C4 (Web Text))

Max Activating Token Index: 494

Text #14

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5148. Min Act: -0.0002

Data Index: 677672 (C4 (Web Text))

Max Activating Token Index: 678

Text #15

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5150. Min Act: -0.0002

Data Index: 704368 (C4 (Web Text))

Max Activating Token Index: 24

Text #16

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5057. Min Act: -0.0002

Data Index: 428741 (C4 (Web Text))

Max Activating Token Index: 866

Text #17

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5055. Min Act: -0.0002

Data Index: 245762 (C4 (Web Text))

Max Activating Token Index: 653

Text #18

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.5094. Min Act: -0.0002

Data Index: 1016241 (C4 (Web Text))

Max Activating Token Index: 297

Text #19

Max Range: 0.6537. Min Range: -0.6537

Max Act: 0.4918. Min Act: -0.0002

Data Index: 1141070 (C4 (Web Text))

Max Activating Token Index: 1013