Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 330 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.4475. Min Act: -0.0000

Data Index: 537557 (C4 (Web Text))

Max Activating Token Index: 789

Text #1

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.4240. Min Act: -0.0000

Data Index: 349602 (C4 (Web Text))

Max Activating Token Index: 540

Text #2

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3680. Min Act: -0.0000

Data Index: 1285661 (C4 (Web Text))

Max Activating Token Index: 138

Text #3

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3595. Min Act: -0.0000

Data Index: 293066 (C4 (Web Text))

Max Activating Token Index: 880

Text #4

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3344. Min Act: -0.0000

Data Index: 1110625 (C4 (Web Text))

Max Activating Token Index: 414

Text #5

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3087. Min Act: -0.0000

Data Index: 1026485 (C4 (Web Text))

Max Activating Token Index: 937

Text #6

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2966. Min Act: -0.0000

Data Index: 1247952 (C4 (Web Text))

Max Activating Token Index: 900

Text #7

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3130. Min Act: -0.0000

Data Index: 334736 (C4 (Web Text))

Max Activating Token Index: 967

Text #8

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.3088. Min Act: -0.0000

Data Index: 74402 (C4 (Web Text))

Max Activating Token Index: 956

Text #9

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2885. Min Act: -0.0000

Data Index: 597237 (C4 (Web Text))

Max Activating Token Index: 1020

Text #10

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2887. Min Act: -0.0000

Data Index: 17366 (C4 (Web Text))

Max Activating Token Index: 123

Text #11

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2803. Min Act: -0.0000

Data Index: 237585 (C4 (Web Text))

Max Activating Token Index: 202

Text #12

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2507. Min Act: -0.0000

Data Index: 930913 (C4 (Web Text))

Max Activating Token Index: 190

Text #13

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2394. Min Act: -0.0000

Data Index: 981842 (C4 (Web Text))

Max Activating Token Index: 383

Text #14

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2279. Min Act: -0.0000

Data Index: 172488 (C4 (Web Text))

Max Activating Token Index: 50

Text #15

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2306. Min Act: -0.0000

Data Index: 651613 (C4 (Web Text))

Max Activating Token Index: 964

Text #16

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2150. Min Act: -0.0000

Data Index: 1290385 (C4 (Web Text))

Max Activating Token Index: 328

Text #17

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2218. Min Act: -0.0000

Data Index: 1117298 (C4 (Web Text))

Max Activating Token Index: 291

Text #18

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2113. Min Act: -0.0000

Data Index: 409528 (C4 (Web Text))

Max Activating Token Index: 147

Text #19

Max Range: 0.4475. Min Range: -0.4475

Max Act: 0.2241. Min Act: -0.0000

Data Index: 1162210 (C4 (Web Text))

Max Activating Token Index: 566