Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 547 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.8917. Min Act: -0.0000

Data Index: 321580 (C4 (Web Text))

Max Activating Token Index: 1007

Text #1

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.7191. Min Act: -0.0000

Data Index: 584447 (C4 (Web Text))

Max Activating Token Index: 62

Text #2

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.7268. Min Act: -0.0000

Data Index: 1255838 (C4 (Web Text))

Max Activating Token Index: 586

Text #3

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.4664. Min Act: -0.0000

Data Index: 1178222 (C4 (Web Text))

Max Activating Token Index: 251

Text #4

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.5427. Min Act: -0.0000

Data Index: 281824 (C4 (Web Text))

Max Activating Token Index: 106

Text #5

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.5438. Min Act: -0.0000

Data Index: 1232382 (C4 (Web Text))

Max Activating Token Index: 265

Text #6

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.3213. Min Act: -0.0000

Data Index: 301496 (C4 (Web Text))

Max Activating Token Index: 558

Text #7

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.3539. Min Act: -0.0000

Data Index: 1252471 (C4 (Web Text))

Max Activating Token Index: 329

Text #8

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.0344. Min Act: -0.0000

Data Index: 412086 (C4 (Web Text))

Max Activating Token Index: 304

Text #9

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.5706. Min Act: -0.0000

Data Index: 671081 (C4 (Web Text))

Max Activating Token Index: 293

Text #10

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.5208. Min Act: -0.0000

Data Index: 1305454 (C4 (Web Text))

Max Activating Token Index: 111

Text #11

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.2046. Min Act: -0.0000

Data Index: 621807 (C4 (Web Text))

Max Activating Token Index: 322

Text #12

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.3311. Min Act: -0.0000

Data Index: 702685 (C4 (Web Text))

Max Activating Token Index: 572

Text #13

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.3445. Min Act: -0.0000

Data Index: 1032608 (C4 (Web Text))

Max Activating Token Index: 898

Text #14

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.6279. Min Act: -0.0000

Data Index: 123488 (C4 (Web Text))

Max Activating Token Index: 221

Text #15

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.0876. Min Act: -0.0000

Data Index: 751121 (C4 (Web Text))

Max Activating Token Index: 466

Text #16

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.2104. Min Act: -0.0000

Data Index: 1049212 (C4 (Web Text))

Max Activating Token Index: 51

Text #17

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.3270. Min Act: -0.0000

Data Index: 532190 (C4 (Web Text))

Max Activating Token Index: 397

Text #18

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.1690. Min Act: -0.0000

Data Index: 1105222 (C4 (Web Text))

Max Activating Token Index: 56

Text #19

Max Range: 4.8917. Min Range: -4.8917

Max Act: 4.1893. Min Act: -0.0000

Data Index: 819719 (C4 (Web Text))

Max Activating Token Index: 109