Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1183 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.4358. Min Act: -0.0000

Data Index: 294662 (C4 (Web Text))

Max Activating Token Index: 289

Text #1

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.6474. Min Act: -0.0000

Data Index: 1066417 (C4 (Web Text))

Max Activating Token Index: 258

Text #2

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.6176. Min Act: -0.0000

Data Index: 517162 (C4 (Web Text))

Max Activating Token Index: 428

Text #3

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.7821. Min Act: -0.0000

Data Index: 299228 (C4 (Web Text))

Max Activating Token Index: 254

Text #4

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.4801. Min Act: -0.0000

Data Index: 835785 (C4 (Web Text))

Max Activating Token Index: 22

Text #5

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.4629. Min Act: -0.0000

Data Index: 20675 (C4 (Web Text))

Max Activating Token Index: 129

Text #6

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.2270. Min Act: -0.0000

Data Index: 227099 (C4 (Web Text))

Max Activating Token Index: 158

Text #7

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.3841. Min Act: -0.0000

Data Index: 916848 (C4 (Web Text))

Max Activating Token Index: 750

Text #8

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.5621. Min Act: -0.0000

Data Index: 256156 (C4 (Web Text))

Max Activating Token Index: 271

Text #9

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.5472. Min Act: -0.0000

Data Index: 172680 (C4 (Web Text))

Max Activating Token Index: 924

Text #10

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.1171. Min Act: -0.0000

Data Index: 1140200 (C4 (Web Text))

Max Activating Token Index: 139

Text #11

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.4782. Min Act: -0.0000

Data Index: 820315 (C4 (Web Text))

Max Activating Token Index: 170

Text #12

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.2722. Min Act: -0.0000

Data Index: 432359 (C4 (Web Text))

Max Activating Token Index: 902

Text #13

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.2812. Min Act: -0.0000

Data Index: 533965 (C4 (Web Text))

Max Activating Token Index: 228

Text #14

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.2422. Min Act: -0.0000

Data Index: 1004445 (C4 (Web Text))

Max Activating Token Index: 967

Text #15

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.0136. Min Act: -0.0000

Data Index: 792524 (C4 (Web Text))

Max Activating Token Index: 966

Text #16

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.1103. Min Act: -0.0000

Data Index: 53851 (C4 (Web Text))

Max Activating Token Index: 1019

Text #17

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.0743. Min Act: -0.0000

Data Index: 482253 (C4 (Web Text))

Max Activating Token Index: 1015

Text #18

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.1497. Min Act: -0.0000

Data Index: 721671 (C4 (Web Text))

Max Activating Token Index: 625

Text #19

Max Range: 5.7821. Min Range: -5.7821

Max Act: 5.0994. Min Act: -0.0000

Data Index: 574229 (C4 (Web Text))

Max Activating Token Index: 451