Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4850 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2435. Min Range: -1.2435

Max Act: 1.2435. Min Act: -0.0000

Data Index: 34189 (C4 (Web Text))

Max Activating Token Index: 27

Text #1

Max Range: 1.2435. Min Range: -1.2435

Max Act: 1.0524. Min Act: -0.0000

Data Index: 923902 (C4 (Web Text))

Max Activating Token Index: 904

Text #2

Max Range: 1.2435. Min Range: -1.2435

Max Act: 1.0105. Min Act: -0.0000

Data Index: 367792 (C4 (Web Text))

Max Activating Token Index: 428

Text #3

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.8677. Min Act: -0.0000

Data Index: 754281 (C4 (Web Text))

Max Activating Token Index: 348

Text #4

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.8450. Min Act: -0.0000

Data Index: 677067 (C4 (Web Text))

Max Activating Token Index: 995

Text #5

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.8098. Min Act: -0.0000

Data Index: 339817 (C4 (Web Text))

Max Activating Token Index: 239

Text #6

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7975. Min Act: -0.0000

Data Index: 512717 (C4 (Web Text))

Max Activating Token Index: 970

Text #7

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.8028. Min Act: -0.0000

Data Index: 57445 (C4 (Web Text))

Max Activating Token Index: 201

Text #8

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7791. Min Act: -0.0000

Data Index: 731151 (C4 (Web Text))

Max Activating Token Index: 899

Text #9

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.8116. Min Act: -0.0000

Data Index: 22570 (C4 (Web Text))

Max Activating Token Index: 699

Text #10

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7749. Min Act: -0.0000

Data Index: 145382 (C4 (Web Text))

Max Activating Token Index: 786

Text #11

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7407. Min Act: -0.0000

Data Index: 747927 (C4 (Web Text))

Max Activating Token Index: 169

Text #12

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.6813. Min Act: -0.0000

Data Index: 200448 (C4 (Web Text))

Max Activating Token Index: 682

Text #13

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7712. Min Act: -0.0000

Data Index: 1165107 (C4 (Web Text))

Max Activating Token Index: 595

Text #14

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7233. Min Act: -0.0000

Data Index: 667576 (C4 (Web Text))

Max Activating Token Index: 384

Text #15

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7239. Min Act: -0.0000

Data Index: 17182 (C4 (Web Text))

Max Activating Token Index: 684

Text #16

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.7267. Min Act: -0.0000

Data Index: 1213414 (C4 (Web Text))

Max Activating Token Index: 542

Text #17

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.6905. Min Act: -0.0000

Data Index: 901670 (C4 (Web Text))

Max Activating Token Index: 423

Text #18

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.6859. Min Act: -0.0000

Data Index: 1336362 (C4 (Web Text))

Max Activating Token Index: 205

Text #19

Max Range: 1.2435. Min Range: -1.2435

Max Act: 0.6849. Min Act: -0.0000

Data Index: 1221553 (C4 (Web Text))

Max Activating Token Index: 368