Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3451 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.4758. Min Act: -0.0000

Data Index: 324354 (C4 (Web Text))

Max Activating Token Index: 238

Text #1

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.3245. Min Act: -0.0000

Data Index: 901451 (C4 (Web Text))

Max Activating Token Index: 516

Text #2

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2781. Min Act: -0.0000

Data Index: 301481 (C4 (Web Text))

Max Activating Token Index: 557

Text #3

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2943. Min Act: -0.0000

Data Index: 1152705 (C4 (Web Text))

Max Activating Token Index: 893

Text #4

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2861. Min Act: -0.0000

Data Index: 686663 (C4 (Web Text))

Max Activating Token Index: 785

Text #5

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2948. Min Act: -0.0000

Data Index: 639029 (C4 (Web Text))

Max Activating Token Index: 220

Text #6

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2816. Min Act: -0.0000

Data Index: 583203 (C4 (Web Text))

Max Activating Token Index: 174

Text #7

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2726. Min Act: -0.0000

Data Index: 290837 (C4 (Web Text))

Max Activating Token Index: 863

Text #8

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2615. Min Act: -0.0000

Data Index: 1169707 (C4 (Web Text))

Max Activating Token Index: 516

Text #9

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2750. Min Act: -0.0000

Data Index: 259682 (C4 (Web Text))

Max Activating Token Index: 286

Text #10

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2622. Min Act: -0.0000

Data Index: 1214997 (C4 (Web Text))

Max Activating Token Index: 71

Text #11

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2423. Min Act: -0.0000

Data Index: 902776 (C4 (Web Text))

Max Activating Token Index: 936

Text #12

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2659. Min Act: -0.0000

Data Index: 577513 (C4 (Web Text))

Max Activating Token Index: 731

Text #13

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2549. Min Act: -0.0000

Data Index: 1340649 (C4 (Web Text))

Max Activating Token Index: 985

Text #14

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2436. Min Act: -0.0000

Data Index: 1026426 (C4 (Web Text))

Max Activating Token Index: 330

Text #15

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2490. Min Act: -0.0000

Data Index: 182181 (C4 (Web Text))

Max Activating Token Index: 459

Text #16

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2390. Min Act: -0.0000

Data Index: 735136 (C4 (Web Text))

Max Activating Token Index: 410

Text #17

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2373. Min Act: -0.0000

Data Index: 422677 (C4 (Web Text))

Max Activating Token Index: 1004

Text #18

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2325. Min Act: -0.0000

Data Index: 770172 (C4 (Web Text))

Max Activating Token Index: 189

Text #19

Max Range: 0.4758. Min Range: -0.4758

Max Act: 0.2257. Min Act: -0.0000

Data Index: 958545 (C4 (Web Text))

Max Activating Token Index: 814