Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3083 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.7302. Min Act: -0.0000

Data Index: 653380 (C4 (Web Text))

Max Activating Token Index: 743

Text #1

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.6249. Min Act: -0.0000

Data Index: 286747 (C4 (Web Text))

Max Activating Token Index: 720

Text #2

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.5952. Min Act: -0.0000

Data Index: 1028553 (C4 (Web Text))

Max Activating Token Index: 825

Text #3

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.5732. Min Act: -0.0000

Data Index: 768392 (C4 (Web Text))

Max Activating Token Index: 690

Text #4

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.5675. Min Act: -0.0000

Data Index: 1199587 (C4 (Web Text))

Max Activating Token Index: 482

Text #5

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.4827. Min Act: -0.0000

Data Index: 878946 (C4 (Web Text))

Max Activating Token Index: 659

Text #6

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.4823. Min Act: -0.0000

Data Index: 1098297 (C4 (Web Text))

Max Activating Token Index: 286

Text #7

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.4257. Min Act: -0.0000

Data Index: 798295 (C4 (Web Text))

Max Activating Token Index: 567

Text #8

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3752. Min Act: 0.0003

Data Index: 647671 (C4 (Web Text))

Max Activating Token Index: 392

Text #9

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3715. Min Act: 0.0001

Data Index: 1092322 (C4 (Web Text))

Max Activating Token Index: 515

Text #10

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3568. Min Act: -0.0000

Data Index: 240917 (C4 (Web Text))

Max Activating Token Index: 186

Text #11

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3527. Min Act: -0.0000

Data Index: 428299 (C4 (Web Text))

Max Activating Token Index: 387

Text #12

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3662. Min Act: -0.0000

Data Index: 1049498 (C4 (Web Text))

Max Activating Token Index: 589

Text #13

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3412. Min Act: 0.0000

Data Index: 1017708 (C4 (Web Text))

Max Activating Token Index: 45

Text #14

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3267. Min Act: 0.0001

Data Index: 857351 (C4 (Web Text))

Max Activating Token Index: 318

Text #15

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3273. Min Act: -0.0000

Data Index: 535668 (C4 (Web Text))

Max Activating Token Index: 738

Text #16

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3085. Min Act: -0.0000

Data Index: 389795 (C4 (Web Text))

Max Activating Token Index: 854

Text #17

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3217. Min Act: -0.0000

Data Index: 287995 (C4 (Web Text))

Max Activating Token Index: 953

Text #18

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3155. Min Act: -0.0000

Data Index: 819050 (C4 (Web Text))

Max Activating Token Index: 487

Text #19

Max Range: 0.7302. Min Range: -0.7302

Max Act: 0.3096. Min Act: -0.0000

Data Index: 1082444 (C4 (Web Text))

Max Activating Token Index: 838