Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2017 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.4373. Min Act: -0.0000

Data Index: 752753 (C4 (Web Text))

Max Activating Token Index: 515

Text #1

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3543. Min Act: -0.0000

Data Index: 92406 (C4 (Web Text))

Max Activating Token Index: 953

Text #2

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3580. Min Act: -0.0000

Data Index: 931736 (C4 (Web Text))

Max Activating Token Index: 198

Text #3

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3430. Min Act: -0.0000

Data Index: 105470 (C4 (Web Text))

Max Activating Token Index: 422

Text #4

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3345. Min Act: -0.0000

Data Index: 486378 (C4 (Web Text))

Max Activating Token Index: 52

Text #5

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3365. Min Act: -0.0000

Data Index: 1306420 (C4 (Web Text))

Max Activating Token Index: 518

Text #6

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3270. Min Act: -0.0000

Data Index: 393423 (C4 (Web Text))

Max Activating Token Index: 249

Text #7

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3189. Min Act: -0.0000

Data Index: 990450 (C4 (Web Text))

Max Activating Token Index: 705

Text #8

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3052. Min Act: -0.0000

Data Index: 264233 (C4 (Web Text))

Max Activating Token Index: 901

Text #9

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3178. Min Act: -0.0000

Data Index: 197372 (C4 (Web Text))

Max Activating Token Index: 399

Text #10

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2954. Min Act: -0.0000

Data Index: 487592 (C4 (Web Text))

Max Activating Token Index: 398

Text #11

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3058. Min Act: -0.0000

Data Index: 566849 (C4 (Web Text))

Max Activating Token Index: 736

Text #12

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.3051. Min Act: -0.0000

Data Index: 573028 (C4 (Web Text))

Max Activating Token Index: 452

Text #13

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2834. Min Act: -0.0000

Data Index: 841565 (C4 (Web Text))

Max Activating Token Index: 189

Text #14

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2899. Min Act: -0.0000

Data Index: 33232 (C4 (Web Text))

Max Activating Token Index: 420

Text #15

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2909. Min Act: -0.0000

Data Index: 188556 (C4 (Web Text))

Max Activating Token Index: 250

Text #16

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2944. Min Act: -0.0000

Data Index: 4522 (C4 (Web Text))

Max Activating Token Index: 638

Text #17

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2895. Min Act: -0.0000

Data Index: 501611 (C4 (Web Text))

Max Activating Token Index: 1009

Text #18

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2684. Min Act: 0.0000

Data Index: 837271 (C4 (Web Text))

Max Activating Token Index: 935

Text #19

Max Range: 0.4373. Min Range: -0.4373

Max Act: 0.2806. Min Act: -0.0000

Data Index: 131798 (C4 (Web Text))

Max Activating Token Index: 713