Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4683 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.6539. Min Act: -0.0000

Data Index: 491412 (C4 (Web Text))

Max Activating Token Index: 605

Text #1

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5978. Min Act: -0.0000

Data Index: 910748 (C4 (Web Text))

Max Activating Token Index: 867

Text #2

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.6024. Min Act: 0.0000

Data Index: 1075600 (C4 (Web Text))

Max Activating Token Index: 55

Text #3

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5737. Min Act: -0.0000

Data Index: 436170 (C4 (Web Text))

Max Activating Token Index: 570

Text #4

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5758. Min Act: 0.0000

Data Index: 667483 (C4 (Web Text))

Max Activating Token Index: 265

Text #5

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5542. Min Act: -0.0000

Data Index: 1313426 (C4 (Web Text))

Max Activating Token Index: 655

Text #6

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.6007. Min Act: -0.0000

Data Index: 2370 (C4 (Web Text))

Max Activating Token Index: 28

Text #7

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5557. Min Act: -0.0000

Data Index: 697754 (C4 (Web Text))

Max Activating Token Index: 305

Text #8

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5481. Min Act: -0.0000

Data Index: 565526 (C4 (Web Text))

Max Activating Token Index: 748

Text #9

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5233. Min Act: -0.0000

Data Index: 1123284 (C4 (Web Text))

Max Activating Token Index: 779

Text #10

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5659. Min Act: -0.0000

Data Index: 484333 (C4 (Web Text))

Max Activating Token Index: 955

Text #11

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5100. Min Act: -0.0000

Data Index: 644858 (C4 (Web Text))

Max Activating Token Index: 502

Text #12

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4935. Min Act: -0.0000

Data Index: 852913 (C4 (Web Text))

Max Activating Token Index: 419

Text #13

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5232. Min Act: -0.0000

Data Index: 37454 (C4 (Web Text))

Max Activating Token Index: 499

Text #14

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.5204. Min Act: -0.0000

Data Index: 393535 (C4 (Web Text))

Max Activating Token Index: 716

Text #15

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4990. Min Act: -0.0000

Data Index: 1174264 (C4 (Web Text))

Max Activating Token Index: 938

Text #16

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4996. Min Act: -0.0000

Data Index: 649654 (C4 (Web Text))

Max Activating Token Index: 819

Text #17

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4972. Min Act: -0.0000

Data Index: 459126 (C4 (Web Text))

Max Activating Token Index: 1017

Text #18

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4950. Min Act: -0.0000

Data Index: 930863 (C4 (Web Text))

Max Activating Token Index: 752

Text #19

Max Range: 0.6539. Min Range: -0.6539

Max Act: 0.4915. Min Act: -0.0000

Data Index: 82813 (C4 (Web Text))

Max Activating Token Index: 172