Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4477 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1380. Min Act: -0.0001

Data Index: 76900 (C4 (Web Text))

Max Activating Token Index: 1007

Text #1

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1102. Min Act: -0.0000

Data Index: 861259 (C4 (Web Text))

Max Activating Token Index: 838

Text #2

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1064. Min Act: -0.0000

Data Index: 928748 (C4 (Web Text))

Max Activating Token Index: 391

Text #3

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1166. Min Act: -0.0000

Data Index: 220332 (C4 (Web Text))

Max Activating Token Index: 1009

Text #4

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1014. Min Act: -0.0000

Data Index: 35679 (C4 (Web Text))

Max Activating Token Index: 808

Text #5

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1026. Min Act: -0.0000

Data Index: 865395 (C4 (Web Text))

Max Activating Token Index: 969

Text #6

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0993. Min Act: -0.0000

Data Index: 735238 (C4 (Web Text))

Max Activating Token Index: 805

Text #7

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1041. Min Act: -0.0000

Data Index: 467838 (C4 (Web Text))

Max Activating Token Index: 992

Text #8

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0969. Min Act: -0.0000

Data Index: 365601 (C4 (Web Text))

Max Activating Token Index: 397

Text #9

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1007. Min Act: -0.0001

Data Index: 49223 (C4 (Web Text))

Max Activating Token Index: 813

Text #10

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0959. Min Act: -0.0000

Data Index: 292636 (C4 (Web Text))

Max Activating Token Index: 796

Text #11

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0929. Min Act: -0.0001

Data Index: 1027324 (C4 (Web Text))

Max Activating Token Index: 750

Text #12

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0874. Min Act: -0.0000

Data Index: 16425 (C4 (Web Text))

Max Activating Token Index: 839

Text #13

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0720. Min Act: -0.0000

Data Index: 825229 (C4 (Web Text))

Max Activating Token Index: 358

Text #14

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0599. Min Act: -0.0000

Data Index: 987457 (C4 (Web Text))

Max Activating Token Index: 534

Text #15

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0636. Min Act: -0.0000

Data Index: 616960 (C4 (Web Text))

Max Activating Token Index: 930

Text #16

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0621. Min Act: -0.0000

Data Index: 866421 (C4 (Web Text))

Max Activating Token Index: 840

Text #17

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0524. Min Act: -0.0000

Data Index: 599459 (C4 (Web Text))

Max Activating Token Index: 849

Text #18

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0542. Min Act: -0.0000

Data Index: 478749 (C4 (Web Text))

Max Activating Token Index: 623

Text #19

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.0534. Min Act: -0.0000

Data Index: 1214992 (C4 (Web Text))

Max Activating Token Index: 673