Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2789 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.5563. Min Act: -0.0000

Data Index: 567840 (C4 (Web Text))

Max Activating Token Index: 353

Text #1

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.5278. Min Act: -0.0000

Data Index: 521000 (C4 (Web Text))

Max Activating Token Index: 146

Text #2

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4993. Min Act: -0.0001

Data Index: 977732 (C4 (Web Text))

Max Activating Token Index: 873

Text #3

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.5114. Min Act: -0.0000

Data Index: 605804 (C4 (Web Text))

Max Activating Token Index: 988

Text #4

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4492. Min Act: -0.0000

Data Index: 1179967 (C4 (Web Text))

Max Activating Token Index: 937

Text #5

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4682. Min Act: -0.0000

Data Index: 370563 (C4 (Web Text))

Max Activating Token Index: 73

Text #6

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.5034. Min Act: -0.0001

Data Index: 238371 (C4 (Web Text))

Max Activating Token Index: 592

Text #7

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4443. Min Act: -0.0001

Data Index: 686683 (C4 (Web Text))

Max Activating Token Index: 859

Text #8

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4439. Min Act: -0.0001

Data Index: 894615 (C4 (Web Text))

Max Activating Token Index: 740

Text #9

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4537. Min Act: -0.0001

Data Index: 502774 (C4 (Web Text))

Max Activating Token Index: 37

Text #10

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4393. Min Act: -0.0000

Data Index: 5075 (C4 (Web Text))

Max Activating Token Index: 249

Text #11

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4347. Min Act: -0.0000

Data Index: 353919 (C4 (Web Text))

Max Activating Token Index: 949

Text #12

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4245. Min Act: -0.0001

Data Index: 739640 (C4 (Web Text))

Max Activating Token Index: 597

Text #13

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4012. Min Act: -0.0000

Data Index: 798442 (C4 (Web Text))

Max Activating Token Index: 595

Text #14

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4274. Min Act: -0.0000

Data Index: 792718 (C4 (Web Text))

Max Activating Token Index: 796

Text #15

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4043. Min Act: -0.0001

Data Index: 493114 (C4 (Web Text))

Max Activating Token Index: 822

Text #16

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4271. Min Act: -0.0001

Data Index: 324977 (C4 (Web Text))

Max Activating Token Index: 315

Text #17

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.4242. Min Act: -0.0001

Data Index: 692891 (C4 (Web Text))

Max Activating Token Index: 925

Text #18

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.3676. Min Act: -0.0001

Data Index: 664669 (C4 (Web Text))

Max Activating Token Index: 830

Text #19

Max Range: 0.5563. Min Range: -0.5563

Max Act: 0.3803. Min Act: -0.0000

Data Index: 193462 (C4 (Web Text))

Max Activating Token Index: 366