Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2467 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.7512. Min Act: -0.0000

Data Index: 761603 (C4 (Web Text))

Max Activating Token Index: 431

Text #1

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.6929. Min Act: -0.0000

Data Index: 453926 (C4 (Web Text))

Max Activating Token Index: 814

Text #2

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.6242. Min Act: -0.0000

Data Index: 603916 (C4 (Web Text))

Max Activating Token Index: 678

Text #3

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.6500. Min Act: -0.0000

Data Index: 1198758 (C4 (Web Text))

Max Activating Token Index: 388

Text #4

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.6243. Min Act: -0.0000

Data Index: 461797 (C4 (Web Text))

Max Activating Token Index: 73

Text #5

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5899. Min Act: -0.0000

Data Index: 1031713 (C4 (Web Text))

Max Activating Token Index: 169

Text #6

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5820. Min Act: -0.0000

Data Index: 1196036 (C4 (Web Text))

Max Activating Token Index: 567

Text #7

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.6266. Min Act: -0.0000

Data Index: 953047 (C4 (Web Text))

Max Activating Token Index: 334

Text #8

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5441. Min Act: -0.0000

Data Index: 1011068 (C4 (Web Text))

Max Activating Token Index: 723

Text #9

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5433. Min Act: -0.0000

Data Index: 542574 (C4 (Web Text))

Max Activating Token Index: 309

Text #10

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5568. Min Act: -0.0000

Data Index: 581 (C4 (Web Text))

Max Activating Token Index: 955

Text #11

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5476. Min Act: -0.0000

Data Index: 695903 (C4 (Web Text))

Max Activating Token Index: 982

Text #12

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5503. Min Act: -0.0000

Data Index: 1347069 (C4 (Web Text))

Max Activating Token Index: 79

Text #13

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5302. Min Act: -0.0000

Data Index: 779121 (C4 (Web Text))

Max Activating Token Index: 808

Text #14

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5248. Min Act: -0.0000

Data Index: 479899 (C4 (Web Text))

Max Activating Token Index: 968

Text #15

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5229. Min Act: -0.0000

Data Index: 311946 (C4 (Web Text))

Max Activating Token Index: 606

Text #16

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5228. Min Act: -0.0000

Data Index: 449936 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5197. Min Act: -0.0000

Data Index: 141691 (C4 (Web Text))

Max Activating Token Index: 861

Text #18

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5331. Min Act: -0.0000

Data Index: 1201604 (C4 (Web Text))

Max Activating Token Index: 792

Text #19

Max Range: 0.7512. Min Range: -0.7512

Max Act: 0.5127. Min Act: -0.0000

Data Index: 599159 (C4 (Web Text))

Max Activating Token Index: 262