Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5210 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3037. Min Range: -1.3037

Max Act: 1.3037. Min Act: 0.0000

Data Index: 724153 (C4 (Web Text))

Max Activating Token Index: 404

Text #1

Max Range: 1.3037. Min Range: -1.3037

Max Act: 1.1049. Min Act: -0.0000

Data Index: 521870 (C4 (Web Text))

Max Activating Token Index: 837

Text #2

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9790. Min Act: 0.0000

Data Index: 970456 (C4 (Web Text))

Max Activating Token Index: 461

Text #3

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9984. Min Act: -0.0000

Data Index: 679680 (C4 (Web Text))

Max Activating Token Index: 249

Text #4

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9764. Min Act: -0.0000

Data Index: 677801 (C4 (Web Text))

Max Activating Token Index: 242

Text #5

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9928. Min Act: -0.0000

Data Index: 670001 (C4 (Web Text))

Max Activating Token Index: 171

Text #6

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9506. Min Act: -0.0000

Data Index: 460302 (C4 (Web Text))

Max Activating Token Index: 371

Text #7

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9514. Min Act: 0.0000

Data Index: 1011342 (C4 (Web Text))

Max Activating Token Index: 162

Text #8

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9987. Min Act: -0.0000

Data Index: 828239 (C4 (Web Text))

Max Activating Token Index: 588

Text #9

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9112. Min Act: -0.0000

Data Index: 795330 (C4 (Web Text))

Max Activating Token Index: 125

Text #10

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9032. Min Act: -0.0000

Data Index: 313631 (C4 (Web Text))

Max Activating Token Index: 692

Text #11

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8876. Min Act: -0.0000

Data Index: 78089 (C4 (Web Text))

Max Activating Token Index: 1015

Text #12

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8803. Min Act: -0.0000

Data Index: 3358 (C4 (Web Text))

Max Activating Token Index: 810

Text #13

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8296. Min Act: -0.0000

Data Index: 377533 (C4 (Web Text))

Max Activating Token Index: 388

Text #14

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8686. Min Act: -0.0000

Data Index: 329098 (C4 (Web Text))

Max Activating Token Index: 306

Text #15

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.9055. Min Act: -0.0000

Data Index: 14585 (C4 (Web Text))

Max Activating Token Index: 701

Text #16

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8542. Min Act: -0.0000

Data Index: 463334 (C4 (Web Text))

Max Activating Token Index: 965

Text #17

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8024. Min Act: -0.0000

Data Index: 994234 (C4 (Web Text))

Max Activating Token Index: 79

Text #18

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8036. Min Act: 0.0000

Data Index: 872049 (C4 (Web Text))

Max Activating Token Index: 746

Text #19

Max Range: 1.3037. Min Range: -1.3037

Max Act: 0.8210. Min Act: -0.0000

Data Index: 771751 (C4 (Web Text))

Max Activating Token Index: 716