Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2588 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0525. Min Range: -1.0525

Max Act: 1.0525. Min Act: -0.0000

Data Index: 748039 (C4 (Web Text))

Max Activating Token Index: 201

Text #1

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.9865. Min Act: -0.0000

Data Index: 26860 (C4 (Web Text))

Max Activating Token Index: 525

Text #2

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.8149. Min Act: -0.0000

Data Index: 405112 (C4 (Web Text))

Max Activating Token Index: 919

Text #3

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.8935. Min Act: -0.0000

Data Index: 518396 (C4 (Web Text))

Max Activating Token Index: 689

Text #4

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.8272. Min Act: -0.0000

Data Index: 264740 (C4 (Web Text))

Max Activating Token Index: 116

Text #5

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7479. Min Act: -0.0000

Data Index: 726643 (C4 (Web Text))

Max Activating Token Index: 126

Text #6

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7701. Min Act: -0.0000

Data Index: 56703 (C4 (Web Text))

Max Activating Token Index: 593

Text #7

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7234. Min Act: -0.0000

Data Index: 1311698 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.6997. Min Act: -0.0000

Data Index: 691044 (C4 (Web Text))

Max Activating Token Index: 882

Text #9

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7140. Min Act: -0.0000

Data Index: 1231186 (C4 (Web Text))

Max Activating Token Index: 985

Text #10

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7731. Min Act: -0.0000

Data Index: 972152 (C4 (Web Text))

Max Activating Token Index: 960

Text #11

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7371. Min Act: -0.0000

Data Index: 1045356 (C4 (Web Text))

Max Activating Token Index: 797

Text #12

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7385. Min Act: -0.0000

Data Index: 473521 (C4 (Web Text))

Max Activating Token Index: 362

Text #13

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7238. Min Act: -0.0001

Data Index: 299790 (C4 (Web Text))

Max Activating Token Index: 520

Text #14

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.6987. Min Act: -0.0000

Data Index: 786075 (C4 (Web Text))

Max Activating Token Index: 344

Text #15

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7649. Min Act: -0.0000

Data Index: 725481 (C4 (Web Text))

Max Activating Token Index: 198

Text #16

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7159. Min Act: -0.0000

Data Index: 763528 (C4 (Web Text))

Max Activating Token Index: 49

Text #17

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.6832. Min Act: -0.0000

Data Index: 1351013 (C4 (Web Text))

Max Activating Token Index: 633

Text #18

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7001. Min Act: -0.0000

Data Index: 958965 (C4 (Web Text))

Max Activating Token Index: 903

Text #19

Max Range: 1.0525. Min Range: -1.0525

Max Act: 0.7607. Min Act: -0.0000

Data Index: 902933 (C4 (Web Text))

Max Activating Token Index: 654