Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2936 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.4010. Min Act: 0.0001

Data Index: 1306988 (C4 (Web Text))

Max Activating Token Index: 674

Text #1

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.3764. Min Act: -0.0000

Data Index: 601238 (C4 (Web Text))

Max Activating Token Index: 58

Text #2

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.2611. Min Act: 0.0000

Data Index: 172254 (C4 (Web Text))

Max Activating Token Index: 265

Text #3

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.2971. Min Act: 0.0001

Data Index: 710951 (C4 (Web Text))

Max Activating Token Index: 800

Text #4

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.1409. Min Act: -0.0000

Data Index: 773958 (C4 (Web Text))

Max Activating Token Index: 76

Text #5

Max Range: 1.4010. Min Range: -1.4010

Max Act: 1.0888. Min Act: -0.0000

Data Index: 1341279 (C4 (Web Text))

Max Activating Token Index: 957

Text #6

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.9744. Min Act: -0.0000

Data Index: 678806 (C4 (Web Text))

Max Activating Token Index: 133

Text #7

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.9468. Min Act: -0.0000

Data Index: 212718 (C4 (Web Text))

Max Activating Token Index: 707

Text #8

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.9350. Min Act: -0.0000

Data Index: 1220902 (C4 (Web Text))

Max Activating Token Index: 786

Text #9

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.9443. Min Act: -0.0000

Data Index: 554896 (C4 (Web Text))

Max Activating Token Index: 948

Text #10

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.8845. Min Act: -0.0000

Data Index: 455255 (C4 (Web Text))

Max Activating Token Index: 941

Text #11

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.8278. Min Act: -0.0000

Data Index: 799090 (C4 (Web Text))

Max Activating Token Index: 296

Text #12

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.8365. Min Act: -0.0000

Data Index: 320448 (C4 (Web Text))

Max Activating Token Index: 289

Text #13

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.7526. Min Act: -0.0000

Data Index: 1266838 (C4 (Web Text))

Max Activating Token Index: 195

Text #14

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.7649. Min Act: -0.0000

Data Index: 705565 (C4 (Web Text))

Max Activating Token Index: 241

Text #15

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.7176. Min Act: -0.0000

Data Index: 899069 (C4 (Web Text))

Max Activating Token Index: 698

Text #16

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.6984. Min Act: -0.0000

Data Index: 1293743 (C4 (Web Text))

Max Activating Token Index: 389

Text #17

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.7355. Min Act: -0.0000

Data Index: 727009 (C4 (Web Text))

Max Activating Token Index: 264

Text #18

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.6962. Min Act: -0.0000

Data Index: 406146 (C4 (Web Text))

Max Activating Token Index: 484

Text #19

Max Range: 1.4010. Min Range: -1.4010

Max Act: 0.6658. Min Act: -0.0000

Data Index: 912581 (C4 (Web Text))

Max Activating Token Index: 287