Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4996 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4059. Min Range: -3.4059

Max Act: 3.4059. Min Act: -0.0000

Data Index: 779482 (C4 (Web Text))

Max Activating Token Index: 301

Text #1

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9056. Min Act: -0.0000

Data Index: 292739 (C4 (Web Text))

Max Activating Token Index: 735

Text #2

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.6935. Min Act: -0.0000

Data Index: 372444 (C4 (Web Text))

Max Activating Token Index: 802

Text #3

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9154. Min Act: -0.0000

Data Index: 1216220 (C4 (Web Text))

Max Activating Token Index: 264

Text #4

Max Range: 3.4059. Min Range: -3.4059

Max Act: 3.0076. Min Act: -0.0000

Data Index: 577361 (C4 (Web Text))

Max Activating Token Index: 353

Text #5

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9506. Min Act: -0.0000

Data Index: 721218 (C4 (Web Text))

Max Activating Token Index: 409

Text #6

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9233. Min Act: -0.0000

Data Index: 717333 (C4 (Web Text))

Max Activating Token Index: 599

Text #7

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9013. Min Act: -0.0000

Data Index: 131551 (C4 (Web Text))

Max Activating Token Index: 561

Text #8

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.8720. Min Act: -0.0000

Data Index: 1023152 (C4 (Web Text))

Max Activating Token Index: 391

Text #9

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.6957. Min Act: -0.0000

Data Index: 570037 (C4 (Web Text))

Max Activating Token Index: 988

Text #10

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.8505. Min Act: -0.0000

Data Index: 766497 (C4 (Web Text))

Max Activating Token Index: 991

Text #11

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.9528. Min Act: -0.0000

Data Index: 773558 (C4 (Web Text))

Max Activating Token Index: 952

Text #12

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.8783. Min Act: -0.0000

Data Index: 225784 (C4 (Web Text))

Max Activating Token Index: 246

Text #13

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.5921. Min Act: -0.0000

Data Index: 1086673 (C4 (Web Text))

Max Activating Token Index: 80

Text #14

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.6953. Min Act: -0.0000

Data Index: 484256 (C4 (Web Text))

Max Activating Token Index: 665

Text #15

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.5196. Min Act: -0.0000

Data Index: 551183 (C4 (Web Text))

Max Activating Token Index: 945

Text #16

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.4828. Min Act: -0.0000

Data Index: 1128852 (C4 (Web Text))

Max Activating Token Index: 83

Text #17

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.5968. Min Act: -0.0000

Data Index: 313852 (C4 (Web Text))

Max Activating Token Index: 398

Text #18

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.5793. Min Act: -0.0000

Data Index: 1318000 (C4 (Web Text))

Max Activating Token Index: 118

Text #19

Max Range: 3.4059. Min Range: -3.4059

Max Act: 2.4213. Min Act: -0.0000

Data Index: 1280100 (C4 (Web Text))

Max Activating Token Index: 285