Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5786 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.9131. Min Act: -0.0000

Data Index: 1029118 (C4 (Web Text))

Max Activating Token Index: 132

Text #1

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6949. Min Act: -0.0000

Data Index: 1199296 (C4 (Web Text))

Max Activating Token Index: 506

Text #2

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6871. Min Act: -0.0000

Data Index: 900619 (C4 (Web Text))

Max Activating Token Index: 208

Text #3

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.7007. Min Act: -0.0000

Data Index: 863080 (C4 (Web Text))

Max Activating Token Index: 903

Text #4

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6596. Min Act: -0.0000

Data Index: 405113 (C4 (Web Text))

Max Activating Token Index: 721

Text #5

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6209. Min Act: -0.0000

Data Index: 696285 (C4 (Web Text))

Max Activating Token Index: 398

Text #6

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6274. Min Act: -0.0000

Data Index: 1069567 (C4 (Web Text))

Max Activating Token Index: 785

Text #7

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5994. Min Act: -0.0000

Data Index: 512687 (C4 (Web Text))

Max Activating Token Index: 716

Text #8

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6550. Min Act: -0.0000

Data Index: 1007256 (C4 (Web Text))

Max Activating Token Index: 255

Text #9

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6102. Min Act: -0.0000

Data Index: 828289 (C4 (Web Text))

Max Activating Token Index: 435

Text #10

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.6126. Min Act: -0.0000

Data Index: 878314 (C4 (Web Text))

Max Activating Token Index: 528

Text #11

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5770. Min Act: -0.0000

Data Index: 438163 (C4 (Web Text))

Max Activating Token Index: 144

Text #12

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5625. Min Act: -0.0000

Data Index: 1298326 (C4 (Web Text))

Max Activating Token Index: 1009

Text #13

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5662. Min Act: -0.0000

Data Index: 68628 (C4 (Web Text))

Max Activating Token Index: 158

Text #14

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5610. Min Act: -0.0000

Data Index: 1214083 (C4 (Web Text))

Max Activating Token Index: 570

Text #15

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5593. Min Act: -0.0000

Data Index: 398712 (C4 (Web Text))

Max Activating Token Index: 649

Text #16

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5543. Min Act: -0.0000

Data Index: 760224 (C4 (Web Text))

Max Activating Token Index: 325

Text #17

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5365. Min Act: -0.0000

Data Index: 934728 (C4 (Web Text))

Max Activating Token Index: 759

Text #18

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5736. Min Act: -0.0000

Data Index: 245269 (C4 (Web Text))

Max Activating Token Index: 191

Text #19

Max Range: 0.9131. Min Range: -0.9131

Max Act: 0.5528. Min Act: -0.0000

Data Index: 394737 (C4 (Web Text))

Max Activating Token Index: 180