Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1255 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0879. Min Range: -1.0879

Max Act: 1.0879. Min Act: -0.0000

Data Index: 1324320 (C4 (Web Text))

Max Activating Token Index: 391

Text #1

Max Range: 1.0879. Min Range: -1.0879

Max Act: 1.0517. Min Act: -0.0000

Data Index: 1146041 (C4 (Web Text))

Max Activating Token Index: 317

Text #2

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.9479. Min Act: -0.0000

Data Index: 549889 (C4 (Web Text))

Max Activating Token Index: 348

Text #3

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.9307. Min Act: -0.0000

Data Index: 215751 (C4 (Web Text))

Max Activating Token Index: 794

Text #4

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.9326. Min Act: -0.0000

Data Index: 154146 (C4 (Web Text))

Max Activating Token Index: 470

Text #5

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.8952. Min Act: -0.0000

Data Index: 135061 (C4 (Web Text))

Max Activating Token Index: 126

Text #6

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.8541. Min Act: -0.0000

Data Index: 30274 (C4 (Web Text))

Max Activating Token Index: 832

Text #7

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7941. Min Act: -0.0000

Data Index: 1334020 (C4 (Web Text))

Max Activating Token Index: 493

Text #8

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.8133. Min Act: -0.0000

Data Index: 1239427 (C4 (Web Text))

Max Activating Token Index: 533

Text #9

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.8324. Min Act: 0.0001

Data Index: 258853 (C4 (Web Text))

Max Activating Token Index: 95

Text #10

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7636. Min Act: -0.0000

Data Index: 469921 (C4 (Web Text))

Max Activating Token Index: 489

Text #11

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7344. Min Act: -0.0000

Data Index: 902776 (C4 (Web Text))

Max Activating Token Index: 22

Text #12

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7638. Min Act: -0.0000

Data Index: 519947 (C4 (Web Text))

Max Activating Token Index: 120

Text #13

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7180. Min Act: -0.0000

Data Index: 1231827 (C4 (Web Text))

Max Activating Token Index: 502

Text #14

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.6809. Min Act: -0.0000

Data Index: 936556 (C4 (Web Text))

Max Activating Token Index: 82

Text #15

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7016. Min Act: -0.0000

Data Index: 1357213 (C4 (Web Text))

Max Activating Token Index: 960

Text #16

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.7194. Min Act: -0.0000

Data Index: 495573 (C4 (Web Text))

Max Activating Token Index: 867

Text #17

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.6714. Min Act: -0.0000

Data Index: 735563 (C4 (Web Text))

Max Activating Token Index: 550

Text #18

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.6948. Min Act: -0.0000

Data Index: 817248 (C4 (Web Text))

Max Activating Token Index: 353

Text #19

Max Range: 1.0879. Min Range: -1.0879

Max Act: 0.6852. Min Act: -0.0000

Data Index: 169381 (C4 (Web Text))

Max Activating Token Index: 900