Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1290 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 4.5295. Min Range: -4.5295

Max Act: 4.5295. Min Act: -0.0001

Data Index: 7091 (C4 (Web Text))

Max Activating Token Index: 987

Text #1

Max Range: 4.5295. Min Range: -4.5295

Max Act: 4.3401. Min Act: -0.0001

Data Index: 1304624 (C4 (Web Text))

Max Activating Token Index: 480

Text #2

Max Range: 4.5295. Min Range: -4.5295

Max Act: 4.1427. Min Act: -0.0001

Data Index: 393570 (C4 (Web Text))

Max Activating Token Index: 453

Text #3

Max Range: 4.5295. Min Range: -4.5295

Max Act: 4.1015. Min Act: -0.0001

Data Index: 212276 (C4 (Web Text))

Max Activating Token Index: 144

Text #4

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.7363. Min Act: -0.0001

Data Index: 167750 (C4 (Web Text))

Max Activating Token Index: 246

Text #5

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.8632. Min Act: -0.0001

Data Index: 411435 (C4 (Web Text))

Max Activating Token Index: 53

Text #6

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.8063. Min Act: -0.0001

Data Index: 1137228 (C4 (Web Text))

Max Activating Token Index: 45

Text #7

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.8599. Min Act: -0.0001

Data Index: 796397 (C4 (Web Text))

Max Activating Token Index: 21

Text #8

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.7247. Min Act: -0.0001

Data Index: 1208906 (C4 (Web Text))

Max Activating Token Index: 811

Text #9

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.5806. Min Act: -0.0001

Data Index: 207181 (C4 (Web Text))

Max Activating Token Index: 532

Text #10

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.5434. Min Act: -0.0001

Data Index: 1303779 (C4 (Web Text))

Max Activating Token Index: 46

Text #11

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.5884. Min Act: -0.0001

Data Index: 1012242 (C4 (Web Text))

Max Activating Token Index: 162

Text #12

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.3790. Min Act: -0.0001

Data Index: 564337 (C4 (Web Text))

Max Activating Token Index: 616

Text #13

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.2376. Min Act: -0.0001

Data Index: 294350 (C4 (Web Text))

Max Activating Token Index: 773

Text #14

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.3087. Min Act: -0.0001

Data Index: 868048 (C4 (Web Text))

Max Activating Token Index: 267

Text #15

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.4385. Min Act: -0.0001

Data Index: 96128 (C4 (Web Text))

Max Activating Token Index: 505

Text #16

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.5269. Min Act: -0.0001

Data Index: 688959 (C4 (Web Text))

Max Activating Token Index: 896

Text #17

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.4032. Min Act: -0.0001

Data Index: 440418 (C4 (Web Text))

Max Activating Token Index: 860

Text #18

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.3922. Min Act: -0.0001

Data Index: 123487 (C4 (Web Text))

Max Activating Token Index: 31

Text #19

Max Range: 4.5295. Min Range: -4.5295

Max Act: 3.3676. Min Act: -0.0001

Data Index: 534545 (C4 (Web Text))

Max Activating Token Index: 157