Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4355 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.4038. Min Act: -0.0001

Data Index: 185339 (C4 (Web Text))

Max Activating Token Index: 664

Text #1

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.4193. Min Act: -0.0001

Data Index: 169854 (C4 (Web Text))

Max Activating Token Index: 783

Text #2

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3764. Min Act: -0.0001

Data Index: 526266 (C4 (Web Text))

Max Activating Token Index: 238

Text #3

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3657. Min Act: -0.0001

Data Index: 50364 (C4 (Web Text))

Max Activating Token Index: 541

Text #4

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3802. Min Act: -0.0001

Data Index: 500033 (C4 (Web Text))

Max Activating Token Index: 578

Text #5

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3432. Min Act: -0.0001

Data Index: 1104191 (C4 (Web Text))

Max Activating Token Index: 679

Text #6

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3635. Min Act: -0.0001

Data Index: 308145 (C4 (Web Text))

Max Activating Token Index: 818

Text #7

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3521. Min Act: -0.0001

Data Index: 797748 (C4 (Web Text))

Max Activating Token Index: 914

Text #8

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3482. Min Act: -0.0001

Data Index: 745973 (C4 (Web Text))

Max Activating Token Index: 125

Text #9

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3364. Min Act: -0.0001

Data Index: 481074 (C4 (Web Text))

Max Activating Token Index: 531

Text #10

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3198. Min Act: -0.0001

Data Index: 560370 (C4 (Web Text))

Max Activating Token Index: 1019

Text #11

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3329. Min Act: -0.0001

Data Index: 1093512 (C4 (Web Text))

Max Activating Token Index: 459

Text #12

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3169. Min Act: -0.0001

Data Index: 264667 (C4 (Web Text))

Max Activating Token Index: 938

Text #13

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3091. Min Act: -0.0001

Data Index: 919121 (C4 (Web Text))

Max Activating Token Index: 37

Text #14

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3017. Min Act: -0.0001

Data Index: 639558 (C4 (Web Text))

Max Activating Token Index: 48

Text #15

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3292. Min Act: -0.0001

Data Index: 785450 (C4 (Web Text))

Max Activating Token Index: 859

Text #16

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3054. Min Act: -0.0001

Data Index: 584121 (C4 (Web Text))

Max Activating Token Index: 247

Text #17

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.2909. Min Act: -0.0001

Data Index: 1325475 (C4 (Web Text))

Max Activating Token Index: 97

Text #18

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.3015. Min Act: -0.0001

Data Index: 733141 (C4 (Web Text))

Max Activating Token Index: 12

Text #19

Max Range: 0.4193. Min Range: -0.4193

Max Act: 0.2837. Min Act: -0.0001

Data Index: 363692 (C4 (Web Text))

Max Activating Token Index: 129