Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1051 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.4202. Min Act: -0.0000

Data Index: 1319684 (C4 (Web Text))

Max Activating Token Index: 525

Text #1

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3846. Min Act: -0.0000

Data Index: 1164163 (C4 (Web Text))

Max Activating Token Index: 719

Text #2

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3717. Min Act: -0.0000

Data Index: 1315823 (C4 (Web Text))

Max Activating Token Index: 939

Text #3

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3744. Min Act: -0.0000

Data Index: 1107816 (C4 (Web Text))

Max Activating Token Index: 825

Text #4

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3456. Min Act: 0.0000

Data Index: 1305641 (C4 (Web Text))

Max Activating Token Index: 702

Text #5

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3537. Min Act: -0.0000

Data Index: 411601 (C4 (Web Text))

Max Activating Token Index: 131

Text #6

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3625. Min Act: -0.0000

Data Index: 992315 (C4 (Web Text))

Max Activating Token Index: 229

Text #7

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3398. Min Act: -0.0000

Data Index: 906490 (C4 (Web Text))

Max Activating Token Index: 956

Text #8

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3339. Min Act: -0.0000

Data Index: 211973 (C4 (Web Text))

Max Activating Token Index: 341

Text #9

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3363. Min Act: -0.0000

Data Index: 1018047 (C4 (Web Text))

Max Activating Token Index: 982

Text #10

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3100. Min Act: -0.0000

Data Index: 54026 (C4 (Web Text))

Max Activating Token Index: 752

Text #11

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3223. Min Act: -0.0000

Data Index: 113911 (C4 (Web Text))

Max Activating Token Index: 767

Text #12

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3007. Min Act: -0.0000

Data Index: 290902 (C4 (Web Text))

Max Activating Token Index: 736

Text #13

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3027. Min Act: -0.0000

Data Index: 106809 (C4 (Web Text))

Max Activating Token Index: 425

Text #14

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3100. Min Act: -0.0000

Data Index: 1199473 (C4 (Web Text))

Max Activating Token Index: 540

Text #15

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.2989. Min Act: -0.0000

Data Index: 328299 (C4 (Web Text))

Max Activating Token Index: 833

Text #16

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3071. Min Act: -0.0000

Data Index: 956325 (C4 (Web Text))

Max Activating Token Index: 863

Text #17

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.2938. Min Act: -0.0000

Data Index: 281388 (C4 (Web Text))

Max Activating Token Index: 286

Text #18

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3024. Min Act: -0.0000

Data Index: 442776 (C4 (Web Text))

Max Activating Token Index: 348

Text #19

Max Range: 0.4202. Min Range: -0.4202

Max Act: 0.3017. Min Act: 0.0000

Data Index: 62690 (C4 (Web Text))

Max Activating Token Index: 660