Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1484 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1931. Min Act: -0.0000

Data Index: 500426 (C4 (Web Text))

Max Activating Token Index: 252

Text #1

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1443. Min Act: -0.0000

Data Index: 1188572 (C4 (Web Text))

Max Activating Token Index: 440

Text #2

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1376. Min Act: -0.0001

Data Index: 34138 (C4 (Web Text))

Max Activating Token Index: 952

Text #3

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1171. Min Act: -0.0001

Data Index: 404831 (C4 (Web Text))

Max Activating Token Index: 751

Text #4

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1210. Min Act: -0.0001

Data Index: 692096 (C4 (Web Text))

Max Activating Token Index: 480

Text #5

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1108. Min Act: -0.0000

Data Index: 955701 (C4 (Web Text))

Max Activating Token Index: 189

Text #6

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1142. Min Act: -0.0001

Data Index: 81476 (C4 (Web Text))

Max Activating Token Index: 701

Text #7

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1153. Min Act: -0.0000

Data Index: 208866 (C4 (Web Text))

Max Activating Token Index: 546

Text #8

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1102. Min Act: -0.0000

Data Index: 423973 (C4 (Web Text))

Max Activating Token Index: 659

Text #9

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1150. Min Act: -0.0000

Data Index: 510525 (C4 (Web Text))

Max Activating Token Index: 617

Text #10

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1092. Min Act: -0.0001

Data Index: 501312 (C4 (Web Text))

Max Activating Token Index: 955

Text #11

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.1007. Min Act: -0.0000

Data Index: 241846 (C4 (Web Text))

Max Activating Token Index: 716

Text #12

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0958. Min Act: -0.0000

Data Index: 1332202 (C4 (Web Text))

Max Activating Token Index: 979

Text #13

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0988. Min Act: -0.0000

Data Index: 243025 (C4 (Web Text))

Max Activating Token Index: 280

Text #14

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0934. Min Act: -0.0001

Data Index: 1023971 (C4 (Web Text))

Max Activating Token Index: 1001

Text #15

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0973. Min Act: -0.0000

Data Index: 1130025 (C4 (Web Text))

Max Activating Token Index: 743

Text #16

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0962. Min Act: -0.0000

Data Index: 796875 (C4 (Web Text))

Max Activating Token Index: 124

Text #17

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0902. Min Act: -0.0001

Data Index: 236558 (C4 (Web Text))

Max Activating Token Index: 745

Text #18

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0962. Min Act: -0.0001

Data Index: 798207 (C4 (Web Text))

Max Activating Token Index: 890

Text #19

Max Range: 0.1931. Min Range: -0.1931

Max Act: 0.0955. Min Act: -0.0000

Data Index: 109448 (C4 (Web Text))

Max Activating Token Index: 122