Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2225 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1367. Min Range: -1.1367

Max Act: 1.1367. Min Act: -0.0000

Data Index: 851618 (C4 (Web Text))

Max Activating Token Index: 319

Text #1

Max Range: 1.1367. Min Range: -1.1367

Max Act: 1.1219. Min Act: -0.0000

Data Index: 1011953 (C4 (Web Text))

Max Activating Token Index: 128

Text #2

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.9637. Min Act: -0.0000

Data Index: 1236684 (C4 (Web Text))

Max Activating Token Index: 941

Text #3

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.9852. Min Act: -0.0000

Data Index: 751399 (C4 (Web Text))

Max Activating Token Index: 961

Text #4

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.9434. Min Act: -0.0000

Data Index: 45264 (C4 (Web Text))

Max Activating Token Index: 447

Text #5

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.9370. Min Act: -0.0000

Data Index: 555373 (C4 (Web Text))

Max Activating Token Index: 18

Text #6

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8413. Min Act: -0.0000

Data Index: 375054 (C4 (Web Text))

Max Activating Token Index: 1010

Text #7

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.9580. Min Act: -0.0000

Data Index: 632990 (C4 (Web Text))

Max Activating Token Index: 918

Text #8

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8776. Min Act: -0.0000

Data Index: 1321769 (C4 (Web Text))

Max Activating Token Index: 603

Text #9

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8528. Min Act: -0.0000

Data Index: 1004471 (C4 (Web Text))

Max Activating Token Index: 551

Text #10

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8460. Min Act: -0.0000

Data Index: 429770 (C4 (Web Text))

Max Activating Token Index: 325

Text #11

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8188. Min Act: -0.0000

Data Index: 793642 (C4 (Web Text))

Max Activating Token Index: 239

Text #12

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8135. Min Act: -0.0000

Data Index: 400529 (C4 (Web Text))

Max Activating Token Index: 55

Text #13

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8595. Min Act: -0.0000

Data Index: 60259 (C4 (Web Text))

Max Activating Token Index: 1015

Text #14

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8819. Min Act: -0.0000

Data Index: 1143393 (C4 (Web Text))

Max Activating Token Index: 258

Text #15

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8074. Min Act: -0.0000

Data Index: 60035 (C4 (Web Text))

Max Activating Token Index: 606

Text #16

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8309. Min Act: -0.0000

Data Index: 1081882 (C4 (Web Text))

Max Activating Token Index: 118

Text #17

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.7695. Min Act: -0.0000

Data Index: 1182212 (C4 (Web Text))

Max Activating Token Index: 232

Text #18

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8425. Min Act: -0.0000

Data Index: 978550 (C4 (Web Text))

Max Activating Token Index: 979

Text #19

Max Range: 1.1367. Min Range: -1.1367

Max Act: 0.8431. Min Act: -0.0000

Data Index: 356287 (C4 (Web Text))

Max Activating Token Index: 930