Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3902 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0448. Min Range: -1.0448

Max Act: 1.0448. Min Act: -0.0001

Data Index: 666453 (C4 (Web Text))

Max Activating Token Index: 182

Text #1

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9814. Min Act: -0.0001

Data Index: 301092 (C4 (Web Text))

Max Activating Token Index: 1006

Text #2

Max Range: 1.0448. Min Range: -1.0448

Max Act: 1.0216. Min Act: -0.0001

Data Index: 1191642 (C4 (Web Text))

Max Activating Token Index: 937

Text #3

Max Range: 1.0448. Min Range: -1.0448

Max Act: 1.0118. Min Act: -0.0001

Data Index: 473742 (C4 (Web Text))

Max Activating Token Index: 829

Text #4

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9710. Min Act: -0.0001

Data Index: 443592 (C4 (Web Text))

Max Activating Token Index: 535

Text #5

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9115. Min Act: -0.0001

Data Index: 680474 (C4 (Web Text))

Max Activating Token Index: 956

Text #6

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.7735. Min Act: -0.0001

Data Index: 38566 (C4 (Web Text))

Max Activating Token Index: 198

Text #7

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9615. Min Act: -0.0001

Data Index: 916593 (C4 (Web Text))

Max Activating Token Index: 301

Text #8

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.8272. Min Act: -0.0001

Data Index: 1067344 (C4 (Web Text))

Max Activating Token Index: 831

Text #9

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9835. Min Act: -0.0001

Data Index: 84817 (C4 (Web Text))

Max Activating Token Index: 261

Text #10

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9383. Min Act: -0.0001

Data Index: 1314338 (C4 (Web Text))

Max Activating Token Index: 548

Text #11

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9795. Min Act: -0.0001

Data Index: 275061 (C4 (Web Text))

Max Activating Token Index: 272

Text #12

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.8646. Min Act: -0.0001

Data Index: 1056319 (C4 (Web Text))

Max Activating Token Index: 256

Text #13

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9690. Min Act: -0.0001

Data Index: 895286 (C4 (Web Text))

Max Activating Token Index: 701

Text #14

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.9075. Min Act: -0.0001

Data Index: 1280650 (C4 (Web Text))

Max Activating Token Index: 695

Text #15

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.8773. Min Act: -0.0001

Data Index: 31315 (C4 (Web Text))

Max Activating Token Index: 824

Text #16

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.7603. Min Act: -0.0001

Data Index: 680419 (C4 (Web Text))

Max Activating Token Index: 257

Text #17

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.8952. Min Act: -0.0001

Data Index: 1320042 (C4 (Web Text))

Max Activating Token Index: 401

Text #18

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.7706. Min Act: -0.0001

Data Index: 112296 (C4 (Web Text))

Max Activating Token Index: 252

Text #19

Max Range: 1.0448. Min Range: -1.0448

Max Act: 0.7504. Min Act: -0.0001

Data Index: 1077674 (C4 (Web Text))

Max Activating Token Index: 721