Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2929 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.8448. Min Act: -0.0000

Data Index: 35536 (C4 (Web Text))

Max Activating Token Index: 690

Text #1

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.8382. Min Act: -0.0000

Data Index: 1175371 (C4 (Web Text))

Max Activating Token Index: 992

Text #2

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.7984. Min Act: -0.0000

Data Index: 703512 (C4 (Web Text))

Max Activating Token Index: 260

Text #3

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.8246. Min Act: -0.0000

Data Index: 596488 (C4 (Web Text))

Max Activating Token Index: 443

Text #4

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.7078. Min Act: -0.0000

Data Index: 223441 (C4 (Web Text))

Max Activating Token Index: 328

Text #5

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.7030. Min Act: -0.0000

Data Index: 548763 (C4 (Web Text))

Max Activating Token Index: 824

Text #6

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6670. Min Act: -0.0000

Data Index: 931147 (C4 (Web Text))

Max Activating Token Index: 405

Text #7

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6331. Min Act: -0.0000

Data Index: 445106 (C4 (Web Text))

Max Activating Token Index: 949

Text #8

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6299. Min Act: -0.0000

Data Index: 997027 (C4 (Web Text))

Max Activating Token Index: 604

Text #9

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6157. Min Act: -0.0000

Data Index: 1050907 (C4 (Web Text))

Max Activating Token Index: 892

Text #10

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.5911. Min Act: -0.0000

Data Index: 29665 (C4 (Web Text))

Max Activating Token Index: 547

Text #11

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6045. Min Act: -0.0000

Data Index: 1049385 (C4 (Web Text))

Max Activating Token Index: 397

Text #12

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6018. Min Act: -0.0000

Data Index: 983923 (C4 (Web Text))

Max Activating Token Index: 296

Text #13

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6044. Min Act: -0.0000

Data Index: 1041431 (C4 (Web Text))

Max Activating Token Index: 254

Text #14

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.5978. Min Act: -0.0000

Data Index: 256749 (C4 (Web Text))

Max Activating Token Index: 913

Text #15

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.5942. Min Act: -0.0000

Data Index: 433876 (C4 (Web Text))

Max Activating Token Index: 47

Text #16

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.5851. Min Act: -0.0000

Data Index: 799354 (C4 (Web Text))

Max Activating Token Index: 627

Text #17

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6344. Min Act: -0.0000

Data Index: 418560 (C4 (Web Text))

Max Activating Token Index: 279

Text #18

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.5508. Min Act: -0.0000

Data Index: 1161740 (C4 (Web Text))

Max Activating Token Index: 520

Text #19

Max Range: 0.8448. Min Range: -0.8448

Max Act: 0.6147. Min Act: -0.0000

Data Index: 396693 (C4 (Web Text))

Max Activating Token Index: 825