Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1780 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.7715. Min Act: -0.0000

Data Index: 323595 (C4 (Web Text))

Max Activating Token Index: 220

Text #1

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.7706. Min Act: -0.0000

Data Index: 818909 (C4 (Web Text))

Max Activating Token Index: 884

Text #2

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.7081. Min Act: -0.0000

Data Index: 76347 (C4 (Web Text))

Max Activating Token Index: 889

Text #3

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6960. Min Act: -0.0000

Data Index: 496707 (C4 (Web Text))

Max Activating Token Index: 383

Text #4

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6610. Min Act: -0.0000

Data Index: 1232719 (C4 (Web Text))

Max Activating Token Index: 374

Text #5

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6796. Min Act: -0.0000

Data Index: 803766 (C4 (Web Text))

Max Activating Token Index: 91

Text #6

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6561. Min Act: -0.0000

Data Index: 441683 (C4 (Web Text))

Max Activating Token Index: 794

Text #7

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6519. Min Act: -0.0000

Data Index: 556001 (C4 (Web Text))

Max Activating Token Index: 794

Text #8

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6426. Min Act: -0.0000

Data Index: 1102274 (C4 (Web Text))

Max Activating Token Index: 544

Text #9

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6290. Min Act: -0.0000

Data Index: 849439 (C4 (Web Text))

Max Activating Token Index: 405

Text #10

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6156. Min Act: -0.0000

Data Index: 419112 (C4 (Web Text))

Max Activating Token Index: 409

Text #11

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.6068. Min Act: -0.0000

Data Index: 999270 (C4 (Web Text))

Max Activating Token Index: 413

Text #12

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5848. Min Act: -0.0000

Data Index: 894463 (C4 (Web Text))

Max Activating Token Index: 266

Text #13

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5483. Min Act: -0.0000

Data Index: 58827 (C4 (Web Text))

Max Activating Token Index: 229

Text #14

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5359. Min Act: -0.0000

Data Index: 218448 (C4 (Web Text))

Max Activating Token Index: 804

Text #15

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5739. Min Act: -0.0000

Data Index: 787279 (C4 (Web Text))

Max Activating Token Index: 897

Text #16

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5505. Min Act: -0.0000

Data Index: 1343291 (C4 (Web Text))

Max Activating Token Index: 431

Text #17

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5729. Min Act: -0.0000

Data Index: 603084 (C4 (Web Text))

Max Activating Token Index: 245

Text #18

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5704. Min Act: -0.0000

Data Index: 864063 (C4 (Web Text))

Max Activating Token Index: 395

Text #19

Max Range: 0.7715. Min Range: -0.7715

Max Act: 0.5825. Min Act: -0.0000

Data Index: 704246 (C4 (Web Text))

Max Activating Token Index: 525