Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 313 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3843. Min Range: -2.3843

Max Act: 2.3843. Min Act: -0.0000

Data Index: 1586055 (Python Code)

Max Activating Token Index: 423

Text #1

Max Range: 2.3843. Min Range: -2.3843

Max Act: 2.2202. Min Act: -0.0000

Data Index: 123740 (C4 (Web Text))

Max Activating Token Index: 496

Text #2

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.7886. Min Act: -0.0000

Data Index: 1527106 (Python Code)

Max Activating Token Index: 994

Text #3

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.8106. Min Act: -0.0000

Data Index: 1464437 (Python Code)

Max Activating Token Index: 249

Text #4

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6628. Min Act: -0.0000

Data Index: 1196009 (C4 (Web Text))

Max Activating Token Index: 406

Text #5

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6979. Min Act: -0.0000

Data Index: 1594132 (Python Code)

Max Activating Token Index: 198

Text #6

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.5990. Min Act: -0.0000

Data Index: 1455524 (Python Code)

Max Activating Token Index: 901

Text #7

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6780. Min Act: -0.0000

Data Index: 1594181 (Python Code)

Max Activating Token Index: 380

Text #8

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6793. Min Act: -0.0000

Data Index: 1488043 (Python Code)

Max Activating Token Index: 8

Text #9

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6789. Min Act: -0.0000

Data Index: 1632362 (Python Code)

Max Activating Token Index: 803

Text #10

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6403. Min Act: -0.0000

Data Index: 1653497 (Python Code)

Max Activating Token Index: 201

Text #11

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6881. Min Act: -0.0000

Data Index: 1476032 (Python Code)

Max Activating Token Index: 437

Text #12

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.5477. Min Act: -0.0000

Data Index: 1494082 (Python Code)

Max Activating Token Index: 756

Text #13

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6744. Min Act: -0.0000

Data Index: 1496931 (Python Code)

Max Activating Token Index: 603

Text #14

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.5390. Min Act: -0.0000

Data Index: 303056 (C4 (Web Text))

Max Activating Token Index: 307

Text #15

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.7017. Min Act: -0.0000

Data Index: 1375466 (Python Code)

Max Activating Token Index: 829

Text #16

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6987. Min Act: -0.0000

Data Index: 1602058 (Python Code)

Max Activating Token Index: 695

Text #17

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.4949. Min Act: 0.0000

Data Index: 1540889 (Python Code)

Max Activating Token Index: 119

Text #18

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.6339. Min Act: -0.0000

Data Index: 1495777 (Python Code)

Max Activating Token Index: 321

Text #19

Max Range: 2.3843. Min Range: -2.3843

Max Act: 1.5253. Min Act: 0.0000

Data Index: 1642851 (Python Code)

Max Activating Token Index: 24