Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 914 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.9750. Min Act: -0.0000

Data Index: 1620379 (Python Code)

Max Activating Token Index: 60

Text #1

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.8290. Min Act: -0.0000

Data Index: 1475225 (Python Code)

Max Activating Token Index: 364

Text #2

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.7221. Min Act: -0.0000

Data Index: 1383586 (Python Code)

Max Activating Token Index: 854

Text #3

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.7383. Min Act: -0.0000

Data Index: 1629925 (Python Code)

Max Activating Token Index: 494

Text #4

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.7710. Min Act: -0.0000

Data Index: 1547296 (Python Code)

Max Activating Token Index: 700

Text #5

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.7204. Min Act: -0.0000

Data Index: 1416937 (Python Code)

Max Activating Token Index: 74

Text #6

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6916. Min Act: -0.0000

Data Index: 1602404 (Python Code)

Max Activating Token Index: 457

Text #7

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6866. Min Act: -0.0000

Data Index: 1573798 (Python Code)

Max Activating Token Index: 57

Text #8

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6812. Min Act: -0.0000

Data Index: 1569525 (Python Code)

Max Activating Token Index: 872

Text #9

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.7583. Min Act: -0.0000

Data Index: 1477135 (Python Code)

Max Activating Token Index: 154

Text #10

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6742. Min Act: -0.0000

Data Index: 1609803 (Python Code)

Max Activating Token Index: 310

Text #11

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6523. Min Act: -0.0000

Data Index: 1643716 (Python Code)

Max Activating Token Index: 277

Text #12

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6708. Min Act: -0.0000

Data Index: 1493634 (Python Code)

Max Activating Token Index: 60

Text #13

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6579. Min Act: -0.0000

Data Index: 1384222 (Python Code)

Max Activating Token Index: 658

Text #14

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6581. Min Act: -0.0000

Data Index: 1519248 (Python Code)

Max Activating Token Index: 313

Text #15

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6115. Min Act: -0.0000

Data Index: 1502719 (Python Code)

Max Activating Token Index: 831

Text #16

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6487. Min Act: -0.0000

Data Index: 1430470 (Python Code)

Max Activating Token Index: 162

Text #17

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6217. Min Act: -0.0000

Data Index: 1623681 (Python Code)

Max Activating Token Index: 455

Text #18

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6277. Min Act: -0.0000

Data Index: 1362034 (Python Code)

Max Activating Token Index: 579

Text #19

Max Range: 0.9750. Min Range: -0.9750

Max Act: 0.6378. Min Act: -0.0000

Data Index: 1606262 (Python Code)

Max Activating Token Index: 375