Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1665 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.9285. Min Range: -3.9285

Max Act: 3.9285. Min Act: -0.0000

Data Index: 1580601 (Python Code)

Max Activating Token Index: 430

Text #1

Max Range: 3.9285. Min Range: -3.9285

Max Act: 3.0857. Min Act: -0.0000

Data Index: 1471630 (Python Code)

Max Activating Token Index: 737

Text #2

Max Range: 3.9285. Min Range: -3.9285

Max Act: 3.0586. Min Act: -0.0000

Data Index: 1448140 (Python Code)

Max Activating Token Index: 375

Text #3

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.8841. Min Act: -0.0000

Data Index: 1605422 (Python Code)

Max Activating Token Index: 683

Text #4

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.4441. Min Act: -0.0000

Data Index: 1508511 (Python Code)

Max Activating Token Index: 784

Text #5

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.5577. Min Act: -0.0000

Data Index: 1447750 (Python Code)

Max Activating Token Index: 315

Text #6

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.3745. Min Act: -0.0000

Data Index: 1611089 (Python Code)

Max Activating Token Index: 234

Text #7

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.2719. Min Act: -0.0000

Data Index: 1426901 (Python Code)

Max Activating Token Index: 342

Text #8

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.3316. Min Act: -0.0000

Data Index: 1362284 (Python Code)

Max Activating Token Index: 318

Text #9

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.3218. Min Act: -0.0000

Data Index: 1409464 (Python Code)

Max Activating Token Index: 135

Text #10

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.4473. Min Act: -0.0000

Data Index: 1411896 (Python Code)

Max Activating Token Index: 502

Text #11

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.2104. Min Act: -0.0000

Data Index: 1548382 (Python Code)

Max Activating Token Index: 877

Text #12

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.2460. Min Act: -0.0000

Data Index: 1511080 (Python Code)

Max Activating Token Index: 835

Text #13

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.3317. Min Act: -0.0000

Data Index: 1619013 (Python Code)

Max Activating Token Index: 974

Text #14

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.1426. Min Act: -0.0000

Data Index: 1592283 (Python Code)

Max Activating Token Index: 881

Text #15

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.0349. Min Act: -0.0000

Data Index: 1410535 (Python Code)

Max Activating Token Index: 343

Text #16

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.0086. Min Act: -0.0000

Data Index: 1414673 (Python Code)

Max Activating Token Index: 193

Text #17

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.1200. Min Act: -0.0000

Data Index: 1407242 (Python Code)

Max Activating Token Index: 277

Text #18

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.0860. Min Act: -0.0000

Data Index: 1531575 (Python Code)

Max Activating Token Index: 413

Text #19

Max Range: 3.9285. Min Range: -3.9285

Max Act: 2.1575. Min Act: -0.0000

Data Index: 1503834 (Python Code)

Max Activating Token Index: 959