Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4594 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3571. Min Act: -0.0000

Data Index: 1559557 (Python Code)

Max Activating Token Index: 500

Text #1

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3274. Min Act: -0.0000

Data Index: 1446308 (Python Code)

Max Activating Token Index: 405

Text #2

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3031. Min Act: -0.0000

Data Index: 1440700 (Python Code)

Max Activating Token Index: 660

Text #3

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3285. Min Act: -0.0000

Data Index: 1639416 (Python Code)

Max Activating Token Index: 575

Text #4

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3011. Min Act: -0.0000

Data Index: 1553884 (Python Code)

Max Activating Token Index: 967

Text #5

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3061. Min Act: -0.0000

Data Index: 848412 (C4 (Web Text))

Max Activating Token Index: 911

Text #6

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2664. Min Act: -0.0000

Data Index: 1556849 (Python Code)

Max Activating Token Index: 304

Text #7

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.3333. Min Act: -0.0000

Data Index: 1619253 (Python Code)

Max Activating Token Index: 965

Text #8

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2886. Min Act: -0.0000

Data Index: 966775 (C4 (Web Text))

Max Activating Token Index: 908

Text #9

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2901. Min Act: -0.0000

Data Index: 196723 (C4 (Web Text))

Max Activating Token Index: 596

Text #10

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2809. Min Act: -0.0000

Data Index: 1518499 (Python Code)

Max Activating Token Index: 871

Text #11

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2763. Min Act: -0.0000

Data Index: 1647210 (Python Code)

Max Activating Token Index: 669

Text #12

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2728. Min Act: -0.0000

Data Index: 1531103 (Python Code)

Max Activating Token Index: 734

Text #13

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2884. Min Act: -0.0000

Data Index: 1438964 (Python Code)

Max Activating Token Index: 397

Text #14

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2938. Min Act: -0.0000

Data Index: 962599 (C4 (Web Text))

Max Activating Token Index: 514

Text #15

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2978. Min Act: -0.0000

Data Index: 1520781 (Python Code)

Max Activating Token Index: 388

Text #16

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2779. Min Act: -0.0000

Data Index: 1233391 (C4 (Web Text))

Max Activating Token Index: 536

Text #17

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2939. Min Act: -0.0000

Data Index: 30403 (C4 (Web Text))

Max Activating Token Index: 960

Text #18

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2894. Min Act: -0.0000

Data Index: 1419440 (Python Code)

Max Activating Token Index: 727

Text #19

Max Range: 0.3571. Min Range: -0.3571

Max Act: 0.2903. Min Act: -0.0000

Data Index: 1468353 (Python Code)

Max Activating Token Index: 934