Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3658 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0540. Min Range: -2.0540

Max Act: 2.0540. Min Act: -0.0000

Data Index: 1238934 (C4 (Web Text))

Max Activating Token Index: 529

Text #1

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.9882. Min Act: -0.0000

Data Index: 107733 (C4 (Web Text))

Max Activating Token Index: 719

Text #2

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.8045. Min Act: -0.0000

Data Index: 1146627 (C4 (Web Text))

Max Activating Token Index: 1010

Text #3

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.5446. Min Act: -0.0000

Data Index: 649525 (C4 (Web Text))

Max Activating Token Index: 761

Text #4

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.7288. Min Act: -0.0000

Data Index: 844590 (C4 (Web Text))

Max Activating Token Index: 900

Text #5

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.9892. Min Act: -0.0000

Data Index: 364359 (C4 (Web Text))

Max Activating Token Index: 743

Text #6

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.7379. Min Act: -0.0000

Data Index: 800332 (C4 (Web Text))

Max Activating Token Index: 466

Text #7

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.8181. Min Act: -0.0000

Data Index: 578040 (C4 (Web Text))

Max Activating Token Index: 674

Text #8

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6944. Min Act: -0.0000

Data Index: 427308 (C4 (Web Text))

Max Activating Token Index: 7

Text #9

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.5970. Min Act: -0.0000

Data Index: 208053 (C4 (Web Text))

Max Activating Token Index: 624

Text #10

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.8144. Min Act: -0.0000

Data Index: 1126069 (C4 (Web Text))

Max Activating Token Index: 54

Text #11

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.5679. Min Act: -0.0000

Data Index: 1611366 (Python Code)

Max Activating Token Index: 130

Text #12

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.7586. Min Act: -0.0000

Data Index: 1105161 (C4 (Web Text))

Max Activating Token Index: 667

Text #13

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.5828. Min Act: -0.0000

Data Index: 150105 (C4 (Web Text))

Max Activating Token Index: 11

Text #14

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6604. Min Act: -0.0000

Data Index: 51475 (C4 (Web Text))

Max Activating Token Index: 30

Text #15

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6941. Min Act: -0.0000

Data Index: 1552714 (Python Code)

Max Activating Token Index: 257

Text #16

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6967. Min Act: -0.0000

Data Index: 322534 (C4 (Web Text))

Max Activating Token Index: 566

Text #17

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6119. Min Act: -0.0000

Data Index: 784415 (C4 (Web Text))

Max Activating Token Index: 724

Text #18

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6484. Min Act: -0.0000

Data Index: 1220987 (C4 (Web Text))

Max Activating Token Index: 829

Text #19

Max Range: 2.0540. Min Range: -2.0540

Max Act: 1.6445. Min Act: -0.0000

Data Index: 146079 (C4 (Web Text))

Max Activating Token Index: 30