Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2565 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.4945. Min Act: -0.0000

Data Index: 16986 (C4 (Web Text))

Max Activating Token Index: 731

Text #1

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.4888. Min Act: 0.0000

Data Index: 1055827 (C4 (Web Text))

Max Activating Token Index: 371

Text #2

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.4562. Min Act: -0.0000

Data Index: 1035178 (C4 (Web Text))

Max Activating Token Index: 211

Text #3

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.4253. Min Act: -0.0000

Data Index: 1350250 (C4 (Web Text))

Max Activating Token Index: 79

Text #4

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.4109. Min Act: 0.0000

Data Index: 1269570 (C4 (Web Text))

Max Activating Token Index: 944

Text #5

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3865. Min Act: -0.0000

Data Index: 639792 (C4 (Web Text))

Max Activating Token Index: 268

Text #6

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3939. Min Act: -0.0000

Data Index: 570402 (C4 (Web Text))

Max Activating Token Index: 548

Text #7

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3632. Min Act: -0.0000

Data Index: 558780 (C4 (Web Text))

Max Activating Token Index: 375

Text #8

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3526. Min Act: -0.0000

Data Index: 544569 (C4 (Web Text))

Max Activating Token Index: 557

Text #9

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3719. Min Act: -0.0000

Data Index: 565379 (C4 (Web Text))

Max Activating Token Index: 692

Text #10

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3584. Min Act: -0.0000

Data Index: 894555 (C4 (Web Text))

Max Activating Token Index: 982

Text #11

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3546. Min Act: -0.0000

Data Index: 1143240 (C4 (Web Text))

Max Activating Token Index: 738

Text #12

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3403. Min Act: -0.0000

Data Index: 1133701 (C4 (Web Text))

Max Activating Token Index: 618

Text #13

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3506. Min Act: -0.0000

Data Index: 732796 (C4 (Web Text))

Max Activating Token Index: 605

Text #14

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3463. Min Act: -0.0000

Data Index: 472441 (C4 (Web Text))

Max Activating Token Index: 980

Text #15

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3572. Min Act: 0.0000

Data Index: 555788 (C4 (Web Text))

Max Activating Token Index: 633

Text #16

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3455. Min Act: -0.0000

Data Index: 1180367 (C4 (Web Text))

Max Activating Token Index: 512

Text #17

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3236. Min Act: -0.0000

Data Index: 1147177 (C4 (Web Text))

Max Activating Token Index: 78

Text #18

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3225. Min Act: -0.0000

Data Index: 300273 (C4 (Web Text))

Max Activating Token Index: 719

Text #19

Max Range: 0.4945. Min Range: -0.4945

Max Act: 0.3238. Min Act: 0.0001

Data Index: 153679 (C4 (Web Text))

Max Activating Token Index: 372