Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3824 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.7478. Min Act: -0.0000

Data Index: 1272392 (C4 (Web Text))

Max Activating Token Index: 821

Text #1

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.6215. Min Act: -0.0000

Data Index: 1172874 (C4 (Web Text))

Max Activating Token Index: 553

Text #2

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.5373. Min Act: -0.0000

Data Index: 634669 (C4 (Web Text))

Max Activating Token Index: 406

Text #3

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4655. Min Act: -0.0000

Data Index: 1048560 (C4 (Web Text))

Max Activating Token Index: 337

Text #4

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4324. Min Act: -0.0000

Data Index: 1066976 (C4 (Web Text))

Max Activating Token Index: 296

Text #5

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4705. Min Act: -0.0000

Data Index: 1171514 (C4 (Web Text))

Max Activating Token Index: 213

Text #6

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4469. Min Act: -0.0000

Data Index: 537785 (C4 (Web Text))

Max Activating Token Index: 509

Text #7

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4427. Min Act: -0.0000

Data Index: 1187948 (C4 (Web Text))

Max Activating Token Index: 309

Text #8

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4204. Min Act: -0.0000

Data Index: 748730 (C4 (Web Text))

Max Activating Token Index: 633

Text #9

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4479. Min Act: -0.0000

Data Index: 682539 (C4 (Web Text))

Max Activating Token Index: 93

Text #10

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4066. Min Act: -0.0000

Data Index: 906793 (C4 (Web Text))

Max Activating Token Index: 909

Text #11

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4107. Min Act: -0.0000

Data Index: 806545 (C4 (Web Text))

Max Activating Token Index: 111

Text #12

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.3975. Min Act: -0.0000

Data Index: 468301 (C4 (Web Text))

Max Activating Token Index: 883

Text #13

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4106. Min Act: -0.0000

Data Index: 540610 (C4 (Web Text))

Max Activating Token Index: 1020

Text #14

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4104. Min Act: -0.0000

Data Index: 679920 (C4 (Web Text))

Max Activating Token Index: 119

Text #15

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.3937. Min Act: -0.0000

Data Index: 446093 (C4 (Web Text))

Max Activating Token Index: 1005

Text #16

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4062. Min Act: -0.0000

Data Index: 220197 (C4 (Web Text))

Max Activating Token Index: 41

Text #17

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4004. Min Act: -0.0000

Data Index: 612948 (C4 (Web Text))

Max Activating Token Index: 371

Text #18

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.3941. Min Act: -0.0000

Data Index: 466880 (C4 (Web Text))

Max Activating Token Index: 200

Text #19

Max Range: 0.7478. Min Range: -0.7478

Max Act: 0.4083. Min Act: -0.0000

Data Index: 698717 (C4 (Web Text))

Max Activating Token Index: 119