Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3898 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0079. Min Range: -1.0079

Max Act: 1.0079. Min Act: -0.0000

Data Index: 493281 (C4 (Web Text))

Max Activating Token Index: 762

Text #1

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.9486. Min Act: -0.0000

Data Index: 678250 (C4 (Web Text))

Max Activating Token Index: 482

Text #2

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.8520. Min Act: -0.0000

Data Index: 807353 (C4 (Web Text))

Max Activating Token Index: 739

Text #3

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7810. Min Act: -0.0000

Data Index: 284880 (C4 (Web Text))

Max Activating Token Index: 855

Text #4

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.8807. Min Act: -0.0000

Data Index: 1166443 (C4 (Web Text))

Max Activating Token Index: 395

Text #5

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.8133. Min Act: -0.0000

Data Index: 695379 (C4 (Web Text))

Max Activating Token Index: 440

Text #6

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.8544. Min Act: -0.0000

Data Index: 1047405 (C4 (Web Text))

Max Activating Token Index: 731

Text #7

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7966. Min Act: -0.0000

Data Index: 652875 (C4 (Web Text))

Max Activating Token Index: 589

Text #8

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7833. Min Act: -0.0000

Data Index: 1094114 (C4 (Web Text))

Max Activating Token Index: 640

Text #9

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.8052. Min Act: -0.0000

Data Index: 605053 (C4 (Web Text))

Max Activating Token Index: 494

Text #10

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.6607. Min Act: -0.0000

Data Index: 992059 (C4 (Web Text))

Max Activating Token Index: 226

Text #11

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7430. Min Act: -0.0000

Data Index: 1312328 (C4 (Web Text))

Max Activating Token Index: 136

Text #12

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7347. Min Act: -0.0000

Data Index: 929832 (C4 (Web Text))

Max Activating Token Index: 406

Text #13

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7297. Min Act: -0.0000

Data Index: 194609 (C4 (Web Text))

Max Activating Token Index: 345

Text #14

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.6669. Min Act: -0.0000

Data Index: 1296684 (C4 (Web Text))

Max Activating Token Index: 209

Text #15

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7002. Min Act: -0.0000

Data Index: 1020570 (C4 (Web Text))

Max Activating Token Index: 726

Text #16

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.6966. Min Act: -0.0000

Data Index: 1020664 (C4 (Web Text))

Max Activating Token Index: 913

Text #17

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.6652. Min Act: -0.0000

Data Index: 1250136 (C4 (Web Text))

Max Activating Token Index: 592

Text #18

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.6949. Min Act: 0.0000

Data Index: 1114429 (C4 (Web Text))

Max Activating Token Index: 415

Text #19

Max Range: 1.0079. Min Range: -1.0079

Max Act: 0.7469. Min Act: -0.0000

Data Index: 93632 (C4 (Web Text))

Max Activating Token Index: 144