Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4591 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4403. Min Range: -2.4403

Max Act: 2.4403. Min Act: -0.0000

Data Index: 903743 (C4 (Web Text))

Max Activating Token Index: 218

Text #1

Max Range: 2.4403. Min Range: -2.4403

Max Act: 2.3989. Min Act: -0.0000

Data Index: 908781 (C4 (Web Text))

Max Activating Token Index: 671

Text #2

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8678. Min Act: -0.0000

Data Index: 726681 (C4 (Web Text))

Max Activating Token Index: 360

Text #3

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.9190. Min Act: -0.0000

Data Index: 748483 (C4 (Web Text))

Max Activating Token Index: 683

Text #4

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.9509. Min Act: -0.0000

Data Index: 1069617 (C4 (Web Text))

Max Activating Token Index: 811

Text #5

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8394. Min Act: -0.0000

Data Index: 999083 (C4 (Web Text))

Max Activating Token Index: 394

Text #6

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.9353. Min Act: -0.0000

Data Index: 308824 (C4 (Web Text))

Max Activating Token Index: 506

Text #7

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8717. Min Act: -0.0000

Data Index: 475073 (C4 (Web Text))

Max Activating Token Index: 484

Text #8

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.9704. Min Act: -0.0000

Data Index: 997343 (C4 (Web Text))

Max Activating Token Index: 181

Text #9

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8133. Min Act: -0.0000

Data Index: 443177 (C4 (Web Text))

Max Activating Token Index: 763

Text #10

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8206. Min Act: -0.0000

Data Index: 1222101 (C4 (Web Text))

Max Activating Token Index: 460

Text #11

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7090. Min Act: -0.0000

Data Index: 377589 (C4 (Web Text))

Max Activating Token Index: 478

Text #12

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8951. Min Act: -0.0000

Data Index: 1307093 (C4 (Web Text))

Max Activating Token Index: 965

Text #13

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7973. Min Act: -0.0000

Data Index: 937184 (C4 (Web Text))

Max Activating Token Index: 127

Text #14

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7925. Min Act: -0.0000

Data Index: 1087700 (C4 (Web Text))

Max Activating Token Index: 382

Text #15

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7932. Min Act: -0.0000

Data Index: 1050281 (C4 (Web Text))

Max Activating Token Index: 117

Text #16

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7706. Min Act: -0.0000

Data Index: 477771 (C4 (Web Text))

Max Activating Token Index: 332

Text #17

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8778. Min Act: -0.0000

Data Index: 146383 (C4 (Web Text))

Max Activating Token Index: 403

Text #18

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.8593. Min Act: -0.0000

Data Index: 62491 (C4 (Web Text))

Max Activating Token Index: 722

Text #19

Max Range: 2.4403. Min Range: -2.4403

Max Act: 1.7399. Min Act: -0.0000

Data Index: 746446 (C4 (Web Text))

Max Activating Token Index: 356