Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4237 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.4202. Min Act: -0.0000

Data Index: 788376 (C4 (Web Text))

Max Activating Token Index: 589

Text #1

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.1506. Min Act: -0.0000

Data Index: 676990 (C4 (Web Text))

Max Activating Token Index: 912

Text #2

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0732. Min Act: -0.0000

Data Index: 512612 (C4 (Web Text))

Max Activating Token Index: 860

Text #3

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0811. Min Act: -0.0000

Data Index: 149094 (C4 (Web Text))

Max Activating Token Index: 883

Text #4

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0167. Min Act: -0.0000

Data Index: 388318 (C4 (Web Text))

Max Activating Token Index: 521

Text #5

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0330. Min Act: -0.0000

Data Index: 388914 (C4 (Web Text))

Max Activating Token Index: 187

Text #6

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0403. Min Act: -0.0000

Data Index: 487712 (C4 (Web Text))

Max Activating Token Index: 346

Text #7

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9508. Min Act: -0.0000

Data Index: 639544 (C4 (Web Text))

Max Activating Token Index: 714

Text #8

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.8471. Min Act: -0.0000

Data Index: 690771 (C4 (Web Text))

Max Activating Token Index: 834

Text #9

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9677. Min Act: -0.0000

Data Index: 262101 (C4 (Web Text))

Max Activating Token Index: 865

Text #10

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9672. Min Act: -0.0000

Data Index: 1090621 (C4 (Web Text))

Max Activating Token Index: 864

Text #11

Max Range: 1.4202. Min Range: -1.4202

Max Act: 1.0299. Min Act: -0.0000

Data Index: 622128 (C4 (Web Text))

Max Activating Token Index: 627

Text #12

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.8833. Min Act: -0.0000

Data Index: 1335294 (C4 (Web Text))

Max Activating Token Index: 838

Text #13

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9412. Min Act: -0.0000

Data Index: 1061856 (C4 (Web Text))

Max Activating Token Index: 1017

Text #14

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9656. Min Act: -0.0000

Data Index: 1341072 (C4 (Web Text))

Max Activating Token Index: 968

Text #15

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.8844. Min Act: -0.0000

Data Index: 384781 (C4 (Web Text))

Max Activating Token Index: 935

Text #16

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.9405. Min Act: -0.0000

Data Index: 836215 (C4 (Web Text))

Max Activating Token Index: 227

Text #17

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.8317. Min Act: -0.0000

Data Index: 405607 (C4 (Web Text))

Max Activating Token Index: 777

Text #18

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.8922. Min Act: -0.0000

Data Index: 1011676 (C4 (Web Text))

Max Activating Token Index: 886

Text #19

Max Range: 1.4202. Min Range: -1.4202

Max Act: 0.7810. Min Act: -0.0000

Data Index: 22483 (C4 (Web Text))

Max Activating Token Index: 216