Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3598 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.6784. Min Act: -0.0000

Data Index: 1218845 (C4 (Web Text))

Max Activating Token Index: 555

Text #1

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.6540. Min Act: -0.0000

Data Index: 1181764 (C4 (Web Text))

Max Activating Token Index: 796

Text #2

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.6456. Min Act: -0.0000

Data Index: 1115324 (C4 (Web Text))

Max Activating Token Index: 838

Text #3

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5614. Min Act: -0.0000

Data Index: 42087 (C4 (Web Text))

Max Activating Token Index: 838

Text #4

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5611. Min Act: 0.0000

Data Index: 93689 (C4 (Web Text))

Max Activating Token Index: 599

Text #5

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5495. Min Act: -0.0000

Data Index: 73752 (C4 (Web Text))

Max Activating Token Index: 535

Text #6

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5479. Min Act: -0.0000

Data Index: 803687 (C4 (Web Text))

Max Activating Token Index: 856

Text #7

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5368. Min Act: -0.0000

Data Index: 725309 (C4 (Web Text))

Max Activating Token Index: 752

Text #8

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5037. Min Act: -0.0000

Data Index: 495854 (C4 (Web Text))

Max Activating Token Index: 85

Text #9

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4898. Min Act: -0.0000

Data Index: 273926 (C4 (Web Text))

Max Activating Token Index: 989

Text #10

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.5191. Min Act: -0.0000

Data Index: 545144 (C4 (Web Text))

Max Activating Token Index: 992

Text #11

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4580. Min Act: -0.0000

Data Index: 787258 (C4 (Web Text))

Max Activating Token Index: 503

Text #12

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4451. Min Act: 0.0000

Data Index: 632680 (C4 (Web Text))

Max Activating Token Index: 933

Text #13

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4510. Min Act: -0.0000

Data Index: 1100267 (C4 (Web Text))

Max Activating Token Index: 353

Text #14

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4417. Min Act: -0.0000

Data Index: 988154 (C4 (Web Text))

Max Activating Token Index: 202

Text #15

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4505. Min Act: -0.0000

Data Index: 1271568 (C4 (Web Text))

Max Activating Token Index: 685

Text #16

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4183. Min Act: -0.0000

Data Index: 470996 (C4 (Web Text))

Max Activating Token Index: 630

Text #17

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4240. Min Act: -0.0000

Data Index: 1292988 (C4 (Web Text))

Max Activating Token Index: 1012

Text #18

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4022. Min Act: -0.0000

Data Index: 1127328 (C4 (Web Text))

Max Activating Token Index: 818

Text #19

Max Range: 0.6784. Min Range: -0.6784

Max Act: 0.4112. Min Act: -0.0000

Data Index: 643300 (C4 (Web Text))

Max Activating Token Index: 1002