Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4156 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.3051. Min Range: -5.3051

Max Act: 5.3051. Min Act: -0.0000

Data Index: 505520 (C4 (Web Text))

Max Activating Token Index: 626

Text #1

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.9044. Min Act: -0.0000

Data Index: 839910 (C4 (Web Text))

Max Activating Token Index: 576

Text #2

Max Range: 5.3051. Min Range: -5.3051

Max Act: 5.1861. Min Act: -0.0000

Data Index: 553764 (C4 (Web Text))

Max Activating Token Index: 376

Text #3

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.5318. Min Act: -0.0000

Data Index: 65859 (C4 (Web Text))

Max Activating Token Index: 460

Text #4

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.5993. Min Act: -0.0000

Data Index: 378623 (C4 (Web Text))

Max Activating Token Index: 170

Text #5

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.2775. Min Act: -0.0000

Data Index: 875320 (C4 (Web Text))

Max Activating Token Index: 957

Text #6

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.4905. Min Act: -0.0000

Data Index: 601508 (C4 (Web Text))

Max Activating Token Index: 192

Text #7

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.0566. Min Act: -0.0000

Data Index: 724463 (C4 (Web Text))

Max Activating Token Index: 536

Text #8

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.4073. Min Act: -0.0000

Data Index: 935570 (C4 (Web Text))

Max Activating Token Index: 147

Text #9

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.7969. Min Act: -0.0000

Data Index: 727512 (C4 (Web Text))

Max Activating Token Index: 909

Text #10

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.2077. Min Act: -0.0000

Data Index: 1262274 (C4 (Web Text))

Max Activating Token Index: 598

Text #11

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.0850. Min Act: -0.0000

Data Index: 684294 (C4 (Web Text))

Max Activating Token Index: 850

Text #12

Max Range: 5.3051. Min Range: -5.3051

Max Act: 4.0263. Min Act: -0.0000

Data Index: 1336922 (C4 (Web Text))

Max Activating Token Index: 638

Text #13

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.6958. Min Act: -0.0000

Data Index: 347492 (C4 (Web Text))

Max Activating Token Index: 323

Text #14

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.6282. Min Act: -0.0000

Data Index: 1037761 (C4 (Web Text))

Max Activating Token Index: 294

Text #15

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.6180. Min Act: -0.0000

Data Index: 699543 (C4 (Web Text))

Max Activating Token Index: 563

Text #16

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.7177. Min Act: -0.0000

Data Index: 383333 (C4 (Web Text))

Max Activating Token Index: 237

Text #17

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.6438. Min Act: -0.0000

Data Index: 723394 (C4 (Web Text))

Max Activating Token Index: 827

Text #18

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.6090. Min Act: -0.0000

Data Index: 521309 (C4 (Web Text))

Max Activating Token Index: 529

Text #19

Max Range: 5.3051. Min Range: -5.3051

Max Act: 3.3825. Min Act: -0.0000

Data Index: 874247 (C4 (Web Text))

Max Activating Token Index: 419