Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1595 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2231. Min Range: -2.2231

Max Act: 2.2231. Min Act: -0.0000

Data Index: 1563728 (Python Code)

Max Activating Token Index: 534

Text #1

Max Range: 2.2231. Min Range: -2.2231

Max Act: 2.0789. Min Act: -0.0000

Data Index: 110327 (C4 (Web Text))

Max Activating Token Index: 9

Text #2

Max Range: 2.2231. Min Range: -2.2231

Max Act: 2.1050. Min Act: -0.0000

Data Index: 1457791 (Python Code)

Max Activating Token Index: 825

Text #3

Max Range: 2.2231. Min Range: -2.2231

Max Act: 2.0772. Min Act: -0.0001

Data Index: 954032 (C4 (Web Text))

Max Activating Token Index: 18

Text #4

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.9466. Min Act: -0.0001

Data Index: 1643503 (Python Code)

Max Activating Token Index: 229

Text #5

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.9351. Min Act: -0.0000

Data Index: 150981 (C4 (Web Text))

Max Activating Token Index: 131

Text #6

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8865. Min Act: -0.0001

Data Index: 508020 (C4 (Web Text))

Max Activating Token Index: 189

Text #7

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8939. Min Act: -0.0000

Data Index: 303511 (C4 (Web Text))

Max Activating Token Index: 21

Text #8

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.7764. Min Act: -0.0000

Data Index: 345924 (C4 (Web Text))

Max Activating Token Index: 548

Text #9

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8528. Min Act: -0.0001

Data Index: 836332 (C4 (Web Text))

Max Activating Token Index: 707

Text #10

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8679. Min Act: -0.0000

Data Index: 374274 (C4 (Web Text))

Max Activating Token Index: 4

Text #11

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.7467. Min Act: -0.0001

Data Index: 1628206 (Python Code)

Max Activating Token Index: 492

Text #12

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.9694. Min Act: -0.0000

Data Index: 1460720 (Python Code)

Max Activating Token Index: 481

Text #13

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8386. Min Act: -0.0000

Data Index: 986424 (C4 (Web Text))

Max Activating Token Index: 799

Text #14

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.6585. Min Act: -0.0000

Data Index: 509940 (C4 (Web Text))

Max Activating Token Index: 916

Text #15

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.9112. Min Act: -0.0001

Data Index: 825883 (C4 (Web Text))

Max Activating Token Index: 798

Text #16

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.8197. Min Act: -0.0000

Data Index: 431734 (C4 (Web Text))

Max Activating Token Index: 98

Text #17

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.7227. Min Act: -0.0001

Data Index: 1506804 (Python Code)

Max Activating Token Index: 646

Text #18

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.9006. Min Act: -0.0000

Data Index: 445831 (C4 (Web Text))

Max Activating Token Index: 994

Text #19

Max Range: 2.2231. Min Range: -2.2231

Max Act: 1.7547. Min Act: -0.0000

Data Index: 751418 (C4 (Web Text))

Max Activating Token Index: 5