Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3021 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.6086. Min Act: -0.0001

Data Index: 1376010 (Python Code)

Max Activating Token Index: 975

Text #1

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4620. Min Act: -0.0001

Data Index: 1568256 (Python Code)

Max Activating Token Index: 929

Text #2

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.6024. Min Act: -0.0001

Data Index: 1501361 (Python Code)

Max Activating Token Index: 941

Text #3

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5557. Min Act: -0.0001

Data Index: 1612806 (Python Code)

Max Activating Token Index: 5

Text #4

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5557. Min Act: -0.0001

Data Index: 1593494 (Python Code)

Max Activating Token Index: 5

Text #5

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5557. Min Act: -0.0001

Data Index: 1552164 (Python Code)

Max Activating Token Index: 5

Text #6

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.6825. Min Act: -0.0001

Data Index: 1548382 (Python Code)

Max Activating Token Index: 991

Text #7

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4854. Min Act: -0.0001

Data Index: 1557784 (Python Code)

Max Activating Token Index: 889

Text #8

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5163. Min Act: -0.0001

Data Index: 1484885 (Python Code)

Max Activating Token Index: 977

Text #9

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5678. Min Act: -0.0001

Data Index: 1525968 (Python Code)

Max Activating Token Index: 169

Text #10

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5103. Min Act: -0.0001

Data Index: 1525774 (Python Code)

Max Activating Token Index: 982

Text #11

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4603. Min Act: -0.0001

Data Index: 1482185 (Python Code)

Max Activating Token Index: 911

Text #12

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4475. Min Act: -0.0001

Data Index: 1647000 (Python Code)

Max Activating Token Index: 346

Text #13

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4811. Min Act: -0.0001

Data Index: 1587381 (Python Code)

Max Activating Token Index: 780

Text #14

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4385. Min Act: -0.0001

Data Index: 1421912 (Python Code)

Max Activating Token Index: 728

Text #15

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4207. Min Act: -0.0001

Data Index: 1553260 (Python Code)

Max Activating Token Index: 975

Text #16

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4160. Min Act: -0.0001

Data Index: 1602940 (Python Code)

Max Activating Token Index: 801

Text #17

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4329. Min Act: -0.0001

Data Index: 1467342 (Python Code)

Max Activating Token Index: 158

Text #18

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.4707. Min Act: -0.0001

Data Index: 1451722 (Python Code)

Max Activating Token Index: 16

Text #19

Max Range: 1.6825. Min Range: -1.6825

Max Act: 1.5401. Min Act: -0.0001

Data Index: 1508992 (Python Code)

Max Activating Token Index: 270