Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2654 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0437. Min Range: -3.0437

Max Act: 3.0346. Min Act: -0.0000

Data Index: 1542177 (Python Code)

Max Activating Token Index: 721

Text #1

Max Range: 3.0437. Min Range: -3.0437

Max Act: 3.0437. Min Act: -0.0000

Data Index: 1578028 (Python Code)

Max Activating Token Index: 49

Text #2

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.8449. Min Act: -0.0000

Data Index: 1496843 (Python Code)

Max Activating Token Index: 49

Text #3

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.9657. Min Act: -0.0000

Data Index: 1392883 (Python Code)

Max Activating Token Index: 35

Text #4

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.8178. Min Act: -0.0000

Data Index: 1592493 (Python Code)

Max Activating Token Index: 37

Text #5

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.8168. Min Act: -0.0000

Data Index: 1625429 (Python Code)

Max Activating Token Index: 170

Text #6

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.6023. Min Act: -0.0000

Data Index: 1599660 (Python Code)

Max Activating Token Index: 385

Text #7

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.6618. Min Act: -0.0000

Data Index: 1515737 (Python Code)

Max Activating Token Index: 987

Text #8

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5090. Min Act: -0.0000

Data Index: 1656260 (Python Code)

Max Activating Token Index: 283

Text #9

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.4700. Min Act: -0.0001

Data Index: 1567843 (Python Code)

Max Activating Token Index: 753

Text #10

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.7263. Min Act: -0.0000

Data Index: 1614444 (Python Code)

Max Activating Token Index: 201

Text #11

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5817. Min Act: -0.0000

Data Index: 1427608 (Python Code)

Max Activating Token Index: 40

Text #12

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5215. Min Act: -0.0000

Data Index: 1418891 (Python Code)

Max Activating Token Index: 534

Text #13

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.6437. Min Act: -0.0000

Data Index: 1538729 (Python Code)

Max Activating Token Index: 88

Text #14

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5032. Min Act: -0.0000

Data Index: 1604448 (Python Code)

Max Activating Token Index: 728

Text #15

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5037. Min Act: -0.0000

Data Index: 1479830 (Python Code)

Max Activating Token Index: 36

Text #16

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.4742. Min Act: -0.0000

Data Index: 1464390 (Python Code)

Max Activating Token Index: 100

Text #17

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.5865. Min Act: -0.0000

Data Index: 1594585 (Python Code)

Max Activating Token Index: 212

Text #18

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.3148. Min Act: -0.0000

Data Index: 1510105 (Python Code)

Max Activating Token Index: 109

Text #19

Max Range: 3.0437. Min Range: -3.0437

Max Act: 2.4239. Min Act: -0.0000

Data Index: 1549308 (Python Code)

Max Activating Token Index: 26