Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1657 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3148. Min Range: -1.3148

Max Act: 1.3148. Min Act: -0.0000

Data Index: 1586575 (Python Code)

Max Activating Token Index: 587

Text #1

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.9735. Min Act: -0.0000

Data Index: 1487961 (Python Code)

Max Activating Token Index: 671

Text #2

Max Range: 1.3148. Min Range: -1.3148

Max Act: 1.0020. Min Act: -0.0000

Data Index: 1649710 (Python Code)

Max Activating Token Index: 311

Text #3

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.8977. Min Act: -0.0000

Data Index: 1613495 (Python Code)

Max Activating Token Index: 977

Text #4

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.8655. Min Act: -0.0000

Data Index: 1385253 (Python Code)

Max Activating Token Index: 744

Text #5

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.8174. Min Act: -0.0000

Data Index: 1422845 (Python Code)

Max Activating Token Index: 878

Text #6

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7914. Min Act: -0.0000

Data Index: 1423848 (Python Code)

Max Activating Token Index: 919

Text #7

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7224. Min Act: -0.0000

Data Index: 1460538 (Python Code)

Max Activating Token Index: 771

Text #8

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7621. Min Act: -0.0000

Data Index: 1516713 (Python Code)

Max Activating Token Index: 633

Text #9

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7853. Min Act: -0.0000

Data Index: 1465284 (Python Code)

Max Activating Token Index: 956

Text #10

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7421. Min Act: -0.0000

Data Index: 1578869 (Python Code)

Max Activating Token Index: 1010

Text #11

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.6843. Min Act: -0.0000

Data Index: 1587130 (Python Code)

Max Activating Token Index: 891

Text #12

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.6235. Min Act: -0.0000

Data Index: 1638978 (Python Code)

Max Activating Token Index: 211

Text #13

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7032. Min Act: -0.0000

Data Index: 1467165 (Python Code)

Max Activating Token Index: 204

Text #14

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.6972. Min Act: -0.0000

Data Index: 1625914 (Python Code)

Max Activating Token Index: 962

Text #15

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7119. Min Act: -0.0000

Data Index: 1606776 (Python Code)

Max Activating Token Index: 438

Text #16

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7090. Min Act: -0.0000

Data Index: 1552988 (Python Code)

Max Activating Token Index: 450

Text #17

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.6547. Min Act: -0.0000

Data Index: 1416474 (Python Code)

Max Activating Token Index: 233

Text #18

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.7007. Min Act: -0.0000

Data Index: 1552615 (Python Code)

Max Activating Token Index: 1015

Text #19

Max Range: 1.3148. Min Range: -1.3148

Max Act: 0.6742. Min Act: -0.0000

Data Index: 1463663 (Python Code)

Max Activating Token Index: 438