Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6020 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2985. Min Act: -0.0000

Data Index: 1528686 (Python Code)

Max Activating Token Index: 513

Text #1

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2640. Min Act: -0.0000

Data Index: 1530101 (Python Code)

Max Activating Token Index: 763

Text #2

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2628. Min Act: -0.0000

Data Index: 1592265 (Python Code)

Max Activating Token Index: 168

Text #3

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2449. Min Act: -0.0000

Data Index: 1555907 (Python Code)

Max Activating Token Index: 682

Text #4

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2431. Min Act: -0.0000

Data Index: 1534203 (Python Code)

Max Activating Token Index: 610

Text #5

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2403. Min Act: -0.0000

Data Index: 1360905 (Python Code)

Max Activating Token Index: 124

Text #6

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2398. Min Act: -0.0000

Data Index: 1408460 (Python Code)

Max Activating Token Index: 441

Text #7

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2405. Min Act: -0.0000

Data Index: 1475624 (Python Code)

Max Activating Token Index: 33

Text #8

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2393. Min Act: -0.0000

Data Index: 1377569 (Python Code)

Max Activating Token Index: 32

Text #9

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2196. Min Act: -0.0000

Data Index: 1641206 (Python Code)

Max Activating Token Index: 624

Text #10

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2319. Min Act: -0.0000

Data Index: 1455972 (Python Code)

Max Activating Token Index: 214

Text #11

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2148. Min Act: -0.0000

Data Index: 1592686 (Python Code)

Max Activating Token Index: 949

Text #12

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2157. Min Act: -0.0000

Data Index: 1453035 (Python Code)

Max Activating Token Index: 517

Text #13

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2222. Min Act: -0.0000

Data Index: 1503332 (Python Code)

Max Activating Token Index: 482

Text #14

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2121. Min Act: -0.0000

Data Index: 1436156 (Python Code)

Max Activating Token Index: 483

Text #15

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.1986. Min Act: -0.0000

Data Index: 1594163 (Python Code)

Max Activating Token Index: 466

Text #16

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2135. Min Act: -0.0000

Data Index: 1598609 (Python Code)

Max Activating Token Index: 547

Text #17

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2199. Min Act: -0.0000

Data Index: 1636845 (Python Code)

Max Activating Token Index: 516

Text #18

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2111. Min Act: -0.0000

Data Index: 1365729 (Python Code)

Max Activating Token Index: 554

Text #19

Max Range: 0.2985. Min Range: -0.2985

Max Act: 0.2176. Min Act: -0.0000

Data Index: 1551791 (Python Code)

Max Activating Token Index: 80