Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3140 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.6128. Min Act: -0.0002

Data Index: 1381759 (Python Code)

Max Activating Token Index: 12

Text #1

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.5910. Min Act: -0.0001

Data Index: 1645919 (Python Code)

Max Activating Token Index: 788

Text #2

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.5579. Min Act: -0.0001

Data Index: 1548023 (Python Code)

Max Activating Token Index: 58

Text #3

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.5190. Min Act: -0.0001

Data Index: 1503717 (Python Code)

Max Activating Token Index: 168

Text #4

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4979. Min Act: -0.0001

Data Index: 1585617 (Python Code)

Max Activating Token Index: 418

Text #5

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4796. Min Act: -0.0001

Data Index: 1572145 (Python Code)

Max Activating Token Index: 531

Text #6

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4515. Min Act: -0.0001

Data Index: 1504880 (Python Code)

Max Activating Token Index: 15

Text #7

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4435. Min Act: -0.0001

Data Index: 1528454 (Python Code)

Max Activating Token Index: 625

Text #8

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4507. Min Act: -0.0001

Data Index: 1495590 (Python Code)

Max Activating Token Index: 600

Text #9

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4444. Min Act: -0.0002

Data Index: 1600055 (Python Code)

Max Activating Token Index: 417

Text #10

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4424. Min Act: -0.0001

Data Index: 1373175 (Python Code)

Max Activating Token Index: 329

Text #11

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4202. Min Act: -0.0001

Data Index: 1470842 (Python Code)

Max Activating Token Index: 8

Text #12

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4342. Min Act: -0.0001

Data Index: 1594657 (Python Code)

Max Activating Token Index: 747

Text #13

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4304. Min Act: -0.0002

Data Index: 1493914 (Python Code)

Max Activating Token Index: 860

Text #14

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4267. Min Act: -0.0001

Data Index: 1615620 (Python Code)

Max Activating Token Index: 519

Text #15

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4201. Min Act: -0.0001

Data Index: 1499977 (Python Code)

Max Activating Token Index: 875

Text #16

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4151. Min Act: -0.0001

Data Index: 1481472 (Python Code)

Max Activating Token Index: 14

Text #17

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4088. Min Act: -0.0001

Data Index: 1559224 (Python Code)

Max Activating Token Index: 27

Text #18

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4182. Min Act: -0.0001

Data Index: 1585058 (Python Code)

Max Activating Token Index: 19

Text #19

Max Range: 0.6128. Min Range: -0.6128

Max Act: 0.4062. Min Act: -0.0001

Data Index: 1458450 (Python Code)

Max Activating Token Index: 493