Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2792 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1405. Min Range: -1.1405

Max Act: 1.1041. Min Act: -0.0000

Data Index: 1409294 (Python Code)

Max Activating Token Index: 557

Text #1

Max Range: 1.1405. Min Range: -1.1405

Max Act: 1.1405. Min Act: -0.0000

Data Index: 1067695 (C4 (Web Text))

Max Activating Token Index: 64

Text #2

Max Range: 1.1405. Min Range: -1.1405

Max Act: 1.0630. Min Act: -0.0000

Data Index: 1492128 (Python Code)

Max Activating Token Index: 385

Text #3

Max Range: 1.1405. Min Range: -1.1405

Max Act: 1.0029. Min Act: -0.0000

Data Index: 1008400 (C4 (Web Text))

Max Activating Token Index: 1007

Text #4

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.9209. Min Act: -0.0000

Data Index: 65567 (C4 (Web Text))

Max Activating Token Index: 91

Text #5

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.9968. Min Act: -0.0000

Data Index: 520749 (C4 (Web Text))

Max Activating Token Index: 819

Text #6

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.9333. Min Act: -0.0000

Data Index: 1541123 (Python Code)

Max Activating Token Index: 245

Text #7

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8871. Min Act: -0.0000

Data Index: 1489586 (Python Code)

Max Activating Token Index: 405

Text #8

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.9022. Min Act: -0.0000

Data Index: 1227506 (C4 (Web Text))

Max Activating Token Index: 108

Text #9

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8738. Min Act: -0.0000

Data Index: 1121777 (C4 (Web Text))

Max Activating Token Index: 77

Text #10

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8367. Min Act: -0.0000

Data Index: 1414379 (Python Code)

Max Activating Token Index: 703

Text #11

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8295. Min Act: -0.0000

Data Index: 1475793 (Python Code)

Max Activating Token Index: 524

Text #12

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.7998. Min Act: -0.0000

Data Index: 1641087 (Python Code)

Max Activating Token Index: 265

Text #13

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8191. Min Act: -0.0000

Data Index: 1539726 (Python Code)

Max Activating Token Index: 56

Text #14

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.7850. Min Act: -0.0000

Data Index: 1582661 (Python Code)

Max Activating Token Index: 342

Text #15

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.7399. Min Act: -0.0000

Data Index: 1521269 (Python Code)

Max Activating Token Index: 391

Text #16

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8092. Min Act: -0.0000

Data Index: 609663 (C4 (Web Text))

Max Activating Token Index: 703

Text #17

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8377. Min Act: -0.0000

Data Index: 63519 (C4 (Web Text))

Max Activating Token Index: 966

Text #18

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8311. Min Act: -0.0000

Data Index: 1427295 (Python Code)

Max Activating Token Index: 713

Text #19

Max Range: 1.1405. Min Range: -1.1405

Max Act: 0.8057. Min Act: -0.0000

Data Index: 1491313 (Python Code)

Max Activating Token Index: 455