Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4505 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2164. Min Range: -1.2164

Max Act: 1.1402. Min Act: -0.0000

Data Index: 50927 (C4 (Web Text))

Max Activating Token Index: 770

Text #1

Max Range: 1.2164. Min Range: -1.2164

Max Act: 1.2164. Min Act: -0.0000

Data Index: 33029 (C4 (Web Text))

Max Activating Token Index: 748

Text #2

Max Range: 1.2164. Min Range: -1.2164

Max Act: 1.0163. Min Act: -0.0000

Data Index: 440795 (C4 (Web Text))

Max Activating Token Index: 1002

Text #3

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9523. Min Act: -0.0000

Data Index: 276409 (C4 (Web Text))

Max Activating Token Index: 966

Text #4

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9679. Min Act: -0.0000

Data Index: 329445 (C4 (Web Text))

Max Activating Token Index: 829

Text #5

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9811. Min Act: -0.0000

Data Index: 40908 (C4 (Web Text))

Max Activating Token Index: 111

Text #6

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9736. Min Act: -0.0000

Data Index: 1471362 (Python Code)

Max Activating Token Index: 554

Text #7

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9412. Min Act: -0.0000

Data Index: 834808 (C4 (Web Text))

Max Activating Token Index: 9

Text #8

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.9614. Min Act: -0.0000

Data Index: 1230015 (C4 (Web Text))

Max Activating Token Index: 458

Text #9

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.8773. Min Act: -0.0000

Data Index: 1092187 (C4 (Web Text))

Max Activating Token Index: 575

Text #10

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7897. Min Act: -0.0000

Data Index: 1553936 (Python Code)

Max Activating Token Index: 975

Text #11

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.8632. Min Act: -0.0000

Data Index: 1203116 (C4 (Web Text))

Max Activating Token Index: 560

Text #12

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.8680. Min Act: -0.0000

Data Index: 1425217 (Python Code)

Max Activating Token Index: 544

Text #13

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7716. Min Act: -0.0000

Data Index: 1247893 (C4 (Web Text))

Max Activating Token Index: 700

Text #14

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.8579. Min Act: -0.0000

Data Index: 1591922 (Python Code)

Max Activating Token Index: 239

Text #15

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7931. Min Act: -0.0000

Data Index: 158954 (C4 (Web Text))

Max Activating Token Index: 641

Text #16

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.8423. Min Act: -0.0000

Data Index: 264214 (C4 (Web Text))

Max Activating Token Index: 443

Text #17

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7909. Min Act: -0.0000

Data Index: 921584 (C4 (Web Text))

Max Activating Token Index: 168

Text #18

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7408. Min Act: -0.0000

Data Index: 1196381 (C4 (Web Text))

Max Activating Token Index: 352

Text #19

Max Range: 1.2164. Min Range: -1.2164

Max Act: 0.7923. Min Act: -0.0000

Data Index: 1594557 (Python Code)

Max Activating Token Index: 735