Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 425 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1177. Min Range: -2.1177

Max Act: 2.0674. Min Act: -0.0000

Data Index: 209218 (C4 (Web Text))

Max Activating Token Index: 381

Text #1

Max Range: 2.1177. Min Range: -2.1177

Max Act: 2.0027. Min Act: -0.0000

Data Index: 140100 (C4 (Web Text))

Max Activating Token Index: 575

Text #2

Max Range: 2.1177. Min Range: -2.1177

Max Act: 2.1177. Min Act: -0.0000

Data Index: 290627 (C4 (Web Text))

Max Activating Token Index: 705

Text #3

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.9253. Min Act: -0.0000

Data Index: 1267398 (C4 (Web Text))

Max Activating Token Index: 908

Text #4

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.9225. Min Act: -0.0000

Data Index: 989450 (C4 (Web Text))

Max Activating Token Index: 592

Text #5

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.8727. Min Act: -0.0000

Data Index: 34084 (C4 (Web Text))

Max Activating Token Index: 446

Text #6

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.8563. Min Act: -0.0000

Data Index: 1241319 (C4 (Web Text))

Max Activating Token Index: 510

Text #7

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.9425. Min Act: -0.0000

Data Index: 648877 (C4 (Web Text))

Max Activating Token Index: 342

Text #8

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.7953. Min Act: -0.0000

Data Index: 1393913 (Python Code)

Max Activating Token Index: 951

Text #9

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6531. Min Act: -0.0000

Data Index: 1124660 (C4 (Web Text))

Max Activating Token Index: 864

Text #10

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6351. Min Act: -0.0000

Data Index: 1254540 (C4 (Web Text))

Max Activating Token Index: 665

Text #11

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.7158. Min Act: -0.0000

Data Index: 290592 (C4 (Web Text))

Max Activating Token Index: 549

Text #12

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6140. Min Act: -0.0000

Data Index: 1174774 (C4 (Web Text))

Max Activating Token Index: 580

Text #13

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.7103. Min Act: -0.0000

Data Index: 1204732 (C4 (Web Text))

Max Activating Token Index: 85

Text #14

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.5434. Min Act: -0.0000

Data Index: 1344902 (C4 (Web Text))

Max Activating Token Index: 875

Text #15

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6294. Min Act: -0.0000

Data Index: 598860 (C4 (Web Text))

Max Activating Token Index: 970

Text #16

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.7028. Min Act: -0.0000

Data Index: 788028 (C4 (Web Text))

Max Activating Token Index: 271

Text #17

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6054. Min Act: -0.0000

Data Index: 113871 (C4 (Web Text))

Max Activating Token Index: 485

Text #18

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.5859. Min Act: -0.0000

Data Index: 63593 (C4 (Web Text))

Max Activating Token Index: 786

Text #19

Max Range: 2.1177. Min Range: -2.1177

Max Act: 1.6721. Min Act: -0.0000

Data Index: 933158 (C4 (Web Text))

Max Activating Token Index: 66