Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2856 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.9486. Min Act: -0.0000

Data Index: 1117777 (C4 (Web Text))

Max Activating Token Index: 725

Text #1

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.8156. Min Act: -0.0000

Data Index: 284416 (C4 (Web Text))

Max Activating Token Index: 67

Text #2

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.8258. Min Act: -0.0000

Data Index: 1229740 (C4 (Web Text))

Max Activating Token Index: 354

Text #3

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.8183. Min Act: -0.0000

Data Index: 512036 (C4 (Web Text))

Max Activating Token Index: 79

Text #4

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.8141. Min Act: -0.0000

Data Index: 1164932 (C4 (Web Text))

Max Activating Token Index: 816

Text #5

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.7678. Min Act: -0.0000

Data Index: 529597 (C4 (Web Text))

Max Activating Token Index: 554

Text #6

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.7475. Min Act: -0.0000

Data Index: 1041240 (C4 (Web Text))

Max Activating Token Index: 72

Text #7

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.7002. Min Act: -0.0000

Data Index: 284057 (C4 (Web Text))

Max Activating Token Index: 783

Text #8

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6444. Min Act: -0.0000

Data Index: 1016499 (C4 (Web Text))

Max Activating Token Index: 57

Text #9

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6572. Min Act: -0.0000

Data Index: 106033 (C4 (Web Text))

Max Activating Token Index: 425

Text #10

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.7093. Min Act: -0.0000

Data Index: 650752 (C4 (Web Text))

Max Activating Token Index: 1006

Text #11

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6905. Min Act: -0.0000

Data Index: 316797 (C4 (Web Text))

Max Activating Token Index: 5

Text #12

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.5751. Min Act: -0.0000

Data Index: 916124 (C4 (Web Text))

Max Activating Token Index: 9

Text #13

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6219. Min Act: -0.0000

Data Index: 423637 (C4 (Web Text))

Max Activating Token Index: 458

Text #14

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6805. Min Act: -0.0000

Data Index: 342864 (C4 (Web Text))

Max Activating Token Index: 226

Text #15

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6043. Min Act: -0.0000

Data Index: 144755 (C4 (Web Text))

Max Activating Token Index: 195

Text #16

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.5292. Min Act: -0.0000

Data Index: 155701 (C4 (Web Text))

Max Activating Token Index: 608

Text #17

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.6430. Min Act: -0.0000

Data Index: 43817 (C4 (Web Text))

Max Activating Token Index: 638

Text #18

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.5286. Min Act: -0.0000

Data Index: 27151 (C4 (Web Text))

Max Activating Token Index: 656

Text #19

Max Range: 1.9486. Min Range: -1.9486

Max Act: 1.4809. Min Act: -0.0000

Data Index: 44205 (C4 (Web Text))

Max Activating Token Index: 52