Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4304 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.4412. Min Act: -0.0001

Data Index: 745354 (C4 (Web Text))

Max Activating Token Index: 523

Text #1

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.4576. Min Act: -0.0001

Data Index: 1188088 (C4 (Web Text))

Max Activating Token Index: 262

Text #2

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.4460. Min Act: -0.0001

Data Index: 967063 (C4 (Web Text))

Max Activating Token Index: 157

Text #3

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.4360. Min Act: -0.0001

Data Index: 251908 (C4 (Web Text))

Max Activating Token Index: 748

Text #4

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3700. Min Act: -0.0001

Data Index: 1056616 (C4 (Web Text))

Max Activating Token Index: 142

Text #5

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.4013. Min Act: -0.0001

Data Index: 784739 (C4 (Web Text))

Max Activating Token Index: 175

Text #6

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3810. Min Act: -0.0001

Data Index: 446177 (C4 (Web Text))

Max Activating Token Index: 91

Text #7

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3477. Min Act: -0.0001

Data Index: 271470 (C4 (Web Text))

Max Activating Token Index: 625

Text #8

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3570. Min Act: -0.0001

Data Index: 366230 (C4 (Web Text))

Max Activating Token Index: 828

Text #9

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3389. Min Act: -0.0001

Data Index: 932282 (C4 (Web Text))

Max Activating Token Index: 39

Text #10

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3721. Min Act: -0.0001

Data Index: 1041775 (C4 (Web Text))

Max Activating Token Index: 650

Text #11

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3580. Min Act: -0.0001

Data Index: 522516 (C4 (Web Text))

Max Activating Token Index: 506

Text #12

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3312. Min Act: -0.0001

Data Index: 1212543 (C4 (Web Text))

Max Activating Token Index: 394

Text #13

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3321. Min Act: -0.0001

Data Index: 661218 (C4 (Web Text))

Max Activating Token Index: 293

Text #14

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3334. Min Act: -0.0001

Data Index: 144877 (C4 (Web Text))

Max Activating Token Index: 312

Text #15

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3429. Min Act: -0.0001

Data Index: 228636 (C4 (Web Text))

Max Activating Token Index: 142

Text #16

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3635. Min Act: -0.0001

Data Index: 1156325 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3355. Min Act: -0.0001

Data Index: 1120791 (C4 (Web Text))

Max Activating Token Index: 896

Text #18

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3227. Min Act: -0.0001

Data Index: 1044498 (C4 (Web Text))

Max Activating Token Index: 6

Text #19

Max Range: 0.4576. Min Range: -0.4576

Max Act: 0.3394. Min Act: -0.0001

Data Index: 1168052 (C4 (Web Text))

Max Activating Token Index: 264