Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2638 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.4472. Min Act: -0.0000

Data Index: 710745 (C4 (Web Text))

Max Activating Token Index: 611

Text #1

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3829. Min Act: -0.0001

Data Index: 1058804 (C4 (Web Text))

Max Activating Token Index: 435

Text #2

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3764. Min Act: -0.0000

Data Index: 371851 (C4 (Web Text))

Max Activating Token Index: 54

Text #3

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3612. Min Act: -0.0000

Data Index: 1059442 (C4 (Web Text))

Max Activating Token Index: 334

Text #4

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3398. Min Act: -0.0001

Data Index: 652446 (C4 (Web Text))

Max Activating Token Index: 1017

Text #5

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2977. Min Act: -0.0001

Data Index: 1197834 (C4 (Web Text))

Max Activating Token Index: 864

Text #6

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3058. Min Act: -0.0001

Data Index: 256252 (C4 (Web Text))

Max Activating Token Index: 979

Text #7

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3213. Min Act: -0.0000

Data Index: 247025 (C4 (Web Text))

Max Activating Token Index: 789

Text #8

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3007. Min Act: -0.0001

Data Index: 661129 (C4 (Web Text))

Max Activating Token Index: 571

Text #9

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3055. Min Act: -0.0001

Data Index: 222545 (C4 (Web Text))

Max Activating Token Index: 394

Text #10

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3053. Min Act: -0.0000

Data Index: 1343136 (C4 (Web Text))

Max Activating Token Index: 431

Text #11

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2977. Min Act: -0.0001

Data Index: 129058 (C4 (Web Text))

Max Activating Token Index: 196

Text #12

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3084. Min Act: -0.0001

Data Index: 587226 (C4 (Web Text))

Max Activating Token Index: 844

Text #13

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2905. Min Act: -0.0000

Data Index: 714245 (C4 (Web Text))

Max Activating Token Index: 756

Text #14

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.3057. Min Act: -0.0001

Data Index: 468155 (C4 (Web Text))

Max Activating Token Index: 233

Text #15

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2816. Min Act: -0.0001

Data Index: 286827 (C4 (Web Text))

Max Activating Token Index: 609

Text #16

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2844. Min Act: -0.0000

Data Index: 1344703 (C4 (Web Text))

Max Activating Token Index: 134

Text #17

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2820. Min Act: -0.0001

Data Index: 1256183 (C4 (Web Text))

Max Activating Token Index: 507

Text #18

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2844. Min Act: -0.0000

Data Index: 119978 (C4 (Web Text))

Max Activating Token Index: 625

Text #19

Max Range: 0.4472. Min Range: -0.4472

Max Act: 0.2725. Min Act: -0.0001

Data Index: 896096 (C4 (Web Text))

Max Activating Token Index: 563