Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2102 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.6506. Min Act: -0.0000

Data Index: 265309 (C4 (Web Text))

Max Activating Token Index: 394

Text #1

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.5130. Min Act: -0.0000

Data Index: 779979 (C4 (Web Text))

Max Activating Token Index: 223

Text #2

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.5581. Min Act: -0.0000

Data Index: 1177432 (C4 (Web Text))

Max Activating Token Index: 847

Text #3

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.4520. Min Act: -0.0000

Data Index: 1307433 (C4 (Web Text))

Max Activating Token Index: 429

Text #4

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.4283. Min Act: -0.0000

Data Index: 524150 (C4 (Web Text))

Max Activating Token Index: 407

Text #5

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.4413. Min Act: -0.0000

Data Index: 577041 (C4 (Web Text))

Max Activating Token Index: 575

Text #6

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3717. Min Act: -0.0000

Data Index: 277887 (C4 (Web Text))

Max Activating Token Index: 643

Text #7

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3931. Min Act: -0.0000

Data Index: 721042 (C4 (Web Text))

Max Activating Token Index: 976

Text #8

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3789. Min Act: -0.0000

Data Index: 712608 (C4 (Web Text))

Max Activating Token Index: 974

Text #9

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3866. Min Act: -0.0000

Data Index: 606157 (C4 (Web Text))

Max Activating Token Index: 502

Text #10

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3969. Min Act: -0.0000

Data Index: 1003531 (C4 (Web Text))

Max Activating Token Index: 338

Text #11

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3932. Min Act: -0.0000

Data Index: 479351 (C4 (Web Text))

Max Activating Token Index: 1021

Text #12

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3786. Min Act: -0.0000

Data Index: 670128 (C4 (Web Text))

Max Activating Token Index: 169

Text #13

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3723. Min Act: -0.0000

Data Index: 1127380 (C4 (Web Text))

Max Activating Token Index: 162

Text #14

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3486. Min Act: -0.0000

Data Index: 433057 (C4 (Web Text))

Max Activating Token Index: 967

Text #15

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3330. Min Act: -0.0000

Data Index: 1030208 (C4 (Web Text))

Max Activating Token Index: 44

Text #16

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3434. Min Act: -0.0000

Data Index: 1112577 (C4 (Web Text))

Max Activating Token Index: 50

Text #17

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3407. Min Act: -0.0000

Data Index: 1123123 (C4 (Web Text))

Max Activating Token Index: 205

Text #18

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3354. Min Act: -0.0000

Data Index: 245336 (C4 (Web Text))

Max Activating Token Index: 199

Text #19

Max Range: 0.6506. Min Range: -0.6506

Max Act: 0.3384. Min Act: -0.0000

Data Index: 122188 (C4 (Web Text))

Max Activating Token Index: 1010