Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6018 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3543. Min Act: -0.0000

Data Index: 654847 (C4 (Web Text))

Max Activating Token Index: 88

Text #1

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3346. Min Act: -0.0000

Data Index: 623578 (C4 (Web Text))

Max Activating Token Index: 396

Text #2

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3501. Min Act: -0.0000

Data Index: 1357137 (C4 (Web Text))

Max Activating Token Index: 613

Text #3

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3354. Min Act: -0.0000

Data Index: 679956 (C4 (Web Text))

Max Activating Token Index: 530

Text #4

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3310. Min Act: -0.0000

Data Index: 1144915 (C4 (Web Text))

Max Activating Token Index: 516

Text #5

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3394. Min Act: -0.0000

Data Index: 249975 (C4 (Web Text))

Max Activating Token Index: 313

Text #6

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3394. Min Act: -0.0000

Data Index: 835938 (C4 (Web Text))

Max Activating Token Index: 686

Text #7

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3103. Min Act: -0.0000

Data Index: 1332578 (C4 (Web Text))

Max Activating Token Index: 840

Text #8

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.3078. Min Act: -0.0000

Data Index: 477896 (C4 (Web Text))

Max Activating Token Index: 669

Text #9

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2882. Min Act: -0.0000

Data Index: 765774 (C4 (Web Text))

Max Activating Token Index: 666

Text #10

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2880. Min Act: -0.0000

Data Index: 476876 (C4 (Web Text))

Max Activating Token Index: 408

Text #11

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2708. Min Act: -0.0000

Data Index: 664528 (C4 (Web Text))

Max Activating Token Index: 664

Text #12

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2869. Min Act: -0.0000

Data Index: 1182206 (C4 (Web Text))

Max Activating Token Index: 936

Text #13

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2670. Min Act: -0.0000

Data Index: 663515 (C4 (Web Text))

Max Activating Token Index: 220

Text #14

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2837. Min Act: -0.0000

Data Index: 826750 (C4 (Web Text))

Max Activating Token Index: 719

Text #15

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2710. Min Act: -0.0000

Data Index: 488354 (C4 (Web Text))

Max Activating Token Index: 478

Text #16

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2603. Min Act: -0.0000

Data Index: 860018 (C4 (Web Text))

Max Activating Token Index: 197

Text #17

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2712. Min Act: -0.0000

Data Index: 677041 (C4 (Web Text))

Max Activating Token Index: 702

Text #18

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2685. Min Act: -0.0000

Data Index: 17276 (C4 (Web Text))

Max Activating Token Index: 248

Text #19

Max Range: 0.3543. Min Range: -0.3543

Max Act: 0.2616. Min Act: -0.0000

Data Index: 242992 (C4 (Web Text))

Max Activating Token Index: 280