Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2832 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3574. Min Act: -0.0000

Data Index: 640821 (C4 (Web Text))

Max Activating Token Index: 356

Text #1

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3466. Min Act: -0.0000

Data Index: 704566 (C4 (Web Text))

Max Activating Token Index: 926

Text #2

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3166. Min Act: -0.0000

Data Index: 798517 (C4 (Web Text))

Max Activating Token Index: 774

Text #3

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3382. Min Act: -0.0000

Data Index: 74019 (C4 (Web Text))

Max Activating Token Index: 901

Text #4

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2993. Min Act: -0.0000

Data Index: 1123343 (C4 (Web Text))

Max Activating Token Index: 1004

Text #5

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2927. Min Act: -0.0000

Data Index: 407854 (C4 (Web Text))

Max Activating Token Index: 884

Text #6

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3005. Min Act: -0.0000

Data Index: 280600 (C4 (Web Text))

Max Activating Token Index: 393

Text #7

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.3112. Min Act: -0.0000

Data Index: 1200527 (C4 (Web Text))

Max Activating Token Index: 658

Text #8

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2866. Min Act: -0.0000

Data Index: 1114099 (C4 (Web Text))

Max Activating Token Index: 560

Text #9

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2836. Min Act: -0.0000

Data Index: 551321 (C4 (Web Text))

Max Activating Token Index: 275

Text #10

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2804. Min Act: -0.0000

Data Index: 340395 (C4 (Web Text))

Max Activating Token Index: 734

Text #11

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2685. Min Act: -0.0000

Data Index: 1128930 (C4 (Web Text))

Max Activating Token Index: 303

Text #12

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2668. Min Act: -0.0000

Data Index: 882605 (C4 (Web Text))

Max Activating Token Index: 885

Text #13

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2662. Min Act: -0.0000

Data Index: 542493 (C4 (Web Text))

Max Activating Token Index: 713

Text #14

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2847. Min Act: -0.0000

Data Index: 466810 (C4 (Web Text))

Max Activating Token Index: 737

Text #15

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2532. Min Act: -0.0000

Data Index: 951679 (C4 (Web Text))

Max Activating Token Index: 401

Text #16

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2466. Min Act: -0.0000

Data Index: 1030676 (C4 (Web Text))

Max Activating Token Index: 152

Text #17

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2499. Min Act: -0.0000

Data Index: 1289703 (C4 (Web Text))

Max Activating Token Index: 585

Text #18

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2390. Min Act: -0.0000

Data Index: 57961 (C4 (Web Text))

Max Activating Token Index: 917

Text #19

Max Range: 0.3574. Min Range: -0.3574

Max Act: 0.2544. Min Act: -0.0000

Data Index: 905340 (C4 (Web Text))

Max Activating Token Index: 489