Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1160 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3721. Min Act: -0.0000

Data Index: 11036 (C4 (Web Text))

Max Activating Token Index: 354

Text #1

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3310. Min Act: 0.0000

Data Index: 870021 (C4 (Web Text))

Max Activating Token Index: 851

Text #2

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3300. Min Act: 0.0000

Data Index: 176961 (C4 (Web Text))

Max Activating Token Index: 429

Text #3

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3137. Min Act: -0.0000

Data Index: 16168 (C4 (Web Text))

Max Activating Token Index: 781

Text #4

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3146. Min Act: -0.0000

Data Index: 897817 (C4 (Web Text))

Max Activating Token Index: 459

Text #5

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3248. Min Act: -0.0000

Data Index: 1259497 (C4 (Web Text))

Max Activating Token Index: 969

Text #6

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.3028. Min Act: -0.0000

Data Index: 117196 (C4 (Web Text))

Max Activating Token Index: 132

Text #7

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2850. Min Act: -0.0000

Data Index: 1103720 (C4 (Web Text))

Max Activating Token Index: 797

Text #8

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2793. Min Act: -0.0000

Data Index: 861101 (C4 (Web Text))

Max Activating Token Index: 461

Text #9

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2589. Min Act: -0.0000

Data Index: 147616 (C4 (Web Text))

Max Activating Token Index: 898

Text #10

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2747. Min Act: -0.0000

Data Index: 662817 (C4 (Web Text))

Max Activating Token Index: 661

Text #11

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2536. Min Act: 0.0001

Data Index: 533150 (C4 (Web Text))

Max Activating Token Index: 178

Text #12

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2684. Min Act: -0.0000

Data Index: 1222542 (C4 (Web Text))

Max Activating Token Index: 847

Text #13

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2576. Min Act: -0.0000

Data Index: 978490 (C4 (Web Text))

Max Activating Token Index: 770

Text #14

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2644. Min Act: -0.0000

Data Index: 1144768 (C4 (Web Text))

Max Activating Token Index: 120

Text #15

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2499. Min Act: 0.0000

Data Index: 1241504 (C4 (Web Text))

Max Activating Token Index: 812

Text #16

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2560. Min Act: -0.0000

Data Index: 990796 (C4 (Web Text))

Max Activating Token Index: 850

Text #17

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2514. Min Act: -0.0000

Data Index: 479293 (C4 (Web Text))

Max Activating Token Index: 589

Text #18

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2498. Min Act: -0.0000

Data Index: 573599 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 0.3721. Min Range: -0.3721

Max Act: 0.2398. Min Act: -0.0000

Data Index: 259683 (C4 (Web Text))

Max Activating Token Index: 729