Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5524 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1893. Min Act: -0.0001

Data Index: 84283 (C4 (Web Text))

Max Activating Token Index: 900

Text #1

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1785. Min Act: -0.0001

Data Index: 101433 (C4 (Web Text))

Max Activating Token Index: 631

Text #2

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1524. Min Act: -0.0001

Data Index: 1335081 (C4 (Web Text))

Max Activating Token Index: 872

Text #3

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1669. Min Act: -0.0001

Data Index: 659707 (C4 (Web Text))

Max Activating Token Index: 110

Text #4

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1385. Min Act: -0.0001

Data Index: 463632 (C4 (Web Text))

Max Activating Token Index: 913

Text #5

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1401. Min Act: -0.0000

Data Index: 617813 (C4 (Web Text))

Max Activating Token Index: 775

Text #6

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1305. Min Act: -0.0000

Data Index: 281360 (C4 (Web Text))

Max Activating Token Index: 520

Text #7

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1303. Min Act: -0.0001

Data Index: 611079 (C4 (Web Text))

Max Activating Token Index: 945

Text #8

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1196. Min Act: -0.0001

Data Index: 904173 (C4 (Web Text))

Max Activating Token Index: 950

Text #9

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1133. Min Act: -0.0001

Data Index: 108866 (C4 (Web Text))

Max Activating Token Index: 567

Text #10

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1300. Min Act: -0.0001

Data Index: 212961 (C4 (Web Text))

Max Activating Token Index: 401

Text #11

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1089. Min Act: -0.0001

Data Index: 1252613 (C4 (Web Text))

Max Activating Token Index: 59

Text #12

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1111. Min Act: -0.0000

Data Index: 897954 (C4 (Web Text))

Max Activating Token Index: 942

Text #13

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1083. Min Act: -0.0001

Data Index: 87681 (C4 (Web Text))

Max Activating Token Index: 889

Text #14

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1112. Min Act: -0.0001

Data Index: 1052969 (C4 (Web Text))

Max Activating Token Index: 569

Text #15

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1085. Min Act: -0.0001

Data Index: 212428 (C4 (Web Text))

Max Activating Token Index: 966

Text #16

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.0999. Min Act: -0.0000

Data Index: 207760 (C4 (Web Text))

Max Activating Token Index: 169

Text #17

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.1093. Min Act: -0.0001

Data Index: 438515 (C4 (Web Text))

Max Activating Token Index: 940

Text #18

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.0962. Min Act: -0.0001

Data Index: 792003 (C4 (Web Text))

Max Activating Token Index: 746

Text #19

Max Range: 0.1893. Min Range: -0.1893

Max Act: 0.0968. Min Act: -0.0001

Data Index: 99985 (C4 (Web Text))

Max Activating Token Index: 206