Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1134 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1546. Min Act: -0.0001

Data Index: 861840 (C4 (Web Text))

Max Activating Token Index: 792

Text #1

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1382. Min Act: -0.0001

Data Index: 550272 (C4 (Web Text))

Max Activating Token Index: 513

Text #2

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1423. Min Act: -0.0001

Data Index: 404109 (C4 (Web Text))

Max Activating Token Index: 374

Text #3

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1422. Min Act: -0.0001

Data Index: 1264202 (C4 (Web Text))

Max Activating Token Index: 623

Text #4

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1304. Min Act: -0.0001

Data Index: 426431 (C4 (Web Text))

Max Activating Token Index: 999

Text #5

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1503. Min Act: -0.0001

Data Index: 336735 (C4 (Web Text))

Max Activating Token Index: 1003

Text #6

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1313. Min Act: -0.0001

Data Index: 925908 (C4 (Web Text))

Max Activating Token Index: 379

Text #7

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1262. Min Act: -0.0001

Data Index: 1203727 (C4 (Web Text))

Max Activating Token Index: 851

Text #8

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1233. Min Act: -0.0001

Data Index: 686002 (C4 (Web Text))

Max Activating Token Index: 394

Text #9

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1252. Min Act: -0.0001

Data Index: 1251531 (C4 (Web Text))

Max Activating Token Index: 494

Text #10

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1237. Min Act: -0.0001

Data Index: 586305 (C4 (Web Text))

Max Activating Token Index: 700

Text #11

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1159. Min Act: -0.0001

Data Index: 248577 (C4 (Web Text))

Max Activating Token Index: 259

Text #12

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1262. Min Act: -0.0001

Data Index: 282515 (C4 (Web Text))

Max Activating Token Index: 880

Text #13

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1212. Min Act: -0.0001

Data Index: 300578 (C4 (Web Text))

Max Activating Token Index: 314

Text #14

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1187. Min Act: -0.0001

Data Index: 166764 (C4 (Web Text))

Max Activating Token Index: 377

Text #15

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1162. Min Act: -0.0001

Data Index: 617075 (C4 (Web Text))

Max Activating Token Index: 624

Text #16

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1169. Min Act: -0.0001

Data Index: 482506 (C4 (Web Text))

Max Activating Token Index: 743

Text #17

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1126. Min Act: -0.0001

Data Index: 890850 (C4 (Web Text))

Max Activating Token Index: 993

Text #18

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1101. Min Act: -0.0000

Data Index: 177450 (C4 (Web Text))

Max Activating Token Index: 81

Text #19

Max Range: 0.1546. Min Range: -0.1546

Max Act: 0.1103. Min Act: -0.0001

Data Index: 1101756 (C4 (Web Text))

Max Activating Token Index: 651