Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4238 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1194. Min Act: -0.0000

Data Index: 510993 (C4 (Web Text))

Max Activating Token Index: 684

Text #1

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1321. Min Act: -0.0000

Data Index: 1304015 (C4 (Web Text))

Max Activating Token Index: 876

Text #2

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1290. Min Act: -0.0000

Data Index: 418036 (C4 (Web Text))

Max Activating Token Index: 546

Text #3

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1175. Min Act: -0.0000

Data Index: 868078 (C4 (Web Text))

Max Activating Token Index: 833

Text #4

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1157. Min Act: -0.0000

Data Index: 237953 (C4 (Web Text))

Max Activating Token Index: 540

Text #5

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1194. Min Act: -0.0000

Data Index: 524223 (C4 (Web Text))

Max Activating Token Index: 769

Text #6

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1140. Min Act: -0.0000

Data Index: 1266321 (C4 (Web Text))

Max Activating Token Index: 943

Text #7

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1170. Min Act: -0.0000

Data Index: 272900 (C4 (Web Text))

Max Activating Token Index: 412

Text #8

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1069. Min Act: -0.0000

Data Index: 683629 (C4 (Web Text))

Max Activating Token Index: 353

Text #9

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1103. Min Act: -0.0000

Data Index: 1359194 (C4 (Web Text))

Max Activating Token Index: 652

Text #10

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1087. Min Act: -0.0000

Data Index: 583165 (C4 (Web Text))

Max Activating Token Index: 801

Text #11

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1042. Min Act: -0.0000

Data Index: 1197722 (C4 (Web Text))

Max Activating Token Index: 678

Text #12

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1043. Min Act: -0.0000

Data Index: 702542 (C4 (Web Text))

Max Activating Token Index: 736

Text #13

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1111. Min Act: -0.0000

Data Index: 221216 (C4 (Web Text))

Max Activating Token Index: 639

Text #14

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1096. Min Act: -0.0000

Data Index: 1000278 (C4 (Web Text))

Max Activating Token Index: 433

Text #15

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1083. Min Act: -0.0000

Data Index: 1000456 (C4 (Web Text))

Max Activating Token Index: 723

Text #16

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1078. Min Act: -0.0000

Data Index: 621172 (C4 (Web Text))

Max Activating Token Index: 835

Text #17

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1046. Min Act: -0.0000

Data Index: 269681 (C4 (Web Text))

Max Activating Token Index: 792

Text #18

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1067. Min Act: -0.0000

Data Index: 108373 (C4 (Web Text))

Max Activating Token Index: 884

Text #19

Max Range: 0.1321. Min Range: -0.1321

Max Act: 0.1050. Min Act: -0.0000

Data Index: 1316141 (C4 (Web Text))

Max Activating Token Index: 953