Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1527 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.3183. Min Act: -0.0000

Data Index: 96850 (C4 (Web Text))

Max Activating Token Index: 165

Text #1

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.2659. Min Act: -0.0000

Data Index: 285747 (C4 (Web Text))

Max Activating Token Index: 285

Text #2

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.1602. Min Act: -0.0000

Data Index: 733607 (C4 (Web Text))

Max Activating Token Index: 317

Text #3

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.1128. Min Act: 0.0000

Data Index: 1145207 (C4 (Web Text))

Max Activating Token Index: 773

Text #4

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.1417. Min Act: -0.0000

Data Index: 748714 (C4 (Web Text))

Max Activating Token Index: 544

Text #5

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.1193. Min Act: -0.0000

Data Index: 173968 (C4 (Web Text))

Max Activating Token Index: 63

Text #6

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.2633. Min Act: -0.0000

Data Index: 333159 (C4 (Web Text))

Max Activating Token Index: 634

Text #7

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.1021. Min Act: -0.0000

Data Index: 745280 (C4 (Web Text))

Max Activating Token Index: 518

Text #8

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0915. Min Act: -0.0000

Data Index: 1082757 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0846. Min Act: -0.0000

Data Index: 740917 (C4 (Web Text))

Max Activating Token Index: 474

Text #10

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0511. Min Act: -0.0000

Data Index: 1024160 (C4 (Web Text))

Max Activating Token Index: 244

Text #11

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0501. Min Act: -0.0000

Data Index: 1093822 (C4 (Web Text))

Max Activating Token Index: 940

Text #12

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0451. Min Act: -0.0000

Data Index: 407832 (C4 (Web Text))

Max Activating Token Index: 444

Text #13

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0473. Min Act: -0.0000

Data Index: 601479 (C4 (Web Text))

Max Activating Token Index: 562

Text #14

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0925. Min Act: -0.0000

Data Index: 239497 (C4 (Web Text))

Max Activating Token Index: 268

Text #15

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0238. Min Act: -0.0000

Data Index: 218453 (C4 (Web Text))

Max Activating Token Index: 971

Text #16

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0457. Min Act: -0.0000

Data Index: 1215872 (C4 (Web Text))

Max Activating Token Index: 314

Text #17

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0166. Min Act: -0.0000

Data Index: 512283 (C4 (Web Text))

Max Activating Token Index: 434

Text #18

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0170. Min Act: -0.0000

Data Index: 1167407 (C4 (Web Text))

Max Activating Token Index: 510

Text #19

Max Range: 1.3183. Min Range: -1.3183

Max Act: 1.0022. Min Act: -0.0000

Data Index: 94555 (C4 (Web Text))

Max Activating Token Index: 933