Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3457 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 8.3216. Min Range: -8.3216

Max Act: 8.3216. Min Act: -0.0000

Data Index: 109096 (C4 (Web Text))

Max Activating Token Index: 54

Text #1

Max Range: 8.3216. Min Range: -8.3216

Max Act: 7.3936. Min Act: -0.0000

Data Index: 673875 (C4 (Web Text))

Max Activating Token Index: 40

Text #2

Max Range: 8.3216. Min Range: -8.3216

Max Act: 7.1352. Min Act: -0.0000

Data Index: 106657 (C4 (Web Text))

Max Activating Token Index: 399

Text #3

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.7850. Min Act: -0.0000

Data Index: 956263 (C4 (Web Text))

Max Activating Token Index: 187

Text #4

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.6653. Min Act: -0.0000

Data Index: 271546 (C4 (Web Text))

Max Activating Token Index: 725

Text #5

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.6589. Min Act: -0.0000

Data Index: 686066 (C4 (Web Text))

Max Activating Token Index: 32

Text #6

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.2619. Min Act: -0.0000

Data Index: 730515 (C4 (Web Text))

Max Activating Token Index: 485

Text #7

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.4246. Min Act: -0.0000

Data Index: 1299113 (C4 (Web Text))

Max Activating Token Index: 734

Text #8

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.3629. Min Act: -0.0000

Data Index: 621759 (C4 (Web Text))

Max Activating Token Index: 915

Text #9

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.3163. Min Act: -0.0000

Data Index: 1358810 (C4 (Web Text))

Max Activating Token Index: 409

Text #10

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.4693. Min Act: -0.0000

Data Index: 373622 (C4 (Web Text))

Max Activating Token Index: 934

Text #11

Max Range: 8.3216. Min Range: -8.3216

Max Act: 6.0732. Min Act: -0.0000

Data Index: 621025 (C4 (Web Text))

Max Activating Token Index: 356

Text #12

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.7979. Min Act: -0.0000

Data Index: 584578 (C4 (Web Text))

Max Activating Token Index: 992

Text #13

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8906. Min Act: -0.0000

Data Index: 13126 (C4 (Web Text))

Max Activating Token Index: 826

Text #14

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8884. Min Act: -0.0000

Data Index: 839210 (C4 (Web Text))

Max Activating Token Index: 499

Text #15

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8645. Min Act: -0.0000

Data Index: 1341762 (C4 (Web Text))

Max Activating Token Index: 510

Text #16

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8765. Min Act: -0.0000

Data Index: 604186 (C4 (Web Text))

Max Activating Token Index: 548

Text #17

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8304. Min Act: -0.0000

Data Index: 684630 (C4 (Web Text))

Max Activating Token Index: 646

Text #18

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8472. Min Act: -0.0000

Data Index: 634559 (C4 (Web Text))

Max Activating Token Index: 665

Text #19

Max Range: 8.3216. Min Range: -8.3216

Max Act: 5.8118. Min Act: -0.0000

Data Index: 1280165 (C4 (Web Text))

Max Activating Token Index: 739