Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1846 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.9200. Min Act: -0.0000

Data Index: 1330806 (C4 (Web Text))

Max Activating Token Index: 323

Text #1

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.7016. Min Act: -0.0000

Data Index: 1013364 (C4 (Web Text))

Max Activating Token Index: 485

Text #2

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.6756. Min Act: -0.0000

Data Index: 1246675 (C4 (Web Text))

Max Activating Token Index: 666

Text #3

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.6527. Min Act: -0.0000

Data Index: 864111 (C4 (Web Text))

Max Activating Token Index: 994

Text #4

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.6053. Min Act: -0.0000

Data Index: 32191 (C4 (Web Text))

Max Activating Token Index: 771

Text #5

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.6112. Min Act: -0.0000

Data Index: 1064192 (C4 (Web Text))

Max Activating Token Index: 303

Text #6

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5827. Min Act: -0.0000

Data Index: 382295 (C4 (Web Text))

Max Activating Token Index: 721

Text #7

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5578. Min Act: -0.0000

Data Index: 426931 (C4 (Web Text))

Max Activating Token Index: 182

Text #8

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5957. Min Act: -0.0000

Data Index: 908433 (C4 (Web Text))

Max Activating Token Index: 811

Text #9

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5783. Min Act: -0.0000

Data Index: 1260706 (C4 (Web Text))

Max Activating Token Index: 131

Text #10

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5191. Min Act: -0.0000

Data Index: 1010229 (C4 (Web Text))

Max Activating Token Index: 618

Text #11

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5402. Min Act: -0.0000

Data Index: 355632 (C4 (Web Text))

Max Activating Token Index: 114

Text #12

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5427. Min Act: -0.0000

Data Index: 314928 (C4 (Web Text))

Max Activating Token Index: 218

Text #13

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5549. Min Act: -0.0000

Data Index: 473278 (C4 (Web Text))

Max Activating Token Index: 429

Text #14

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5354. Min Act: -0.0000

Data Index: 510327 (C4 (Web Text))

Max Activating Token Index: 94

Text #15

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.4782. Min Act: -0.0000

Data Index: 801526 (C4 (Web Text))

Max Activating Token Index: 844

Text #16

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.4991. Min Act: -0.0000

Data Index: 1031841 (C4 (Web Text))

Max Activating Token Index: 811

Text #17

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5167. Min Act: -0.0000

Data Index: 867074 (C4 (Web Text))

Max Activating Token Index: 744

Text #18

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.5183. Min Act: -0.0000

Data Index: 41294 (C4 (Web Text))

Max Activating Token Index: 622

Text #19

Max Range: 0.9200. Min Range: -0.9200

Max Act: 0.4630. Min Act: -0.0000

Data Index: 238522 (C4 (Web Text))

Max Activating Token Index: 589