Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 374 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.4148. Min Act: -0.0000

Data Index: 1093441 (C4 (Web Text))

Max Activating Token Index: 654

Text #1

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.2761. Min Act: -0.0000

Data Index: 1178054 (C4 (Web Text))

Max Activating Token Index: 804

Text #2

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.4040. Min Act: -0.0000

Data Index: 658423 (C4 (Web Text))

Max Activating Token Index: 799

Text #3

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.2689. Min Act: -0.0000

Data Index: 1089179 (C4 (Web Text))

Max Activating Token Index: 901

Text #4

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.1469. Min Act: -0.0000

Data Index: 688374 (C4 (Web Text))

Max Activating Token Index: 384

Text #5

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0999. Min Act: -0.0000

Data Index: 752719 (C4 (Web Text))

Max Activating Token Index: 1002

Text #6

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.9804. Min Act: -0.0000

Data Index: 129213 (C4 (Web Text))

Max Activating Token Index: 736

Text #7

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.2008. Min Act: -0.0000

Data Index: 1211784 (C4 (Web Text))

Max Activating Token Index: 736

Text #8

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0725. Min Act: -0.0000

Data Index: 715819 (C4 (Web Text))

Max Activating Token Index: 314

Text #9

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0684. Min Act: -0.0000

Data Index: 792950 (C4 (Web Text))

Max Activating Token Index: 881

Text #10

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0449. Min Act: -0.0000

Data Index: 602719 (C4 (Web Text))

Max Activating Token Index: 208

Text #11

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.1654. Min Act: -0.0000

Data Index: 28279 (C4 (Web Text))

Max Activating Token Index: 94

Text #12

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0272. Min Act: -0.0000

Data Index: 1181746 (C4 (Web Text))

Max Activating Token Index: 53

Text #13

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.1015. Min Act: -0.0000

Data Index: 1300715 (C4 (Web Text))

Max Activating Token Index: 248

Text #14

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.9792. Min Act: -0.0000

Data Index: 1191449 (C4 (Web Text))

Max Activating Token Index: 667

Text #15

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.9876. Min Act: -0.0000

Data Index: 263180 (C4 (Web Text))

Max Activating Token Index: 857

Text #16

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.7804. Min Act: -0.0000

Data Index: 431232 (C4 (Web Text))

Max Activating Token Index: 135

Text #17

Max Range: 2.4148. Min Range: -2.4148

Max Act: 2.0902. Min Act: -0.0000

Data Index: 133461 (C4 (Web Text))

Max Activating Token Index: 217

Text #18

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.9835. Min Act: -0.0000

Data Index: 863395 (C4 (Web Text))

Max Activating Token Index: 56

Text #19

Max Range: 2.4148. Min Range: -2.4148

Max Act: 1.9659. Min Act: -0.0000

Data Index: 293449 (C4 (Web Text))

Max Activating Token Index: 686