Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 564 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.6250. Min Act: -0.0000

Data Index: 902424 (C4 (Web Text))

Max Activating Token Index: 573

Text #1

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0501. Min Act: -0.0000

Data Index: 1301219 (C4 (Web Text))

Max Activating Token Index: 737

Text #2

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.9182. Min Act: -0.0000

Data Index: 470196 (C4 (Web Text))

Max Activating Token Index: 111

Text #3

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.8846. Min Act: -0.0000

Data Index: 1103892 (C4 (Web Text))

Max Activating Token Index: 700

Text #4

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.8959. Min Act: -0.0000

Data Index: 448836 (C4 (Web Text))

Max Activating Token Index: 481

Text #5

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.7423. Min Act: -0.0000

Data Index: 150229 (C4 (Web Text))

Max Activating Token Index: 215

Text #6

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.7336. Min Act: -0.0000

Data Index: 566005 (C4 (Web Text))

Max Activating Token Index: 971

Text #7

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.8194. Min Act: -0.0000

Data Index: 540936 (C4 (Web Text))

Max Activating Token Index: 844

Text #8

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.6006. Min Act: -0.0000

Data Index: 1064359 (C4 (Web Text))

Max Activating Token Index: 982

Text #9

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.7076. Min Act: -0.0000

Data Index: 754757 (C4 (Web Text))

Max Activating Token Index: 166

Text #10

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.6931. Min Act: -0.0000

Data Index: 1166999 (C4 (Web Text))

Max Activating Token Index: 691

Text #11

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5084. Min Act: -0.0000

Data Index: 937164 (C4 (Web Text))

Max Activating Token Index: 858

Text #12

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5767. Min Act: -0.0000

Data Index: 1280517 (C4 (Web Text))

Max Activating Token Index: 820

Text #13

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5466. Min Act: -0.0000

Data Index: 602905 (C4 (Web Text))

Max Activating Token Index: 993

Text #14

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5213. Min Act: -0.0000

Data Index: 159064 (C4 (Web Text))

Max Activating Token Index: 164

Text #15

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5572. Min Act: -0.0000

Data Index: 1112194 (C4 (Web Text))

Max Activating Token Index: 29

Text #16

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.5487. Min Act: -0.0000

Data Index: 29892 (C4 (Web Text))

Max Activating Token Index: 369

Text #17

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.4516. Min Act: -0.0000

Data Index: 108126 (C4 (Web Text))

Max Activating Token Index: 847

Text #18

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.4606. Min Act: -0.0000

Data Index: 905341 (C4 (Web Text))

Max Activating Token Index: 973

Text #19

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.4337. Min Act: -0.0000

Data Index: 88042 (C4 (Web Text))

Max Activating Token Index: 640