Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3890 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.4641. Min Act: -0.0000

Data Index: 302849 (C4 (Web Text))

Max Activating Token Index: 843

Text #1

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.4283. Min Act: -0.0000

Data Index: 856548 (C4 (Web Text))

Max Activating Token Index: 174

Text #2

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.3848. Min Act: -0.0000

Data Index: 883790 (C4 (Web Text))

Max Activating Token Index: 129

Text #3

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.3117. Min Act: -0.0000

Data Index: 941415 (C4 (Web Text))

Max Activating Token Index: 560

Text #4

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.2950. Min Act: -0.0000

Data Index: 537379 (C4 (Web Text))

Max Activating Token Index: 821

Text #5

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.2489. Min Act: -0.0000

Data Index: 101726 (C4 (Web Text))

Max Activating Token Index: 559

Text #6

Max Range: 3.4641. Min Range: -3.4641

Max Act: 3.0797. Min Act: -0.0000

Data Index: 1329941 (C4 (Web Text))

Max Activating Token Index: 674

Text #7

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.8359. Min Act: -0.0000

Data Index: 1196943 (C4 (Web Text))

Max Activating Token Index: 223

Text #8

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.7990. Min Act: -0.0000

Data Index: 88415 (C4 (Web Text))

Max Activating Token Index: 779

Text #9

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.6490. Min Act: -0.0000

Data Index: 934133 (C4 (Web Text))

Max Activating Token Index: 194

Text #10

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.6879. Min Act: -0.0000

Data Index: 1310843 (C4 (Web Text))

Max Activating Token Index: 27

Text #11

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.7718. Min Act: -0.0000

Data Index: 1192903 (C4 (Web Text))

Max Activating Token Index: 549

Text #12

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.7720. Min Act: -0.0000

Data Index: 1301455 (C4 (Web Text))

Max Activating Token Index: 232

Text #13

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.5007. Min Act: -0.0000

Data Index: 924246 (C4 (Web Text))

Max Activating Token Index: 274

Text #14

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.4580. Min Act: -0.0000

Data Index: 1284183 (C4 (Web Text))

Max Activating Token Index: 219

Text #15

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.6148. Min Act: -0.0000

Data Index: 953629 (C4 (Web Text))

Max Activating Token Index: 55

Text #16

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.2286. Min Act: -0.0000

Data Index: 647995 (C4 (Web Text))

Max Activating Token Index: 319

Text #17

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.6093. Min Act: -0.0000

Data Index: 1041102 (C4 (Web Text))

Max Activating Token Index: 13

Text #18

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.3359. Min Act: -0.0000

Data Index: 1033844 (C4 (Web Text))

Max Activating Token Index: 1002

Text #19

Max Range: 3.4641. Min Range: -3.4641

Max Act: 2.6655. Min Act: -0.0000

Data Index: 1042829 (C4 (Web Text))

Max Activating Token Index: 330