Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1708 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.7641. Min Act: -0.0000

Data Index: 1114969 (C4 (Web Text))

Max Activating Token Index: 389

Text #1

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6479. Min Act: -0.0000

Data Index: 134405 (C4 (Web Text))

Max Activating Token Index: 861

Text #2

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6626. Min Act: -0.0000

Data Index: 506699 (C4 (Web Text))

Max Activating Token Index: 686

Text #3

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6784. Min Act: -0.0000

Data Index: 501173 (C4 (Web Text))

Max Activating Token Index: 118

Text #4

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6977. Min Act: -0.0000

Data Index: 1133862 (C4 (Web Text))

Max Activating Token Index: 438

Text #5

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6796. Min Act: -0.0000

Data Index: 1337938 (C4 (Web Text))

Max Activating Token Index: 805

Text #6

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.5791. Min Act: -0.0000

Data Index: 216727 (C4 (Web Text))

Max Activating Token Index: 567

Text #7

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6659. Min Act: -0.0000

Data Index: 1005239 (C4 (Web Text))

Max Activating Token Index: 822

Text #8

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6347. Min Act: -0.0000

Data Index: 1041404 (C4 (Web Text))

Max Activating Token Index: 67

Text #9

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6300. Min Act: -0.0000

Data Index: 310431 (C4 (Web Text))

Max Activating Token Index: 199

Text #10

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6423. Min Act: -0.0000

Data Index: 349608 (C4 (Web Text))

Max Activating Token Index: 883

Text #11

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6249. Min Act: -0.0000

Data Index: 1208076 (C4 (Web Text))

Max Activating Token Index: 1021

Text #12

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6203. Min Act: -0.0000

Data Index: 1341855 (C4 (Web Text))

Max Activating Token Index: 808

Text #13

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6885. Min Act: -0.0000

Data Index: 21631 (C4 (Web Text))

Max Activating Token Index: 292

Text #14

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6428. Min Act: -0.0000

Data Index: 1126366 (C4 (Web Text))

Max Activating Token Index: 380

Text #15

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.6366. Min Act: -0.0000

Data Index: 363827 (C4 (Web Text))

Max Activating Token Index: 226

Text #16

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.5966. Min Act: -0.0000

Data Index: 1253616 (C4 (Web Text))

Max Activating Token Index: 127

Text #17

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.5495. Min Act: -0.0000

Data Index: 892743 (C4 (Web Text))

Max Activating Token Index: 973

Text #18

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.5666. Min Act: -0.0000

Data Index: 657503 (C4 (Web Text))

Max Activating Token Index: 285

Text #19

Max Range: 0.7641. Min Range: -0.7641

Max Act: 0.5987. Min Act: -0.0000

Data Index: 127346 (C4 (Web Text))

Max Activating Token Index: 994