Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5117 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.6990. Min Act: -0.0000

Data Index: 1216147 (C4 (Web Text))

Max Activating Token Index: 880

Text #1

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.8138. Min Act: -0.0000

Data Index: 1134590 (C4 (Web Text))

Max Activating Token Index: 592

Text #2

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.7800. Min Act: -0.0000

Data Index: 210854 (C4 (Web Text))

Max Activating Token Index: 493

Text #3

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.5339. Min Act: -0.0000

Data Index: 4241 (C4 (Web Text))

Max Activating Token Index: 751

Text #4

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.5131. Min Act: -0.0000

Data Index: 772014 (C4 (Web Text))

Max Activating Token Index: 154

Text #5

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.6460. Min Act: -0.0000

Data Index: 803219 (C4 (Web Text))

Max Activating Token Index: 771

Text #6

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.4137. Min Act: -0.0000

Data Index: 848579 (C4 (Web Text))

Max Activating Token Index: 441

Text #7

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.3912. Min Act: -0.0000

Data Index: 803196 (C4 (Web Text))

Max Activating Token Index: 446

Text #8

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2574. Min Act: -0.0000

Data Index: 457835 (C4 (Web Text))

Max Activating Token Index: 637

Text #9

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2941. Min Act: -0.0000

Data Index: 1063939 (C4 (Web Text))

Max Activating Token Index: 87

Text #10

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2078. Min Act: -0.0000

Data Index: 848138 (C4 (Web Text))

Max Activating Token Index: 728

Text #11

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.1630. Min Act: -0.0000

Data Index: 883592 (C4 (Web Text))

Max Activating Token Index: 361

Text #12

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.3068. Min Act: -0.0000

Data Index: 623979 (C4 (Web Text))

Max Activating Token Index: 345

Text #13

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.1729. Min Act: -0.0000

Data Index: 1130681 (C4 (Web Text))

Max Activating Token Index: 650

Text #14

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2315. Min Act: -0.0000

Data Index: 357088 (C4 (Web Text))

Max Activating Token Index: 769

Text #15

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2732. Min Act: -0.0000

Data Index: 852139 (C4 (Web Text))

Max Activating Token Index: 366

Text #16

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2146. Min Act: -0.0000

Data Index: 33773 (C4 (Web Text))

Max Activating Token Index: 709

Text #17

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.2134. Min Act: -0.0000

Data Index: 114655 (C4 (Web Text))

Max Activating Token Index: 387

Text #18

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.1773. Min Act: -0.0000

Data Index: 1188741 (C4 (Web Text))

Max Activating Token Index: 397

Text #19

Max Range: 1.8138. Min Range: -1.8138

Max Act: 1.1379. Min Act: -0.0000

Data Index: 483407 (C4 (Web Text))

Max Activating Token Index: 570