Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4439 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.0469. Min Range: -4.0469

Max Act: 4.0469. Min Act: -0.0000

Data Index: 40243 (C4 (Web Text))

Max Activating Token Index: 80

Text #1

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.4842. Min Act: -0.0000

Data Index: 842569 (C4 (Web Text))

Max Activating Token Index: 199

Text #2

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.2003. Min Act: -0.0000

Data Index: 100821 (C4 (Web Text))

Max Activating Token Index: 699

Text #3

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.0631. Min Act: -0.0000

Data Index: 150302 (C4 (Web Text))

Max Activating Token Index: 312

Text #4

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.0632. Min Act: -0.0000

Data Index: 707158 (C4 (Web Text))

Max Activating Token Index: 592

Text #5

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7578. Min Act: -0.0000

Data Index: 867939 (C4 (Web Text))

Max Activating Token Index: 242

Text #6

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.1070. Min Act: -0.0000

Data Index: 1163324 (C4 (Web Text))

Max Activating Token Index: 150

Text #7

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.1283. Min Act: -0.0000

Data Index: 932481 (C4 (Web Text))

Max Activating Token Index: 104

Text #8

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.9225. Min Act: -0.0000

Data Index: 1282525 (C4 (Web Text))

Max Activating Token Index: 911

Text #9

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.8938. Min Act: -0.0000

Data Index: 1115823 (C4 (Web Text))

Max Activating Token Index: 185

Text #10

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7506. Min Act: -0.0000

Data Index: 703931 (C4 (Web Text))

Max Activating Token Index: 458

Text #11

Max Range: 4.0469. Min Range: -4.0469

Max Act: 3.0069. Min Act: -0.0000

Data Index: 434258 (C4 (Web Text))

Max Activating Token Index: 308

Text #12

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7296. Min Act: -0.0000

Data Index: 989765 (C4 (Web Text))

Max Activating Token Index: 771

Text #13

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7365. Min Act: -0.0000

Data Index: 437568 (C4 (Web Text))

Max Activating Token Index: 773

Text #14

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7238. Min Act: -0.0000

Data Index: 610946 (C4 (Web Text))

Max Activating Token Index: 109

Text #15

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7540. Min Act: -0.0000

Data Index: 382239 (C4 (Web Text))

Max Activating Token Index: 235

Text #16

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7146. Min Act: -0.0000

Data Index: 1006381 (C4 (Web Text))

Max Activating Token Index: 592

Text #17

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.6431. Min Act: -0.0000

Data Index: 669075 (C4 (Web Text))

Max Activating Token Index: 455

Text #18

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.7729. Min Act: -0.0000

Data Index: 1229519 (C4 (Web Text))

Max Activating Token Index: 755

Text #19

Max Range: 4.0469. Min Range: -4.0469

Max Act: 2.6175. Min Act: -0.0000

Data Index: 821728 (C4 (Web Text))

Max Activating Token Index: 729