Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 244 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.4821. Min Act: 0.0000

Data Index: 786000 (C4 (Web Text))

Max Activating Token Index: 44

Text #1

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.3141. Min Act: -0.0000

Data Index: 696876 (C4 (Web Text))

Max Activating Token Index: 192

Text #2

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.2342. Min Act: -0.0000

Data Index: 208488 (C4 (Web Text))

Max Activating Token Index: 521

Text #3

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.9034. Min Act: -0.0000

Data Index: 998183 (C4 (Web Text))

Max Activating Token Index: 23

Text #4

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.0793. Min Act: -0.0000

Data Index: 35517 (C4 (Web Text))

Max Activating Token Index: 71

Text #5

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.8923. Min Act: -0.0000

Data Index: 637228 (C4 (Web Text))

Max Activating Token Index: 37

Text #6

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.8544. Min Act: -0.0000

Data Index: 75125 (C4 (Web Text))

Max Activating Token Index: 44

Text #7

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.2150. Min Act: 0.0000

Data Index: 423188 (C4 (Web Text))

Max Activating Token Index: 42

Text #8

Max Range: 4.4821. Min Range: -4.4821

Max Act: 4.0031. Min Act: -0.0000

Data Index: 16302 (C4 (Web Text))

Max Activating Token Index: 60

Text #9

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.2541. Min Act: -0.0000

Data Index: 854019 (C4 (Web Text))

Max Activating Token Index: 226

Text #10

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.8629. Min Act: -0.0000

Data Index: 331124 (C4 (Web Text))

Max Activating Token Index: 27

Text #11

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.6629. Min Act: 0.0000

Data Index: 975948 (C4 (Web Text))

Max Activating Token Index: 27

Text #12

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.8605. Min Act: -0.0000

Data Index: 1019681 (C4 (Web Text))

Max Activating Token Index: 597

Text #13

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.9590. Min Act: -0.0000

Data Index: 242678 (C4 (Web Text))

Max Activating Token Index: 756

Text #14

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.7547. Min Act: -0.0000

Data Index: 722398 (C4 (Web Text))

Max Activating Token Index: 112

Text #15

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.5805. Min Act: -0.0000

Data Index: 1167599 (C4 (Web Text))

Max Activating Token Index: 36

Text #16

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.6967. Min Act: -0.0000

Data Index: 13237 (C4 (Web Text))

Max Activating Token Index: 130

Text #17

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.7125. Min Act: -0.0000

Data Index: 686893 (C4 (Web Text))

Max Activating Token Index: 121

Text #18

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.6674. Min Act: 0.0000

Data Index: 670093 (C4 (Web Text))

Max Activating Token Index: 41

Text #19

Max Range: 4.4821. Min Range: -4.4821

Max Act: 3.6697. Min Act: -0.0000

Data Index: 366169 (C4 (Web Text))

Max Activating Token Index: 48