Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2337 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1450. Min Range: -1.1450

Max Act: 1.1450. Min Act: -0.0000

Data Index: 844439 (C4 (Web Text))

Max Activating Token Index: 910

Text #1

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.9497. Min Act: -0.0000

Data Index: 1348787 (C4 (Web Text))

Max Activating Token Index: 284

Text #2

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8447. Min Act: -0.0000

Data Index: 546131 (C4 (Web Text))

Max Activating Token Index: 730

Text #3

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8793. Min Act: -0.0000

Data Index: 242863 (C4 (Web Text))

Max Activating Token Index: 46

Text #4

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8976. Min Act: -0.0000

Data Index: 700016 (C4 (Web Text))

Max Activating Token Index: 926

Text #5

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8177. Min Act: -0.0000

Data Index: 796421 (C4 (Web Text))

Max Activating Token Index: 541

Text #6

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8344. Min Act: -0.0000

Data Index: 57818 (C4 (Web Text))

Max Activating Token Index: 524

Text #7

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8708. Min Act: -0.0000

Data Index: 1288499 (C4 (Web Text))

Max Activating Token Index: 44

Text #8

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8216. Min Act: -0.0000

Data Index: 1324856 (C4 (Web Text))

Max Activating Token Index: 71

Text #9

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7699. Min Act: -0.0000

Data Index: 16386 (C4 (Web Text))

Max Activating Token Index: 788

Text #10

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8049. Min Act: -0.0000

Data Index: 540092 (C4 (Web Text))

Max Activating Token Index: 129

Text #11

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8509. Min Act: -0.0000

Data Index: 66601 (C4 (Web Text))

Max Activating Token Index: 442

Text #12

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8214. Min Act: -0.0000

Data Index: 213233 (C4 (Web Text))

Max Activating Token Index: 797

Text #13

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7949. Min Act: -0.0000

Data Index: 902163 (C4 (Web Text))

Max Activating Token Index: 610

Text #14

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7557. Min Act: -0.0000

Data Index: 880570 (C4 (Web Text))

Max Activating Token Index: 757

Text #15

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7888. Min Act: -0.0000

Data Index: 241075 (C4 (Web Text))

Max Activating Token Index: 871

Text #16

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.6943. Min Act: -0.0000

Data Index: 289298 (C4 (Web Text))

Max Activating Token Index: 853

Text #17

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.8079. Min Act: -0.0000

Data Index: 289712 (C4 (Web Text))

Max Activating Token Index: 152

Text #18

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7766. Min Act: -0.0000

Data Index: 793771 (C4 (Web Text))

Max Activating Token Index: 131

Text #19

Max Range: 1.1450. Min Range: -1.1450

Max Act: 0.7363. Min Act: -0.0000

Data Index: 550981 (C4 (Web Text))

Max Activating Token Index: 730