Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2062 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.5311. Min Act: -0.0000

Data Index: 1284479 (C4 (Web Text))

Max Activating Token Index: 997

Text #1

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.4658. Min Act: -0.0000

Data Index: 1052963 (C4 (Web Text))

Max Activating Token Index: 973

Text #2

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.4279. Min Act: -0.0000

Data Index: 1545595 (Python Code)

Max Activating Token Index: 719

Text #3

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.4273. Min Act: -0.0000

Data Index: 1573642 (Python Code)

Max Activating Token Index: 630

Text #4

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3800. Min Act: -0.0000

Data Index: 395175 (C4 (Web Text))

Max Activating Token Index: 759

Text #5

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3943. Min Act: -0.0000

Data Index: 1583697 (Python Code)

Max Activating Token Index: 1020

Text #6

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3893. Min Act: -0.0000

Data Index: 1553591 (Python Code)

Max Activating Token Index: 82

Text #7

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3701. Min Act: -0.0000

Data Index: 322454 (C4 (Web Text))

Max Activating Token Index: 845

Text #8

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3677. Min Act: -0.0000

Data Index: 448024 (C4 (Web Text))

Max Activating Token Index: 971

Text #9

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3267. Min Act: -0.0000

Data Index: 528963 (C4 (Web Text))

Max Activating Token Index: 825

Text #10

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3586. Min Act: -0.0000

Data Index: 1183737 (C4 (Web Text))

Max Activating Token Index: 967

Text #11

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3517. Min Act: -0.0000

Data Index: 129622 (C4 (Web Text))

Max Activating Token Index: 282

Text #12

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3198. Min Act: -0.0000

Data Index: 578864 (C4 (Web Text))

Max Activating Token Index: 630

Text #13

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3800. Min Act: -0.0000

Data Index: 1059061 (C4 (Web Text))

Max Activating Token Index: 194

Text #14

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3284. Min Act: -0.0000

Data Index: 882733 (C4 (Web Text))

Max Activating Token Index: 844

Text #15

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3247. Min Act: -0.0000

Data Index: 509082 (C4 (Web Text))

Max Activating Token Index: 981

Text #16

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3000. Min Act: -0.0000

Data Index: 24252 (C4 (Web Text))

Max Activating Token Index: 143

Text #17

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.3124. Min Act: -0.0000

Data Index: 565179 (C4 (Web Text))

Max Activating Token Index: 936

Text #18

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.2862. Min Act: -0.0000

Data Index: 352942 (C4 (Web Text))

Max Activating Token Index: 131

Text #19

Max Range: 0.5311. Min Range: -0.5311

Max Act: 0.2952. Min Act: -0.0000

Data Index: 944628 (C4 (Web Text))

Max Activating Token Index: 975