Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3669 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.8795. Min Act: -0.0000

Data Index: 539371 (C4 (Web Text))

Max Activating Token Index: 421

Text #1

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.8438. Min Act: -0.0000

Data Index: 508369 (C4 (Web Text))

Max Activating Token Index: 642

Text #2

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.6023. Min Act: -0.0000

Data Index: 144744 (C4 (Web Text))

Max Activating Token Index: 696

Text #3

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.9122. Min Act: -0.0000

Data Index: 898512 (C4 (Web Text))

Max Activating Token Index: 915

Text #4

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.3064. Min Act: -0.0000

Data Index: 251335 (C4 (Web Text))

Max Activating Token Index: 124

Text #5

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.2390. Min Act: -0.0000

Data Index: 992953 (C4 (Web Text))

Max Activating Token Index: 848

Text #6

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.3548. Min Act: -0.0000

Data Index: 801246 (C4 (Web Text))

Max Activating Token Index: 1018

Text #7

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.2399. Min Act: -0.0000

Data Index: 914210 (C4 (Web Text))

Max Activating Token Index: 179

Text #8

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.1675. Min Act: -0.0000

Data Index: 258733 (C4 (Web Text))

Max Activating Token Index: 190

Text #9

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.0063. Min Act: -0.0000

Data Index: 855267 (C4 (Web Text))

Max Activating Token Index: 400

Text #10

Max Range: 3.9122. Min Range: -3.9122

Max Act: 2.8218. Min Act: -0.0000

Data Index: 139890 (C4 (Web Text))

Max Activating Token Index: 1007

Text #11

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.1130. Min Act: -0.0001

Data Index: 1213063 (C4 (Web Text))

Max Activating Token Index: 536

Text #12

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.2497. Min Act: -0.0000

Data Index: 858111 (C4 (Web Text))

Max Activating Token Index: 241

Text #13

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.0748. Min Act: -0.0000

Data Index: 395233 (C4 (Web Text))

Max Activating Token Index: 972

Text #14

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.0235. Min Act: -0.0000

Data Index: 608816 (C4 (Web Text))

Max Activating Token Index: 475

Text #15

Max Range: 3.9122. Min Range: -3.9122

Max Act: 2.8266. Min Act: -0.0000

Data Index: 619103 (C4 (Web Text))

Max Activating Token Index: 909

Text #16

Max Range: 3.9122. Min Range: -3.9122

Max Act: 2.9791. Min Act: -0.0000

Data Index: 293238 (C4 (Web Text))

Max Activating Token Index: 685

Text #17

Max Range: 3.9122. Min Range: -3.9122

Max Act: 2.9118. Min Act: -0.0000

Data Index: 383165 (C4 (Web Text))

Max Activating Token Index: 416

Text #18

Max Range: 3.9122. Min Range: -3.9122

Max Act: 2.7707. Min Act: -0.0000

Data Index: 1216645 (C4 (Web Text))

Max Activating Token Index: 604

Text #19

Max Range: 3.9122. Min Range: -3.9122

Max Act: 3.0026. Min Act: -0.0000

Data Index: 742710 (C4 (Web Text))

Max Activating Token Index: 678