Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3734 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.6707. Min Act: -0.0000

Data Index: 562206 (C4 (Web Text))

Max Activating Token Index: 532

Text #1

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.6550. Min Act: -0.0000

Data Index: 112984 (C4 (Web Text))

Max Activating Token Index: 98

Text #2

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.6453. Min Act: -0.0000

Data Index: 134153 (C4 (Web Text))

Max Activating Token Index: 41

Text #3

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5748. Min Act: -0.0000

Data Index: 366520 (C4 (Web Text))

Max Activating Token Index: 597

Text #4

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.6106. Min Act: -0.0000

Data Index: 1053921 (C4 (Web Text))

Max Activating Token Index: 25

Text #5

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.6150. Min Act: -0.0000

Data Index: 1207016 (C4 (Web Text))

Max Activating Token Index: 37

Text #6

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5915. Min Act: -0.0000

Data Index: 1131817 (C4 (Web Text))

Max Activating Token Index: 393

Text #7

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5561. Min Act: -0.0000

Data Index: 271155 (C4 (Web Text))

Max Activating Token Index: 337

Text #8

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5681. Min Act: -0.0000

Data Index: 748629 (C4 (Web Text))

Max Activating Token Index: 239

Text #9

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5871. Min Act: -0.0000

Data Index: 1116159 (C4 (Web Text))

Max Activating Token Index: 231

Text #10

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5442. Min Act: -0.0000

Data Index: 429073 (C4 (Web Text))

Max Activating Token Index: 142

Text #11

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5505. Min Act: -0.0000

Data Index: 482573 (C4 (Web Text))

Max Activating Token Index: 940

Text #12

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5673. Min Act: -0.0000

Data Index: 1074228 (C4 (Web Text))

Max Activating Token Index: 646

Text #13

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5340. Min Act: 0.0000

Data Index: 237602 (C4 (Web Text))

Max Activating Token Index: 199

Text #14

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5358. Min Act: -0.0000

Data Index: 263961 (C4 (Web Text))

Max Activating Token Index: 605

Text #15

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5460. Min Act: -0.0000

Data Index: 276408 (C4 (Web Text))

Max Activating Token Index: 860

Text #16

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5208. Min Act: -0.0000

Data Index: 1262089 (C4 (Web Text))

Max Activating Token Index: 591

Text #17

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5121. Min Act: -0.0000

Data Index: 654681 (C4 (Web Text))

Max Activating Token Index: 992

Text #18

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5227. Min Act: -0.0000

Data Index: 1344647 (C4 (Web Text))

Max Activating Token Index: 584

Text #19

Max Range: 0.6707. Min Range: -0.6707

Max Act: 0.5024. Min Act: -0.0000

Data Index: 1115306 (C4 (Web Text))

Max Activating Token Index: 975