Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1296 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.4022. Min Act: -0.0000

Data Index: 1014154 (C4 (Web Text))

Max Activating Token Index: 681

Text #1

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.3611. Min Act: -0.0000

Data Index: 1040050 (C4 (Web Text))

Max Activating Token Index: 264

Text #2

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.3306. Min Act: -0.0000

Data Index: 444813 (C4 (Web Text))

Max Activating Token Index: 382

Text #3

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2739. Min Act: -0.0000

Data Index: 1033224 (C4 (Web Text))

Max Activating Token Index: 257

Text #4

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.3251. Min Act: -0.0000

Data Index: 235194 (C4 (Web Text))

Max Activating Token Index: 289

Text #5

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2881. Min Act: -0.0000

Data Index: 222075 (C4 (Web Text))

Max Activating Token Index: 226

Text #6

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2807. Min Act: -0.0000

Data Index: 1013298 (C4 (Web Text))

Max Activating Token Index: 910

Text #7

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2613. Min Act: -0.0000

Data Index: 67657 (C4 (Web Text))

Max Activating Token Index: 981

Text #8

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2921. Min Act: -0.0000

Data Index: 1618934 (Python Code)

Max Activating Token Index: 129

Text #9

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2622. Min Act: -0.0000

Data Index: 1130818 (C4 (Web Text))

Max Activating Token Index: 797

Text #10

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2625. Min Act: -0.0000

Data Index: 468552 (C4 (Web Text))

Max Activating Token Index: 449

Text #11

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2690. Min Act: -0.0000

Data Index: 750634 (C4 (Web Text))

Max Activating Token Index: 543

Text #12

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2412. Min Act: -0.0000

Data Index: 859711 (C4 (Web Text))

Max Activating Token Index: 20

Text #13

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2238. Min Act: -0.0000

Data Index: 832287 (C4 (Web Text))

Max Activating Token Index: 316

Text #14

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2189. Min Act: -0.0000

Data Index: 1182625 (C4 (Web Text))

Max Activating Token Index: 191

Text #15

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2378. Min Act: -0.0000

Data Index: 377640 (C4 (Web Text))

Max Activating Token Index: 124

Text #16

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2234. Min Act: -0.0000

Data Index: 658314 (C4 (Web Text))

Max Activating Token Index: 999

Text #17

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2200. Min Act: -0.0000

Data Index: 721808 (C4 (Web Text))

Max Activating Token Index: 723

Text #18

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2073. Min Act: -0.0000

Data Index: 254103 (C4 (Web Text))

Max Activating Token Index: 747

Text #19

Max Range: 0.4022. Min Range: -0.4022

Max Act: 0.2103. Min Act: -0.0000

Data Index: 252039 (C4 (Web Text))

Max Activating Token Index: 255