Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2138 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.6022. Min Act: -0.0000

Data Index: 381227 (C4 (Web Text))

Max Activating Token Index: 44

Text #1

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4262. Min Act: -0.0000

Data Index: 1097146 (C4 (Web Text))

Max Activating Token Index: 859

Text #2

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4868. Min Act: -0.0000

Data Index: 1294229 (C4 (Web Text))

Max Activating Token Index: 244

Text #3

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4847. Min Act: -0.0000

Data Index: 276813 (C4 (Web Text))

Max Activating Token Index: 848

Text #4

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4343. Min Act: -0.0000

Data Index: 443055 (C4 (Web Text))

Max Activating Token Index: 958

Text #5

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4335. Min Act: -0.0000

Data Index: 847393 (C4 (Web Text))

Max Activating Token Index: 477

Text #6

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.4123. Min Act: -0.0000

Data Index: 1041731 (C4 (Web Text))

Max Activating Token Index: 977

Text #7

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3635. Min Act: -0.0000

Data Index: 581616 (C4 (Web Text))

Max Activating Token Index: 644

Text #8

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3499. Min Act: -0.0000

Data Index: 27212 (C4 (Web Text))

Max Activating Token Index: 172

Text #9

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3509. Min Act: -0.0000

Data Index: 39723 (C4 (Web Text))

Max Activating Token Index: 605

Text #10

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3484. Min Act: -0.0000

Data Index: 183983 (C4 (Web Text))

Max Activating Token Index: 735

Text #11

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3156. Min Act: -0.0000

Data Index: 228328 (C4 (Web Text))

Max Activating Token Index: 124

Text #12

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3361. Min Act: -0.0000

Data Index: 483832 (C4 (Web Text))

Max Activating Token Index: 542

Text #13

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3208. Min Act: -0.0000

Data Index: 365947 (C4 (Web Text))

Max Activating Token Index: 696

Text #14

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3301. Min Act: -0.0000

Data Index: 360209 (C4 (Web Text))

Max Activating Token Index: 539

Text #15

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.2959. Min Act: -0.0000

Data Index: 935484 (C4 (Web Text))

Max Activating Token Index: 908

Text #16

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3357. Min Act: -0.0000

Data Index: 335865 (C4 (Web Text))

Max Activating Token Index: 239

Text #17

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3136. Min Act: -0.0000

Data Index: 141212 (C4 (Web Text))

Max Activating Token Index: 364

Text #18

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.3110. Min Act: -0.0000

Data Index: 513217 (C4 (Web Text))

Max Activating Token Index: 357

Text #19

Max Range: 0.6022. Min Range: -0.6022

Max Act: 0.2944. Min Act: -0.0000

Data Index: 720698 (C4 (Web Text))

Max Activating Token Index: 541