Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4901 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.4286. Min Act: -0.0000

Data Index: 139004 (C4 (Web Text))

Max Activating Token Index: 1018

Text #1

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.4117. Min Act: -0.0000

Data Index: 391572 (C4 (Web Text))

Max Activating Token Index: 286

Text #2

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3613. Min Act: -0.0000

Data Index: 398139 (C4 (Web Text))

Max Activating Token Index: 195

Text #3

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3182. Min Act: -0.0000

Data Index: 315437 (C4 (Web Text))

Max Activating Token Index: 832

Text #4

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3363. Min Act: 0.0000

Data Index: 976853 (C4 (Web Text))

Max Activating Token Index: 526

Text #5

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3335. Min Act: -0.0000

Data Index: 1070676 (C4 (Web Text))

Max Activating Token Index: 904

Text #6

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3481. Min Act: -0.0000

Data Index: 1114826 (C4 (Web Text))

Max Activating Token Index: 222

Text #7

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3327. Min Act: -0.0000

Data Index: 12333 (C4 (Web Text))

Max Activating Token Index: 682

Text #8

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3098. Min Act: 0.0000

Data Index: 259593 (C4 (Web Text))

Max Activating Token Index: 558

Text #9

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.2996. Min Act: -0.0000

Data Index: 1209957 (C4 (Web Text))

Max Activating Token Index: 528

Text #10

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3121. Min Act: -0.0000

Data Index: 1101898 (C4 (Web Text))

Max Activating Token Index: 856

Text #11

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.2910. Min Act: -0.0000

Data Index: 900344 (C4 (Web Text))

Max Activating Token Index: 228

Text #12

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3023. Min Act: -0.0000

Data Index: 134093 (C4 (Web Text))

Max Activating Token Index: 372

Text #13

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3114. Min Act: 0.0000

Data Index: 897720 (C4 (Web Text))

Max Activating Token Index: 584

Text #14

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3280. Min Act: -0.0000

Data Index: 617161 (C4 (Web Text))

Max Activating Token Index: 360

Text #15

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3095. Min Act: -0.0000

Data Index: 606820 (C4 (Web Text))

Max Activating Token Index: 362

Text #16

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.2694. Min Act: -0.0000

Data Index: 1346185 (C4 (Web Text))

Max Activating Token Index: 806

Text #17

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.2949. Min Act: -0.0000

Data Index: 335682 (C4 (Web Text))

Max Activating Token Index: 803

Text #18

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.2942. Min Act: -0.0000

Data Index: 9605 (C4 (Web Text))

Max Activating Token Index: 411

Text #19

Max Range: 0.4286. Min Range: -0.4286

Max Act: 0.3054. Min Act: -0.0000

Data Index: 843576 (C4 (Web Text))

Max Activating Token Index: 391