Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 621 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5946. Min Act: -0.0000

Data Index: 640299 (C4 (Web Text))

Max Activating Token Index: 689

Text #1

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.6211. Min Act: -0.0000

Data Index: 1359824 (C4 (Web Text))

Max Activating Token Index: 637

Text #2

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5825. Min Act: -0.0000

Data Index: 1181481 (C4 (Web Text))

Max Activating Token Index: 138

Text #3

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5975. Min Act: -0.0000

Data Index: 1347648 (C4 (Web Text))

Max Activating Token Index: 388

Text #4

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5688. Min Act: -0.0000

Data Index: 925289 (C4 (Web Text))

Max Activating Token Index: 552

Text #5

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5625. Min Act: -0.0000

Data Index: 509045 (C4 (Web Text))

Max Activating Token Index: 155

Text #6

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5135. Min Act: -0.0000

Data Index: 923227 (C4 (Web Text))

Max Activating Token Index: 631

Text #7

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5114. Min Act: -0.0000

Data Index: 714376 (C4 (Web Text))

Max Activating Token Index: 703

Text #8

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4932. Min Act: -0.0000

Data Index: 198542 (C4 (Web Text))

Max Activating Token Index: 9

Text #9

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.5113. Min Act: -0.0000

Data Index: 380695 (C4 (Web Text))

Max Activating Token Index: 964

Text #10

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4846. Min Act: -0.0000

Data Index: 354334 (C4 (Web Text))

Max Activating Token Index: 315

Text #11

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4821. Min Act: -0.0000

Data Index: 978835 (C4 (Web Text))

Max Activating Token Index: 349

Text #12

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4737. Min Act: -0.0000

Data Index: 538354 (C4 (Web Text))

Max Activating Token Index: 6

Text #13

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4701. Min Act: -0.0000

Data Index: 792034 (C4 (Web Text))

Max Activating Token Index: 339

Text #14

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4533. Min Act: -0.0000

Data Index: 26121 (C4 (Web Text))

Max Activating Token Index: 622

Text #15

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4657. Min Act: -0.0000

Data Index: 209209 (C4 (Web Text))

Max Activating Token Index: 359

Text #16

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4531. Min Act: -0.0000

Data Index: 554991 (C4 (Web Text))

Max Activating Token Index: 396

Text #17

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4441. Min Act: -0.0000

Data Index: 408635 (C4 (Web Text))

Max Activating Token Index: 884

Text #18

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4372. Min Act: -0.0000

Data Index: 590349 (C4 (Web Text))

Max Activating Token Index: 396

Text #19

Max Range: 0.6211. Min Range: -0.6211

Max Act: 0.4396. Min Act: -0.0000

Data Index: 1122745 (C4 (Web Text))

Max Activating Token Index: 524