Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2637 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.9388. Min Act: -0.0000

Data Index: 356703 (C4 (Web Text))

Max Activating Token Index: 353

Text #1

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.7126. Min Act: -0.0000

Data Index: 987368 (C4 (Web Text))

Max Activating Token Index: 255

Text #2

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6343. Min Act: -0.0000

Data Index: 1117344 (C4 (Web Text))

Max Activating Token Index: 359

Text #3

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4972. Min Act: -0.0000

Data Index: 869647 (C4 (Web Text))

Max Activating Token Index: 537

Text #4

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6129. Min Act: -0.0000

Data Index: 568803 (C4 (Web Text))

Max Activating Token Index: 1002

Text #5

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.9025. Min Act: -0.0000

Data Index: 292252 (C4 (Web Text))

Max Activating Token Index: 253

Text #6

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6049. Min Act: -0.0000

Data Index: 282821 (C4 (Web Text))

Max Activating Token Index: 1000

Text #7

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6054. Min Act: -0.0000

Data Index: 548612 (C4 (Web Text))

Max Activating Token Index: 220

Text #8

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6946. Min Act: -0.0000

Data Index: 747597 (C4 (Web Text))

Max Activating Token Index: 581

Text #9

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.6172. Min Act: -0.0000

Data Index: 699664 (C4 (Web Text))

Max Activating Token Index: 49

Text #10

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.5638. Min Act: -0.0000

Data Index: 128674 (C4 (Web Text))

Max Activating Token Index: 60

Text #11

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4930. Min Act: -0.0000

Data Index: 260536 (C4 (Web Text))

Max Activating Token Index: 778

Text #12

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4808. Min Act: -0.0000

Data Index: 1244215 (C4 (Web Text))

Max Activating Token Index: 817

Text #13

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.5377. Min Act: -0.0000

Data Index: 28693 (C4 (Web Text))

Max Activating Token Index: 887

Text #14

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.3372. Min Act: -0.0000

Data Index: 379763 (C4 (Web Text))

Max Activating Token Index: 296

Text #15

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.5702. Min Act: -0.0000

Data Index: 348962 (C4 (Web Text))

Max Activating Token Index: 467

Text #16

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4269. Min Act: -0.0000

Data Index: 1501530 (Python Code)

Max Activating Token Index: 857

Text #17

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.5088. Min Act: -0.0000

Data Index: 1242766 (C4 (Web Text))

Max Activating Token Index: 322

Text #18

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4194. Min Act: -0.0000

Data Index: 135790 (C4 (Web Text))

Max Activating Token Index: 319

Text #19

Max Range: 1.9388. Min Range: -1.9388

Max Act: 1.4302. Min Act: -0.0000

Data Index: 670901 (C4 (Web Text))

Max Activating Token Index: 431