Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5504 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.9916. Min Act: -0.0000

Data Index: 471559 (C4 (Web Text))

Max Activating Token Index: 482

Text #1

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.6336. Min Act: -0.0001

Data Index: 364677 (C4 (Web Text))

Max Activating Token Index: 341

Text #2

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.5879. Min Act: -0.0000

Data Index: 367081 (C4 (Web Text))

Max Activating Token Index: 322

Text #3

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.5506. Min Act: -0.0000

Data Index: 122635 (C4 (Web Text))

Max Activating Token Index: 806

Text #4

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.3416. Min Act: -0.0001

Data Index: 549311 (C4 (Web Text))

Max Activating Token Index: 59

Text #5

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.3484. Min Act: -0.0000

Data Index: 528210 (C4 (Web Text))

Max Activating Token Index: 363

Text #6

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1938. Min Act: -0.0001

Data Index: 814871 (C4 (Web Text))

Max Activating Token Index: 245

Text #7

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.2273. Min Act: -0.0000

Data Index: 519238 (C4 (Web Text))

Max Activating Token Index: 283

Text #8

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1345. Min Act: -0.0000

Data Index: 1044335 (C4 (Web Text))

Max Activating Token Index: 559

Text #9

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1808. Min Act: -0.0000

Data Index: 926548 (C4 (Web Text))

Max Activating Token Index: 297

Text #10

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1270. Min Act: -0.0001

Data Index: 736016 (C4 (Web Text))

Max Activating Token Index: 380

Text #11

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1001. Min Act: -0.0000

Data Index: 44210 (C4 (Web Text))

Max Activating Token Index: 429

Text #12

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1467. Min Act: -0.0000

Data Index: 1072495 (C4 (Web Text))

Max Activating Token Index: 968

Text #13

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1663. Min Act: -0.0000

Data Index: 398038 (C4 (Web Text))

Max Activating Token Index: 934

Text #14

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1021. Min Act: -0.0000

Data Index: 865172 (C4 (Web Text))

Max Activating Token Index: 180

Text #15

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.1016. Min Act: -0.0000

Data Index: 344158 (C4 (Web Text))

Max Activating Token Index: 622

Text #16

Max Range: 1.9916. Min Range: -1.9916

Max Act: 0.8474. Min Act: -0.0000

Data Index: 326852 (C4 (Web Text))

Max Activating Token Index: 254

Text #17

Max Range: 1.9916. Min Range: -1.9916

Max Act: 1.0073. Min Act: -0.0000

Data Index: 1116426 (C4 (Web Text))

Max Activating Token Index: 502

Text #18

Max Range: 1.9916. Min Range: -1.9916

Max Act: 0.9258. Min Act: -0.0001

Data Index: 940126 (C4 (Web Text))

Max Activating Token Index: 222

Text #19

Max Range: 1.9916. Min Range: -1.9916

Max Act: 0.9243. Min Act: -0.0000

Data Index: 1126773 (C4 (Web Text))

Max Activating Token Index: 980